Question about time of execution of a big sort



IBM's flagship sort product DFSORT for sorting, merging, copying, data manipulation and reporting. Includes ICETOOL and ICEGENER

Re: Question about time of execution of a big sort

Postby BillyBoyo » Tue Mar 12, 2013 2:49 pm

It is the correct stuff, but we need to see the message numbers as well, all the things which start with "ICE".

Even without the message numbers, I can see that these are pointing towards some changes that can be made:

RECORDS SORTED - PROCESSED: 67451219, EXPECTED: 1725982                       
AVERAGE RECORD LENGTH - PROCESSED: 338, EXPECTED: 13997                       
TOTAL WORK DATA SET TRACKS ALLOCATED: 1766025 , TRACKS USED: 5460


I don't know why you thought there were 8m records on the file, there were 67,451,219.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Re: Question about time of execution of a big sort

Postby samb01 » Tue Mar 12, 2013 3:06 pm

Sorry this example is not the same tha the 8 M lines.

ICE088I 0 EPRSDPH4.D       .S89B    , INPUT LRECL = 27994, BLKSIZE = 27998, TYP
ICE092I 0 MAIN STORAGE = (31457280,31457280,31457280)                         
ICE156I 0 MAIN STORAGE ABOVE 16MB = (31530992,31399920)                       
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y
ICE128I 0 OPTIONS: SIZE=31457280,MAXLIM=1048576,MINLIM=450560,EQUALS=Y,LIST=Y,E
ICE129I 0 OPTIONS: VIO=Y,RESDNT=ALL ,SMF=NO   ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT
ICE130I 0 OPTIONS: RESALL=0,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=N,OUTREL=Y,CKPT=N,
ICE131I 0 OPTIONS: TMAXLIM=8388608,ARESALL=131072,ARESINV=0,OVERRGN=65536,CINV=
ICE132I 0 OPTIONS: VLSHRT=Y,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXIT
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAM
ICE235I 0 OPTIONS: NULLOUT=RC0                                                 
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y                                         
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                 
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTIN                                   
ICE750I 0 DC 0 TC 24158578264 CS DSVRR KSZ 74 VSZ 74                           

ICE752I 0 FSZ=24158578264 BC  IGN=0 E  AVG=14035 0  WSP=31463017 C  DYN=0 0   
ICE751I 1 D8-K58148 D4-K59452 EA-K79084 F1-K58148 E8-K79990                   
ICE090I 0 OUTPUT LRECL = 27994, BLKSIZE = 27998, TYPE = VB                     
ICE055I 0 INSERT 0, DELETE 0                                                   
ICE054I 0 RECORDS - IN: 67451219, OUT: 67451219                               
ICE134I 0 NUMBER OF BYTES SORTED: 22823101372                                 
ICE253I 0 RECORDS SORTED - PROCESSED: 67451219, EXPECTED: 1725982             
ICE098I 0 AVERAGE RECORD LENGTH - PROCESSED: 338, EXPECTED: 13997             
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 1766025 , TRACKS USED: 5460   
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES                       
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES                       
ICE180I 0 HIPERSPACE STORAGE USED = 22330368K BYTES                           
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                   
ICE052I 0 END OF DFSORT                                                       
ICE201I H RECORD TYPE IS V - DATA STARTS IN POSITION 5                         
ICE751I 0 C5-K76982 C6-K90026 C7-K82419 C8-K67572 E9-K60824 C9-BASE   E5-K80744
ICE143I 0 BLOCKSET     SORT  TECHNIQUE SELECTED                               
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES A
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 07:57 ON SAT M

           SORT FIELDS=(5,19,BI,A,25,50,BI,A,24,1,BI,A)                       
ICE201I H RECORD TYPE IS V - DATA STARTS IN POSITION 5                         
ICE751I 0 C5-K76982 C6-K90026 C7-K82419 C8-K67572 E9-K60824 C9-BASE   E5-K80744
ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
ICE252I 1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS       

samb01
 
Posts: 431
Joined: Mon Nov 16, 2009 7:24 pm
Has thanked: 1 time
Been thanked: 0 time

Re: Question about time of execution of a big sort

Postby steve-myers » Tue Mar 12, 2013 3:41 pm

It's amazing: 8 x 10^6 records morphed into 67 x 10^6 records! More important, though I'm far from being a sort expert, is the average record length is 338 bytes. If I read the times correctly, 21 minutes 46 seconds elapsed time. Again, if I read the messages correctly, there was 30 times more input than sort expected, which slows it down. That's why the record count you give to sort is so important! Well, hopefully a real sort expert can can clear things up and produce valid advice. Anything more I say is mostly guessing. The data may be too large for a virtual tape product to process efficiently, but that's more personal prejudice speaking, not real knowledge. You might be better off trying to force the data onto real tapes. Well, I'm awfully close to getting foot in mouth disease, if I haven't caught it already! I better shut up.
steve-myers
Global moderator
 
Posts: 2105
Joined: Thu Jun 03, 2010 6:21 pm
Has thanked: 4 times
Been thanked: 243 times

Re: Question about time of execution of a big sort

Postby BillyBoyo » Tue Mar 12, 2013 3:48 pm

Did you paste everything in the order in which it appears? If so, you seem to have parts of two Sorts, so we'd need to see the rest. Otherwise, please post again in the correct order.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Re: Question about time of execution of a big sort

Postby BillyBoyo » Tue Mar 12, 2013 4:13 pm

Please run this:

//STEP0100 EXEC PGM=SORT
//SYSOUT   DD SYSOUT=*
//SORTOUT  DD SYSOUT=*
//SYSIN    DD *
  OPTION NOEQUALS
                                                               
  INREC BUILD=(1,4,1,2,BI,TO=ZDF,LENGTH=5,C'1',9X)
  SORT FIELDS=(5,5,CH,A)
  OUTFIL REMOVECC,NODETAIL,
         HEADER1=(C' RECL',11:C'COUNT'),
         SECTIONS=(5,5,TRAILER3=(5,5,COUNT=(EDIT=(II,III,IIT))))
//SORTIN   DD DISP=SHR,DSN=your dataset here,


First, with your test file, then, if possible, with your big file.

Post the results from SORTOUT (it will produce a little report) and the SYSOUT from the step, please.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Previous

Return to DFSORT/ICETOOL/ICEGENER

 


  • Related topics
    Replies
    Views
    Last post