Page 1 of 2

Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 8:33 pm
by samb01
Hello,

i have o very big file : About 8 000 000 lines.

And the sysin of sort is :

 SORT FIELDS=(5,19,BI,A,25,50,BI,A,24,1,BI,A) 



This sort takes o long time.

I wonder if i have two files containing each one : 4 000 000 lines and i sort it one after the other file (not à the same time )

Does il will take less time...

Sorry for my bad english...

thank's for your help.

Re: Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 8:38 pm
by BillyBoyo
Can you post the sysout from the step please - all the Sort messages?

What do you consider to be "long". Eight million records is not so many. Are there any "tapes" involved?

Re: Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 9:00 pm
by samb01
This is the jcl :

//STRI     EXEC PGM=SORT,PARM='SIZE=30M'                           
//    INCLUDE MEMBER=SORTWK                                         
//SORTIN   DD DSN=B1.SORT1,DISP=OLD                                 
//SORTOUT DD UNIT=VTS2,LABEL=RETPD=3,VOL=(,,,20),                   
//    DISP=(,CATLG,KEEP),                                           
//    DCB=(RECFM=VB,LRECL=27994,BLKSIZE=27998,TRTCH=COMP)           
//SORTWK05 DD SPACE=(CYL,(200,50))                                 
//SORTWK06 DD SPACE=(CYL,9500),STORCLAS=SCRORP,DSNTYPE=LARGE,       
//         DSN=&VVQUAL1..&VVQUAL2..A,DISP=(,DELETE),VOL=SER=FRGTO01
//SORTWK07 DD SPACE=(CYL,9500),STORCLAS=SCRORP,DSNTYPE=LARGE,       
//         DSN=&VVQUAL1..&VVQUAL2..B,DISP=(,DELETE),VOL=SER=FRGTO02
//SORTWK08 DD SPACE=(CYL,9500),STORCLAS=SCRORP,DSNTYPE=LARGE,       
//         DSN=&VVQUAL1..&VVQUAL2..C,DISP=(,DELETE),VOL=SER=FRGTO03
//SORTWK09 DD SPACE=(CYL,9500),STORCLAS=SCRORP,DSNTYPE=LARGE,       
//         DSN=&VVQUAL1..&VVQUAL2..D,DISP=(,DELETE),VOL=SER=FRGTO04
//SORTWK10 DD SPACE=(CYL,9500),STORCLAS=SCRORP,DSNTYPE=LARGE,       
//         DSN=&VVQUAL1..&VVQUAL2..E,DISP=(,DELETE),VOL=SER=FRGTO05
//SORTWK11 DD SPACE=(CYL,9500),STORCLAS=SCRORP,DSNTYPE=LARGE,       


and the sysout :

07.32.26 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.26 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK06,FGRBG.EGBHGFR.A     
07.32.26 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.26 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK06,FGRBG.EGBHGFR.A     
07.32.26 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL   
07.32.28 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.28 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK07,FGRBG.EGBHGFR.B     
07.32.28 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.28 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK07,FGRBG.EGBHGFR.B     
07.32.28 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL   
07.32.30 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.30 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK08,FGRBG.EGBHGFR.C     
07.32.30 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.30 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK08,FGRBG.EGBHGFR.C     
07.32.30 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL   
07.32.33 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.33 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK09,FGRBG.EGBHGFR.D     
07.32.33 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.33 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK09,FGRBG.EGBHGFR.D     
07.32.33 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL   
07.32.37 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.37 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK10,FGRBG.EGBHGFR.E     
07.32.37 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.37 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK10,FGRBG.EGBHGFR.E     
07.32.37 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.32.40 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.40 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK11,FGRBG.EGBHGFR.F     
07.32.40 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.40 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK11,FGRBG.EGBHGFR.F     
07.32.40 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.32.44 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.44 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK12,FGRBG.EGBHGFR.G     
07.32.44 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.44 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK12,FGRBG.EGBHGFR.G     
07.32.44 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.32.48 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.48 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK13,FGRBG.EGBHGFR.H     
07.32.48 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.48 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK13,FGRBG.EGBHGFR.H     
07.32.48 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.32.51 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.51 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK14,FGRBG.EGBHGFR.I     
07.32.51 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.51 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK14,FGRBG.EGBHGFR.I     
07.32.51 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.32.54 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.54 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK15,FGRBG.EGBHGFR.J     
07.32.54 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.54 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK15,FGRBG.EGBHGFR.J     
07.32.54 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.32.58 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.32.58 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK16,FGRBG.EGBHGFR.K     
07.32.58 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.32.58 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK16,FGRBG.EGBHGFR.K     
07.32.58 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 8550 CYL 
07.33.03 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.33.03 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK17,FGRBG.EGBHGFR.L     
07.33.03 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.33.03 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK17,FGRBG.EGBHGFR.L     
07.33.03 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.33.08 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.33.08 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK18,FGRBG.EGBHGFR.M     
07.33.08 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.33.08 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK18,FGRBG.EGBHGFR.M     
07.33.08 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 8550 CYL 
07.33.11 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.33.11 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK19,FGRBG.EGBHGFR.N     
07.33.11 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.33.11 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK19,FGRBG.EGBHGFR.N     
07.33.11 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.33.15 JOB00104  VAM0096 DOES IT FIT SUPPORT REDUCE TO FIT OPERATION
07.33.15 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK20,FGRBG.EGBHGFR.O     
07.33.15 JOB00104  VAM0096 REDUCE TO FIT OPERATION SUCCESSFUL FOR     
07.33.15 JOB00104  VAM0096 FGRPJCLA,S89B,SORTWK20,FGRBG.EGBHGFR.O     
07.33.15 JOB00104  VAM0096 PRIMARY REDUCED FROM 9500 CYL TO 7695 CYL 
07.35.28 JOB00104  IEF233A M 0EAE,269131,,FGRPJCLA,S89B,FGRBG.EGBHGFR.
07.35.28 JOB00104  IEF233A M 0D41,PRIVAT,SL,FGRPJCLA,S89B,FGRBG.EGBHGF
07.36.47 JOB00104  IEC502E K 0EAE,269131,SL,FGRPJCLA,S89B,FGRBG.EGBHGF
07.36.47 JOB00104  TMS014  IEC502E K 0EAE,269131,SL,FGRPJCLA,S89B,FGRB
07.36.48 JOB00104  IEC501A M 0EAE,221189,SL,COMP,FGRPJCLA,S89B.D,FGRBG
07.38.11 JOB00104  IEC502E K 0EAE,221189,SL,FGRPJCLA,S89B,FGRBG.EGBHGF
07.38.11 JOB00104  TMS014  IEC502E K 0EAE,221189,SL,FGRPJCLA,S89B,FGRB
07.38.11 JOB00104  IEC501A M 0EAE,223495,SL,COMP,FGRPJCLA,S89B.D,FGRBG
07.39.30 JOB00104  IEC502E K 0EAE,223495,SL,FGRPJCLA,S89B,FGRBG.EGBHGFR
07.39.30 JOB00104  TMS014  IEC502E K 0EAE,223495,SL,FGRPJCLA,S89B,FGRBG
07.39.30 JOB00104  IEC501A M 0EAE,318470,SL,COMP,FGRPJCLA,S89B.D,FGRBG.
07.40.54 JOB00104  IEC502E K 0EAE,318470,SL,FGRPJCLA,S89B,FGRBG.EGBHGFR
07.40.54 JOB00104  TMS014  IEC502E K 0EAE,318470,SL,FGRPJCLA,S89B,FGRBG
07.40.54 JOB00104  IEC501A M 0EAE,319554,SL,COMP,FGRPJCLA,S89B.D,FGRBG.
07.41.40 JOB00104  IECTMS9 0D41,228305,FGRPJCLA,SORTOUT ,12/03/2013,000
07.41.48 JOB00104  IEC705I TAPE ON 0D41,228305,SL,COMP,FGRPJCLA,S89B.D,
07.43.43 JOB00104  IEC502E K 0D41,228305,SL,FGRPJCLA,S89B,FGRBG.EGBHGFR
07.43.43 JOB00104  TMS014  IEC502E K 0D41,228305,SL,FGRPJCLA,S89B,FGRBG
07.43.43 JOB00104  IEC501A M 0D41,PRIVAT,SL,COMP,FGRPJCLA,S89B.D,FGRBG.
07.43.47 JOB00104  IECTMS9 0D41,234658,FGRPJCLA,SORTOUT ,12/03/2013,000
07.44.15 JOB00104  IEC705I TAPE ON 0D41,234658,SL,COMP,FGRPJCLA,S89B.D,
07.45.53 JOB00104  IEC502E K 0D41,234658,SL,FGRPJCLA,S89B,FGRBG.EGBHGFR
07.45.53 JOB00104  TMS014  IEC502E K 0D41,234658,SL,FGRPJCLA,S89B,FGRBG
07.45.53 JOB00104  IEC501A M 0D41,PRIVAT,SL,COMP,FGRPJCLA,S89B.D,FGRBG.
07.45.56 JOB00104  IECTMS9 0D41,217808,FGRPJCLA,SORTOUT ,12/03/2013,000
07.46.26 JOB00104  IEC705I TAPE ON 0D41,217808,SL,COMP,FGRPJCLA,S89B.D,
07.48.13 JOB00104  IEC502E K 0D41,217808,SL,FGRPJCLA,S89B,FGRBG.EGBHGFR
07.48.13 JOB00104  TMS014  IEC502E K 0D41,217808,SL,FGRPJCLA,S89B,FGRBG
07.48.13 JOB00104  IEC501A M 0D41,PRIVAT,SL,COMP,FGRPJCLA,S89B.D,FGRBG.
07.48.17 JOB00104  IECTMS9 0D41,253159,FGRPJCLA,SORTOUT ,12/03/2013,000
07.48.55 JOB00104  IEC705I TAPE ON 0D41,253159,SL,COMP,FGRPJCLA,S89B.D,
07.50.43 JOB00104  IEC502E K 0D41,253159,SL,FGRPJCLA,S89B,FGRBG.EGBHGFR
07.50.43 JOB00104  TMS014  IEC502E K 0D41,253159,SL,FGRPJCLA,S89B,FGRBG
07.50.43 JOB00104  IEC501A M 0D41,PRIVAT,SL,COMP,FGRPJCLA,S89B.D,FGRBG.
07.50.49 JOB00104  IECTMS9 0D41,243910,FGRPJCLA,SORTOUT ,12/03/2013,000
07.51.14 JOB00104  IEC705I TAPE ON 0D41,243910,SL,COMP,FGRPJCLA,S89B.D,
07.51.24 JOB00104  IEC205I SORTOUT,FGRPJCLA,S89B,FILESEQ=1, COMPLETE VO
   063             DSN=FGRBG.EGBHGFR.SLISEA,VOLS=228305,234658,217808,2
   063             TOTALBLOCKS=864770                                 
07.54.00 JOB00104  IEF234E K 0EAE,319554,PVT,FGRPJCLA,S89B             
07.54.00 JOB00104  TMS014  IEF234E K 0EAE,319554,PVT,FGRPJCLA,S89B     
07.54.01 JOB00104  IEF234E K 0D41,243910,PVT,FGRPJCLA,S89B             
07.54.01 JOB00104  TMS014  IEF234E K 0D41,243910,PVT,FGRPJCLA,S89B     
07.54.01 JOB00104  -IEFACTRT D        SRTB        00   187K    .79    .



Re: Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 9:27 pm
by enrico-sorichetti
around 8 minutes of tape mounts

Re: Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 9:28 pm
by Akatsukami
Well, the JCL calls for the unnamed (!) data set on SORTOUT to be written to UNIT=VTS2,, and the IEF234E messages show that tapes are used, so despite your cleverly posting only a fraction of the JCL and sysouts, we can definitely say that tapes are involved. You failed to post any of the SORT messages as was requested, so nothing more can be said.

Re: Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 9:29 pm
by BillyBoyo
Well, I'd still like to see all the Sort message output. I see that your LRECL is pretty big, what is the average record size? I can also see something interfering with your SORTWK space allocations.

Re: Question about time of execution of a big sort

PostPosted: Mon Mar 11, 2013 9:36 pm
by Akatsukami
I think the VAM messages may be from SAMS:ALLOCATE.

Re: Question about time of execution of a big sort

PostPosted: Tue Mar 12, 2013 2:23 am
by steve-myers
This UNIT=VTS2 (and also the time between the K message and the M message for the next volume) smells like a virtual tape software product, though I won't guess whose product. I know CA is in this space, and I'm sure there are others. Some years ago I found myself the unwilling user of the CA virtual tape product, and I got my knuckles rapped when I tried to get around it to get timings for real tape. I thought at the time the capacity of a virtual "tape" volume was substantially less than the capacity of a real tape volume, but I never had the time (or interest) to validate this theory.

Billyboyo asked for documentation which you did not bother to provide. skolusu, when he checks in, will need this documentation to assist you.

Re: Question about time of execution of a big sort

PostPosted: Tue Mar 12, 2013 8:59 am
by dick scherrer
Hello,

From my experience (with CA V-Tape and others), virtual tape is excellent.

Suggest you talk with your storage management people for clarification.

Re: Question about time of execution of a big sort

PostPosted: Tue Mar 12, 2013 2:27 pm
by samb01
Hello,

here is the sort message output :

H RECORD TYPE IS V - DATA STARTS IN POSITION 5                                 
0 C5-K76982 C6-K90026 C7-K82419 C8-K67572 E9-K60824 C9-BASE   E5-K80744 E7-K7999
0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED         
1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS                 
0 EPRSDPH4.D       .S89B    , INPUT LRECL = 27994, BLKSIZE = 27998, TYPE = VB   
0 MAIN STORAGE = (31457280,31457280,31457280)                                   
0 MAIN STORAGE ABOVE 16MB = (31530992,31399920)                                 
0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,
0 OPTIONS: SIZE=31457280,MAXLIM=1048576,MINLIM=450560,EQUALS=Y,LIST=Y,ERET=ABEND
0 OPTIONS: VIO=Y,RESDNT=ALL ,SMF=NO   ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALO
0 OPTIONS: RESALL=0,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=N,OUTREL=Y,CKPT=N,COBEXIT=C
0 OPTIONS: TMAXLIM=8388608,ARESALL=131072,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,D
0 OPTIONS: VLSHRT=Y,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARM
0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSI
0 OPTIONS: NULLOUT=RC0                                                         
0 OPTIONS: DYNAPCT=10 ,MOWRK=Y                                                 
0 EXCP ACCESS METHOD USED FOR SORTOUT                                           
0 EXCP ACCESS METHOD USED FOR SORTIN                                           
0 DC 0 TC 24158578264 CS DSVRR KSZ 74 VSZ 74                                   

FSZ=24158578264 BC  IGN=0 E  AVG=14035 0  WSP=31463017 C  DYN=0 0             
D8-K58148 D4-K59452 EA-K79084 F1-K58148 E8-K79990                             
OUTPUT LRECL = 27994, BLKSIZE = 27998, TYPE = VB                             
INSERT 0, DELETE 0                                                           
RECORDS - IN: 67451219, OUT: 67451219                                         
NUMBER OF BYTES SORTED: 22823101372                                           
RECORDS SORTED - PROCESSED: 67451219, EXPECTED: 1725982                       
AVERAGE RECORD LENGTH - PROCESSED: 338, EXPECTED: 13997                       
TOTAL WORK DATA SET TRACKS ALLOCATED: 1766025 , TRACKS USED: 5460             
MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES                                 
MEMORY OBJECT USED AS WORK STORAGE = 0M BYTES                                 
HIPERSPACE STORAGE USED = 22330368K BYTES                                     
DATA SPACE STORAGE USED = 0K BYTES                                           
END OF DFSORT                                                                 
RECORD TYPE IS V - DATA STARTS IN POSITION 5                                 
C5-K76982 C6-K90026 C7-K82419 C8-K67572 E9-K60824 C9-BASE   E5-K80744 E7-K7999
BLOCKSET     SORT  TECHNIQUE SELECTED                                         
VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE 
- CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 07:57 ON SAT MAR 09, 20

   SORT FIELDS=(5,19,BI,A,25,50,BI,A,24,1,BI,A)                                 
H RECORD TYPE IS V - DATA STARTS IN POSITION 5                                 
0 C5-K76982 C6-K90026 C7-K82419 C8-K67572 E9-K60824 C9-BASE   E5-K80744 E7-K7999
0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED         
1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS                 



Is it enought ?