IMPROVE Performance of a DFSORT process



IBM's flagship sort product DFSORT for sorting, merging, copying, data manipulation and reporting. Includes ICETOOL and ICEGENER

IMPROVE Performance of a DFSORT process

Postby javivi » Wed Jun 20, 2012 4:22 pm

Hi I have a BIG file (more than 70 millions of records of 389 LRECL) that I'm sorting with the next code:

//SORT005  EXEC PGM=SORT                                                     
//SYSOUT   DD SYSOUT=A                                                       
//SORTIN   DD DSN=FICHERO1,DISP=SHR                                         
//SALIDA1  DD DSN=FICHERO1.SORT,                                             
//            DISP=(,CATLG,CATLG),                                           
//            UNIT=(SYSALLDA,5),SPACE=(CYL,(150,100),RLSE)                         
//SALIDA2  DD DSN=DEBN.LSQ.ZJCFPHOP.SORT005.OPERACIO,                       
//            DISP=(,CATLG,CATLG),                                           
//            UNIT=(SYSALLDA,5),SPACE=(CYL,(150,100),RLSE)                     
//SYSIN    DD *                                                             
  SORT FIELDS=(371,8,A,92,10,A,37,5,A,42,5,A,47,3,A),                       
               FORMAT=BI,SIZE=E90000                                         
  INCLUDE COND=(50,2,SS,EQ,C'  ,X , X')                                     
  OUTFIL FNAMES=SALIDA1,                                                     
         INCLUDE=(31,1,CH,EQ,C'X'),BUILD=(1,389)                             
  OUTFIL FNAMES=SALIDA2,                                                     
         INCLUDE=(31,1,CH,NE,C'X'),BUILD=(1,389)                             
  END 



It work very well but the problem is that the sort take about 30 minutes to end.

Any Idea to optimize the sort in order to improve the efficiency? it is possible?

Here the dfsort log:

ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 07:59 ON WED JUN 20, 2012 -
SORT FIELDS=(371,8,A,92,10,A,37,5,A,42,5,A,47,3,A),
FORMAT=BI,SIZE=E90000
INCLUDE COND=(50,2,SS,EQ,C' ,X , X')
OUTFIL FNAMES=SALIDA1,
INCLUDE=(31,1,CH,EQ,C'X'),BUILD=(1,389)
OUTFIL FNAMES=SALIDA2,
INCLUDE=(31,1,CH,NE,C'X'),BUILD=(1,389)
END
ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE858I 0 LA=1712768, DA=61769, AA=1650999, BA=4093, CP=0, TA=4141
ICE859I 0 LB=9756, DB=2044, AB=7712, BB=0, CP=3, RS=8, TB=31
ICE860I 0 F=YN, P=2, M=N, B=2048
ICE992I 0 RA 0 WR 0 TR 1
ICE751I 0 C5-K90025 C6-K90025 C7-K54603 C8-K62201 E4-K51707 C9-BASE E5-K62201 E6-K51707 C4-K62201 E7-K62201
ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
ICE252I 1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
ICE088I 1 ZJCFPHOP.SORT005 . , INPUT LRECL = 389, BLKSIZE = 27619, TYPE = FB
ICE093I 0 MAIN STORAGE = (MAX,63250892,63222220)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (63164860,63164860)
ICE127I 0 OPTIONS: OVFLO=RC16,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC4 ,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=4000
ICE128I 0 OPTIONS: SIZE=63250892,MAXLIM=2097152,MINLIM=524288,EQUALS=N,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(SYSALLDA,003),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=12288,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=20971520,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=256
ICE132I 0 OPTIONS: VLSHRT=Y,ZDPRINT=Y,IEXIT=Y,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=0 ,ODMAXBF=2097152,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=0
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE084I 0 BSAM ACCESS METHOD USED FOR SORTIN
ICE750I 0 DC 27368415147 TC 0 CS DSVVV KSZ 31 VSZ 31
ICE752I 0 FSZ=70355823 RC IGN=90000 E AVG=392 0 WSP=35821008 C DYN=419616 53216
ICE915I 0 MOFSZ=0,MOSZ=0,MOSYS=0(1),MOSTG=86833,MEML=17592186040320(3)
ICE916I 0 MOFR=0003,MOVR=VV
ICE996I 0 ESM=29147136,ESO=8744140,ESR=0,ESP=4096,ESS=16384,CES=29147136,HSZ=524288
ICE997I 0 HWSP=16833180,HMAX=3538944,HES=3538944,ASV=22229440,EQ=I9,HN=1
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=3538944,CMAX=524288,HU=99,BUN=53216,MD=M3,DU=90,DR=0,HN=1
ICE880I 0 QP=128 QA=1091 HI=4591 LI=4338 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=1
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=3014656,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=2
ICE880I 0 QP=128 QA=1237 HI=4337 LI=4192 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=2
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=2490368,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=3
ICE880I 0 QP=128 QA=2603 HI=4191 LI=2826 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=3
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=1966080,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=4
ICE880I 0 QP=128 QA=4899 HI=2825 LI=530 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=4
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=1441792,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=5
ICE880I 0 QP=128 QA=5026 HI=5053 LI=403 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=5
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=917504,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=6
ICE880I 0 QP=128 QA=5153 HI=5055 LI=276 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=6
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=393216,CMAX=393216,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=7

ICE880I 0 QP=96 QA=5426 HI=275 LI=3 MI=5428 TZ=393216 N1=393216 N2=393216 SZ=97 HN=7
ICE889I 0 CT=MAX , SB=3, L=0, D=0000, CCW=1MAM
ICE901I 0 W 02PP17 01PP17 03PP13
ICE902I 0 O I PP10
ICE897I 0 AVAL=14762821,BVAL=21747845,ENQT=0,ASV=21747845,HSZ=0,HM=M1,HN=10
ICE231I 0 STORAGE USED FOR OUTFIL : BELOW 16M = 31744, ABOVE 16M = 4240384
ICE855I 0 SALIDA1 : TX=N, R=J, L=J, B=D, BL=3, BR=9, DCT=32, VS=N, RU=X, SB=8
ICE210I 0 SALIDA1 : BSAM USED, LRECL = 389, BLKSIZE = 32676, TYPE = FB
ICE855I 0 SALIDA2 : TX=N, R=J, L=J, B=D, BL=3, BR=9, DCT=32, VS=N, RU=X, SB=8
ICE210I 0 SALIDA2 : BSAM USED, LRECL = 389, BLKSIZE = 32676, TYPE = FB
ICE751I 1 DE-K61785 D5-K62201 D3-BASE D7-BASE CB-K64631 E8-K61438
ICE900I 0 CON=1,MUV=1,VOL=30,ENU=0,SBK=0,SRC=0,VEM=0
ICE999I 0 PWK=2 PSP=419616 SWK=0 SSP=0 TWK=0 TSP=0 RWK=0 RSP=0 AWK=2 AWP=419616
ICE055I 0 INSERT 0, DELETE 1
ICE054I 0 RECORDS - IN: 70355823, OUT: 70355822
ICE227I 0 SALIDA1 : DELETED = 70238819, REPORT = 0, DATA = 117003
ICE228I 0 SALIDA1 : TOTAL IN = 70355822, TOTAL OUT = 117003
ICE227I 0 SALIDA2 : DELETED = 117003, REPORT = 0, DATA = 70238819
ICE228I 0 SALIDA2 : TOTAL IN = 70355822, TOTAL OUT = 70238819
ICE134I 0 NUMBER OF BYTES SORTED: 27368414758
ICE253I 0 RECORDS SORTED - PROCESSED: 70355822, EXPECTED: 70355823
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 419640 , TRACKS USED: 253905
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 14858260K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE891I 1 63213368 WMAIN, 8848 CMAIN, MAX CALLOC, N SCN, A BA, 0 AZ, 1 BZ, NN QC, 0 CZ, 0 DZ, 0 PLE
ICE892I 1 389 RIN 27619 BLI 0 BLO 392 RUN 53216 BUN 2817 CPU 00 CVC
ICE893I 1 255 XIN 14 WIN 136392 GIN NDEY PFP42 BJ0 CMB00 CIX UPTH LMD VS RUX
ICE894I 0 178 STR 178 MOR 1028 IPB 0 OPB 16927 CYL O MN
ICE881I 0 EQ=I9 DX=0 D2=0 D3=127 D4=1 AS=0 SA=0 SB=0 SC=0 HN=1
ICE885I 0 DAT 00 DSR 0003 BINS 0 BSZ 0 RCP 0 AJC 0 RLC 0 DUNIT 0
ICE895I 0 296 MUNIT 42 SUNIT 31 OUNIT
ICE896I 0 34825 SET 6681620 DEXTOT 521242 BLK 70355823 CSZ 33028 WE
ICE804I 1 SORTWK03 EXCP COUNT: 1
ICE804I 1 SORTWK02 EXCP COUNT: 16526
ICE804I 1 SORTWK01 EXCP COUNT: 16501
ICE899I 0 HSR=2735,HSW=2885,HRE=40276,HWE=4046,HRP=523588,HWP=523588,HWM=524242,HNM=1
ICE052I 0 END OF DFSORT


Thanks.javivi

Posts: 30
Joined: Fri Jan 21, 2011 2:53 pm
Private message
javivi
 
Posts: 47
Joined: Fri Jan 21, 2011 2:53 pm
Has thanked: 0 time
Been thanked: 0 time

Re: IMPROVE Performance of a DFSORT process

Postby NicC » Wed Jun 20, 2012 10:41 pm

Your last 3 sort fields are contiguous - why not make them into one (sort may do that already anyway - not sure about that) but 30 minutes isn't all that long.
The problem I have is that people can explain things quickly but I can only comprehend slowly.
Regards
Nic
NicC
Global moderator
 
Posts: 3025
Joined: Sun Jul 04, 2010 12:13 am
Location: Pushing up the daisies (almost)
Has thanked: 4 times
Been thanked: 136 times

Re: IMPROVE Performance of a DFSORT process

Postby skolusu » Thu Jun 21, 2012 12:56 am

Javivi,

Our performance expert(Dave Betten) suggests that you increase the ODMAXBF to 6M (currently you have it at 2M) and see if it improves your elapsed time. Add the following to your JCL
//DFSPARM   DD *
  OPTION ODMAXBF=6M
//*
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times

Re: IMPROVE Performance of a DFSORT process

Postby BillyBoyo » Thu Jun 21, 2012 2:32 am

You have SIZE=E90000, you could check what that means with your installation options. If not used (so DFSORT able to reasonably estimate iput file size) you could remove it. If it is being used, it should be a much closer approximation to the filesize.

You have a BUILD on each of the OUTFILs. You have not extended/shortened the records, so I can't see that they are necessary. Perhaps doing it 70,000,0000 times is something noticeable? If not needed, remove.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times


Return to DFSORT/ICETOOL/ICEGENER

 


  • Related topics
    Replies
    Views
    Last post