Page 1 of 1

Job in infinite loop

PostPosted: Sat Jan 17, 2009 9:13 am
by j2422tw
Hi,

It is a strange situation.

Today, our batch is in loop, the step is SORT.

When we try to CANCEL the looping job, is not work; CANCEL with asid is not work too.

Than we using the FORCE command, then the job is abend with S2F3, and no any sysout for the step.

The step JCL is follow:

//SORT01   EXEC PGM=SORT,REGION=4096K,COND=(0,LT)               
//SORTLIB  DD DSN=SYS1.SORTLIB,DISP=SHR                         
//SYSOUT   DD SYSOUT=*                                           
//SORTIN   DD DSN=&PREFIX..DEPRPTMF.SAMF,DISP=SHR               
//SORTOUT  DD DSN=&PREFIX..DEPD054.SAMF,                         
//            DISP=(NEW,CATLG,DELETE),                           
//            UNIT=SYSDA,SPACE=(TRK,(200,100),RLSE),             
//            DCB=(RECFM=FB,LRECL=600,BLKSIZE=27600)             
//SORTWK01 DD UNIT=SYSDA,SPACE=(CYL,(40,20),RLSE)               
//SORTWK02 DD UNIT=SYSDA,SPACE=(CYL,(40,20),RLSE)               
//SORTWK03 DD UNIT=SYSDA,SPACE=(CYL,(40,20),RLSE)               
//SORTWK04 DD UNIT=SYSDA,SPACE=(CYL,(40,20),RLSE)               
//SORTWK05 DD UNIT=SYSDA,SPACE=(CYL,(40,20),RLSE)               
//SYSIN    DD *                                                 
   SORT    FIELDS=(303,4,CH,A)                                   
   INCLUDE COND=((255,1,ZD,EQ,2,OR,255,1,ZD,EQ,3),AND,           
                 (256,4,BI,GT,0),AND,(149,3,CH,NE,C'TWD'),AND,   
                 (125,2,CH,EQ,C'00'))                           
   RECORD  TYPE=F,LENGTH=600                                     
//*                                                             



After FORCE, AP re-submit the job-step, it work fine, and the step sysout is :

ICE143I 0 BLOCKSET     SORT  TECHNIQUE SELECTED                                                                   
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE                             
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R5 - 09:05 ON SAT JAN 17, 2009 -                       
             SORT    FIELDS=(303,4,CH,A)                                                                           
             INCLUDE COND=((255,1,ZD,EQ,2,OR,255,1,ZD,EQ,3),AND,                                                   
                           (256,4,BI,GT,0),AND,(149,3,CH,NE,C'TWD'),AND,                                           
                           (125,2,CH,EQ,C'00'))                                                                   
             RECORD  TYPE=F,LENGTH=600                                                                             
ICE201I E RECORD TYPE IS F - DATA STARTS IN POSITION 1                                                             
ICE751I 0 C5-K26318 C6-K90007 C7-K90000 C8-K23476 E4-K90007 C9-BASE   E5-K31980 E7-K24705                         
ICE193I 0 ICEAM1 ENVIRONMENT IN EFFECT - ICEAM1 INSTALLATION MODULE SELECTED                                       
ICE088I 1 DEPDPR03.SORT01  .        , INPUT LRECL = 600, BLKSIZE = 27000, TYPE = FB                               
ICE093I 0 MAIN STORAGE = (MAX,32380611,32363297)                                                                   
ICE156I 0 MAIN STORAGE ABOVE 16MB = (32225979,32225979)                                                           
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256       
ICE128I 0 OPTIONS: SIZE=32380611,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=RC16 ,MSGDDN=SYSOUT             
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=SHORT,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N             ,ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,STIMER=Y,COBEXIT=COB2           
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=64                     
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE    ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX               
ICE235I 0 OPTIONS: NULLOUT=RC0                                                                               
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT                                                                 
ICE084I 0 BSAM ACCESS METHOD USED FOR SORTIN                                                                 
ICE750I 0 DC 7228079400 TC 0 CS DSVVV KSZ 4 VSZ 4                                                             
ICE752I 0 FSZ=12046799 RC  IGN=0 E  AVG=600 0  WSP=9388033 C  DYN=0 0                                         
ICE751I 1 DE-K24705 D5-K24705 D9-K24705 E8-K24705                                                             
ICE090I 0 OUTPUT LRECL = 600, BLKSIZE = 27600, TYPE = FB                                                     
ICE080I 0 IN MAIN STORAGE SORT                                                                               
ICE055I 0 INSERT 0, DELETE 12035750                                                                           
ICE054I 0 RECORDS - IN: 12046799, OUT: 11049                                                                 
ICE134I 0 NUMBER OF BYTES SORTED: 6629400                                                                     
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 3000 , TRACKS USED: 0                                         
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES                                                               
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES                                                                 
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES                                                                 
ICE052I 0 END OF DFSORT                                                                                       


I had check the SMF record, and find when the time of sort is start, the SMF type 14 and SMF type 42 for SORTIN DATASET is been record, but no any SORTOUT SMF record.

It look like when the SORT is start, reading the SORTIN, then the job is looping.

I have no idea to explain this situation.

Hope anyone can help me.

Thank in advance.

Re: DFSORT in infinite loop

PostPosted: Sat Jan 17, 2009 10:33 am
by dick scherrer
Hello,

How did you determine this step was in a "loop"? How rapidly was the amount of cpu used increasing?

How long was the step allowed to "loop"? How long has it been since this dataset (DSN=&PREFIX..DEPRPTMF.SAMF) was used by some job?

It is possible that the job was waiting for the file to be recalled because it had been migrated from dasd. If the file was stored on tape/cart, the job may have been waiting for a volume to be mounted.

Suggest you closely review the system log for any messages related to this job/step.

Re: DFSORT in infinite loop

PostPosted: Sat Jan 17, 2009 11:15 am
by j2422tw
Dear Dick:

Thanks for you reply.

The abend message of JESMSGLG is follow:

06.05.46 JOB31290  -DEPDPR03          IND058A     00    661    .00    .00    .07   3471   0      0      0      0     0    91 
06.05.57 JOB31290  -DEPDPR03          IND058B     00    647    .01    .00    .19   8149   0      0      0      0     0    92 
06.05.58 JOB31290  -DEPDPR03          DELDEF      00     13    .00    .00    .00     35   0      0      0      0     0    93 
09.03.44 JOB31290  ICH70001I TWSC     LAST ACCESS AT 09:02:40 ON SATURDAY, JANUARY 17, 2009                                   
09.03.45 JOB31290  $HASP373 DEPDPR03 STARTED - INIT 2    - CLASS A - SYS F102                                                 
09.03.45 JOB31290  IEF450I DEPDPR03 SORT01 - ABEND=S2F3 U0000 REASON=00000000                                                 
09.03.46 JOB31290  -DEPDPR03                   *S2F3      0    .00    .00 177.79      0   0      0      0      0     0     0 
09.03.46 JOB31290  IEF453I DEPDPR03 - JOB FAILED - JCL ERROR                                                                 
09.03.46 JOB31290  -DEPDPR03 ENDED.  NAME-                     TOTAL CPU TIME=   .33  TOTAL ELAPSED TIME=183.79               


When issue the FORCE command, the job is re-enter the input queue,

and abend when start to execute the job.

The step using CPU is very high, and no any SIO and EXCP.

It is daily job, so no any HSM action needed, and no any one using the SORTIN dataset.

By the way, the SORTIN dataset is extend format and compressed, SORTOUT
is normal QSAM.

Thanks and regards,

Jerry

Re: DFSORT in infinite loop

PostPosted: Sat Jan 17, 2009 12:10 pm
by dick scherrer
Hi Jerry,

TOTAL CPU TIME= .33
This does not appear to be very much cpu time - especially not since the input file is over 12 million records. How much cpu time does this step usually require?

You might want to talk with your storage management people and see if they were aware of any problems.

Re: DFSORT in infinite loop

PostPosted: Sat Jan 17, 2009 9:24 pm
by Frank Yaeger
Jerry,

I would question your assumption that DFSORT is looping. However, this board is not the right place to get help for this kind of situation. If you're serious about pursuing it, then you need to open a problem with IBM service so they can collect the appropriate information and help you resolve it.