Splitting the records dynamicall with certain logic



IBM's flagship sort product DFSORT for sorting, merging, copying, data manipulation and reporting. Includes ICETOOL and ICEGENER

Splitting the records dynamicall with certain logic

Postby ragsara » Sun Mar 10, 2013 11:55 am

Hi ,

I have read "Split a file to n output files dynamically" from "Smart DFSORT Tricks" but i couldnt fit my below logic there, please find my below requirements :

1) If a file contains more than 2 Million records, then additonal records should get into the next file dynamically.

2) For eg) If a file conatins 7.5 Million records , File 1 should contain 2 Million, File 2 Should contain 2 Million , File 3 should conatin 2 Million and File 4 should conatin 0.5 Million records.

3) Input record counts will vary, it may even go to an extent of 21 Million records. (In this case there would be 11 output files)

4) A member present in a file can have many record types, for eg) A,B,C ..etc but record type A is compulsory for all members and it will be the first record type to occur for for all members. So in case of split, a member should not be present in two different files. For eg) While splitting the file for a member, all record types should be present in a single file.

Example of file lay out :

Rec01 Member01 A School information
Rec02 Member01 B College information
Rec03 Member01 C Job information
Rec04 Member02 A School information
Rec05 Member02 B College information
Rec06 Member03 A School information
Rec07 Member03 C Job information

Queries :

1) Based on record count is it possible to dynamically create any number of output files ? If we declare 12 output files (max) and if the record count is low, how to handle unused output files ?
2) How to handle the record types logic for the members while splittin the file ?

Is it possible to handle this requirement in JCL or should i go for a COBOL program ?

Your help is highly appreciated !!
ragsara
 
Posts: 23
Joined: Thu Mar 22, 2012 5:17 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Splitting the records dynamicall with certain logic

Postby NicC » Sun Mar 10, 2013 12:02 pm

JCL cannot do what you want - it does not manipulate data but telss the OS which programs you want to run and what resources those programs need. How do you propose to execute the COBOL program if not with JCL?

You looked at the DFSort tricks manual so I presume you have DFSort: if so, why not post in the DFSort part of the forum. Topic being moved.
The problem I have is that people can explain things quickly but I can only comprehend slowly.
Regards
Nic
NicC
Global moderator
 
Posts: 3025
Joined: Sun Jul 04, 2010 12:13 am
Location: Pushing up the daisies (almost)
Has thanked: 4 times
Been thanked: 136 times

Re: Splitting the records dynamicall with certain logic

Postby BillyBoyo » Sun Mar 10, 2013 4:17 pm

If "groups" of data need to be kept together, you need IFTHEN=(WHEN=GROUP and use the record sequence number (which your control cards would have to add) for selection into the correct output file.

The various OUTFIL "split" options will not work with variable numbers of records to be selected at run-time, so you'd need OUTFIL INCLUDE/OMIT.

If you "know" the number of records prior to the step, you could generate the JCL and either SUBmit or use the INTRDR with the generated JCL.

Otherwise, you can set a maximum number of output datasets, and remove any "empty" ones after the step.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Re: Splitting the records dynamicall with certain logic

Postby skolusu » Tue Mar 12, 2013 1:24 am

ragsara,

You can follow BillyBoyo's suggestion to split the records into multiple output files using WHEN=GROUP. You can remove empty dataset via IDCAMS print function and then delete them right away. Try with that approach and lets us know if you are having trouble with that. If you do need help then please provide us with the following.

1. Show us what you have tried so far.
2. What is the LRECL and RECFM of the input and output files?
3. How are groups of records identified?
4. Last but not least run the following job and show us the complete sysout which will help us determine the level of DFSORT you are running.

//STEP0100 EXEC PGM=SORT   
//SYSOUT   DD SYSOUT=*     
//SORTIN   DD *           
//SORTOUT  DD SYSOUT=*     
//SYSIN    DD *           
  OPTION COPY             
//*
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times


Return to DFSORT/ICETOOL/ICEGENER

 


  • Related topics
    Replies
    Views
    Last post