Split a file into x outfiles, max 20k in each



IBM's flagship sort product DFSORT for sorting, merging, copying, data manipulation and reporting. Includes ICETOOL and ICEGENER

Split a file into x outfiles, max 20k in each

Postby LasseH » Fri Oct 05, 2012 2:33 pm

Read the DFSORT tricks but...

Problem:
Inputfile up to 200k rows (fb10) can also be 100 rows

Want that file to be split into x outfiles with maximum 20k rows in each.

(The "tricks" shows how to split into x files (x=hardcoded amount))
Can You help me with this?

//Lasse
LasseH
 
Posts: 70
Joined: Mon Nov 08, 2010 2:51 pm
Has thanked: 7 times
Been thanked: 1 time

Re: Split a file into x outfiles, max 20k in each

Postby skolusu » Fri Oct 05, 2012 9:03 pm

Lasse,

20k each file with an lrecl of 10 would only have (20 X 1024 = 20480 /10) = 2048 Records. You would be creating 100 datasets to accommodate all the rows. You can define all the 100 datasets and use SPLIT1R=2048 to split the records. If you are concerned about the number of empty datasets , you can run an IDCAMS step to check for an empty file and delete all the empty files in a single step.
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times

Re: Split a file into x outfiles, max 20k in each

Postby Akatsukami » Fri Oct 05, 2012 9:59 pm

Is not the TS looking to create output data sets with a maximum of 20K records (which he miscalls "rows") rather than 20K bytes?
"You have sat too long for any good you have been doing lately ... Depart, I say; and let us have done with you. In the name of God, go!" -- what I say to a junior programmer at least once a day
User avatar
Akatsukami
Global moderator
 
Posts: 1058
Joined: Sat Oct 16, 2010 2:31 am
Location: Bloomington, IL
Has thanked: 6 times
Been thanked: 51 times

Re: Split a file into x outfiles, max 20k in each

Postby LasseH » Mon Oct 08, 2012 11:27 am

Sorry for the miscall, yes, I ment 20k records.
(200.000 records will end up in 10 outputfiles, with 20.000 records each, and 21.000 ends up in 2 files)
LasseH
 
Posts: 70
Joined: Mon Nov 08, 2010 2:51 pm
Has thanked: 7 times
Been thanked: 1 time

Re: Split a file into x outfiles, max 20k in each

Postby skolusu » Mon Oct 08, 2012 10:04 pm

LasseH wrote:Sorry for the miscall, yes, I ment 20k records.
(200.000 records will end up in 10 outputfiles, with 20.000 records each, and 21.000 ends up in 2 files)


It is still not clear as to how you want to split. Do you dynamically allocate the output files each with 20k records?

ex:
100,000 records  = SPLIT files =5 (5 x 20,000 )
200,000 records  = SPLIT files =5 (10 x 20,000 )
250,000 records  = SPLIT files =5 (12 x 20,000  + 1 X 10,000)
...


If that is indeed the case as I mentioned earlier define the max outfil files (100) and then in the next step use IDCAMS to delete all the empty datasets which is the simplest task.

Another solution is to read the file, get the count of records and then dynamically generate the ICETOOL JCL and submit it via INTRDR.
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times

Re: Split a file into x outfiles, max 20k in each

Postby LasseH » Tue Oct 09, 2012 11:25 am

Thanks, I go with the "create a lot, delete after"
//Lasse
LasseH
 
Posts: 70
Joined: Mon Nov 08, 2010 2:51 pm
Has thanked: 7 times
Been thanked: 1 time


Return to DFSORT/ICETOOL/ICEGENER

 


  • Related topics
    Replies
    Views
    Last post