Splitting a very large file



Compuware's data management products: File-AID for IMS, File-AID/MVS, File-AID for DB2 and DBA-XPERT for DB2

Splitting a very large file

Postby socker_dad » Thu Dec 24, 2020 10:35 pm

I have need to split a very large file of roughly 17.8 million records into many "sub-files" of about 50,000 records each. It's for tax season and our mainframe isn't allowed to run unfettered.

I know I can use steps of successive "$$DD01 SPACE IN=50000 $$DD01 COPY OUT=50000", but the calculations indicate that this will have to be repeated about 355 times. Ouch.

Is there another way to get File Aid to split more efficiently? Would the be better in DFSORT? Or should I just pop out a quick and dirty program?
socker_dad
 
Posts: 9
Joined: Tue Sep 22, 2020 5:37 am
Has thanked: 0 time
Been thanked: 0 time

Re: Splitting a very large file

Postby NicC » Fri Dec 25, 2020 2:42 pm

Look up the DFSort Tricks publication which gives several ways to split datasets.
The problem I have is that people can explain things quickly but I can only comprehend slowly.
Regards
Nic
NicC
Global moderator
 
Posts: 3026
Joined: Sun Jul 04, 2010 12:13 am
Location: Pushing up the daisies (almost)
Has thanked: 4 times
Been thanked: 136 times

Re: Splitting a very large file

Postby chaat » Thu Jan 28, 2021 12:09 am

I've written an E15 sort exit to count the number of records, it then passes this count to an E35 sort exit which splits the sort out using Dynamic allocation to multiple output files. This number of files is controlled by an input split control file.

If you are interested in the source code, send me a private message and include your email address. I will send you the two COBOL Sort Exit programs.

Chuck Haatvedt
chaat
 
Posts: 26
Joined: Sun Aug 16, 2009 11:07 pm
Location: St. Cloud, Minnesota
Has thanked: 0 time
Been thanked: 1 time


Return to File-AID

 


  • Related topics
    Replies
    Views
    Last post