SORT utility to eliminate the duplicate records



JES, JES2, JCL utilities, IDCAMS, Compile & Run JCLs, PROCs etc...

SORT utility to eliminate the duplicate records

Postby Raja » Sat Sep 22, 2007 12:47 pm

Hi,

I am having a report like this...

*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
2 BBBB 12
3 CCCC 13
4 DDDD 14
3 CCCC 13

Emp info:
Total no of recs:7

I need to eliminate the duplicate records in the following order.''

*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
4 DDDD 14

Emp info:
Total no of recs:7

can any body help me?

regards,
Raja
Raja
 
Posts: 7
Joined: Fri Sep 21, 2007 8:01 pm
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility

Postby CICS Guy » Sat Sep 22, 2007 3:33 pm

How much of the following is actual in the file?
*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
2 BBBB 12
3 CCCC 13
4 DDDD 14
3 CCCC 13

Emp info:
Total no of recs:7
And on the first duplicate dropped (2 BBBB 12), what is the key?
CICS Guy
 
Posts: 246
Joined: Wed Jun 20, 2007 4:08 am
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility

Postby dick scherrer » Sun Sep 23, 2007 8:57 am

Hello,

You show a count of 7 in both sets of records. Neither has 7?

If you answer the questions asked, we may be able to offer suggestions.
Hope this helps,
d.sch.
User avatar
dick scherrer
Global moderator
 
Posts: 6268
Joined: Sat Jun 09, 2007 8:58 am
Has thanked: 3 times
Been thanked: 93 times

Re: SORT utility

Postby Raja » Mon Sep 24, 2007 1:48 pm

CICS Guy wrote:How much of the following is actual in the file?
*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
2 BBBB 12
3 CCCC 13
4 DDDD 14
3 CCCC 13

Emp info:
Total no of recs:7
And on the first duplicate dropped (2 BBBB 12), what is the key?
whole records are actual in th file.there is no key in the file.i have to avoid the duplicates in the whole line.
Raja
 
Posts: 7
Joined: Fri Sep 21, 2007 8:01 pm
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility

Postby CICS Guy » Mon Sep 24, 2007 4:51 pm

Raja wrote:whole records are actual in th file.there is no key in the file.i have to avoid the duplicates in the whole line.
Then what is the lrecl and recfm?
CICS Guy
 
Posts: 246
Joined: Wed Jun 20, 2007 4:08 am
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility to eliminate the duplicate records

Postby Raja » Mon Sep 24, 2007 5:38 pm

LRECL=120,RECFM=V
Raja
 
Posts: 7
Joined: Fri Sep 21, 2007 8:01 pm
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility to eliminate the duplicate records

Postby CICS Guy » Mon Sep 24, 2007 6:23 pm

There is only one thing holding you back, and that is "How can you tell is a record is one you want to eliminate dups on or not?"
Take, for instance, the string of asterisk on either side of "* EMP reports"?
And the blank lines after "No Name Age" and "Emp info:"?
CICS Guy
 
Posts: 246
Joined: Wed Jun 20, 2007 4:08 am
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility to eliminate the duplicate records

Postby Frank Yaeger » Mon Sep 24, 2007 9:48 pm

Raja,

This is rather tricky. Assuming that you want to remove duplicates only for the records with a numeric value in position 5 (e.g. 5, 1, 2, 3 and 4 in your example) and you want to keep all of the records in their original order, here's a DFSORT/ICETOOL job that will do what you asked for:

//S1    EXEC  PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG  DD SYSOUT=*
//IN DD DSN=...  input file (VB/120)
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//T2 DD DSN=&&T2,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//OUT DD DSN=...  output file (VB/120)
//TOOLIN   DD    *
COPY FROM(IN) TO(T1) USING(CTL1)
SORT FROM(T1) TO(T2) USING(CTL2)
SORT FROM(T2) TO(OUT) USING(CTL3)
//CTL1CNTL DD *
  INREC IFTHEN=(WHEN=INIT,
     BUILD=(1,4,5:SEQNUM,8,ZD,21:SEQNUM,8,ZD,29:5)),
        IFTHEN=(WHEN=(29,1,FS,EQ,NUM),
                OVERLAY=(5:C'00000001',21:SEQNUM,8,ZD)),
        IFTHEN=(WHEN=NONE,
                OVERLAY=(13:SEQNUM,8,ZD,
                         5:5,8,ZD,SUB,13,8,ZD,M11,LENGTH=8))
/*
//CTL2CNTL DD *
  OPTION EQUALS,VLSHRT
  SORT FIELDS=(5,16,BI,A,29,116,CH,A)
  SUM FIELDS=NONE
/*
//CTL3CNTL DD *
  OPTION EQUALS
  SORT FIELDS=(5,8,ZD,A,21,8,ZD,A)
  OUTREC BUILD=(1,4,5:29)
/*
Frank Yaeger - DFSORT Development Team (IBM) - yaeger@us.ibm.com
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
=> DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort
User avatar
Frank Yaeger
Global moderator
 
Posts: 1079
Joined: Sat Jun 09, 2007 8:44 pm
Has thanked: 0 time
Been thanked: 15 times

Re: SORT utility to eliminate the duplicate records

Postby jaganmoni » Tue Oct 28, 2008 12:23 am

Hi,

I want to eliminate dulpicate records from my input file. The output file should contain only unique records with the input file order. In simple what I want exactly is 'I want to eliminate the duplicate records without sorting'.
My Input file is FB with LRECL=101. It should consider the entire record (i.e. 101 bytes as key) while eliminating the duplicates.
jaganmoni
 
Posts: 6
Joined: Tue Oct 28, 2008 12:13 am
Has thanked: 0 time
Been thanked: 0 time

Re: SORT utility to eliminate the duplicate records

Postby Frank Yaeger » Tue Oct 28, 2008 1:30 am

Here's a DFSORT/ICETOOL job that will do what you asked for. In the future, please start a new topic rather than asking a different question in an existing topic.

//S1   EXEC  PGM=ICETOOL
//TOOLMSG   DD  SYSOUT=*
//DFSMSG    DD  SYSOUT=*
//IN DD DSN=...  input file (FB/101)
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//OUT DD DSN=...  output file (FB/101)
//TOOLIN DD *
SELECT FROM(IN) TO(T1) ON(1,101,BI) FIRST USING(CTL1)
SORT FROM(T1) TO(OUT) USING(CTL2)
/*
//CTL1CNTL DD *
  INREC OVERLAY=(102:SEQNUM,8,ZD)
/*
//CTL2CNTL DD *
  SORT FIELDS=(102,8,ZD,A)
  OUTREC BUILD=(1,101)
/*
Frank Yaeger - DFSORT Development Team (IBM) - yaeger@us.ibm.com
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
=> DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort
User avatar
Frank Yaeger
Global moderator
 
Posts: 1079
Joined: Sat Jun 09, 2007 8:44 pm
Has thanked: 0 time
Been thanked: 15 times

Next

Return to JCL

 


  • Related topics
    Replies
    Views
    Last post