Page 1 of 2

SORT utility to eliminate the duplicate records

PostPosted: Sat Sep 22, 2007 12:47 pm
by Raja
Hi,

I am having a report like this...

*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
2 BBBB 12
3 CCCC 13
4 DDDD 14
3 CCCC 13

Emp info:
Total no of recs:7

I need to eliminate the duplicate records in the following order.''

*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
4 DDDD 14

Emp info:
Total no of recs:7

can any body help me?

regards,
Raja

Re: SORT utility

PostPosted: Sat Sep 22, 2007 3:33 pm
by CICS Guy
How much of the following is actual in the file?
*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
2 BBBB 12
3 CCCC 13
4 DDDD 14
3 CCCC 13

Emp info:
Total no of recs:7
And on the first duplicate dropped (2 BBBB 12), what is the key?

Re: SORT utility

PostPosted: Sun Sep 23, 2007 8:57 am
by dick scherrer
Hello,

You show a count of 7 in both sets of records. Neither has 7?

If you answer the questions asked, we may be able to offer suggestions.

Re: SORT utility

PostPosted: Mon Sep 24, 2007 1:48 pm
by Raja
CICS Guy wrote:How much of the following is actual in the file?
*********************************************
* EMP reports
*********************************************
No Name Age

5 EEEEE 15
1 AAAA 11
2 BBBB 12
3 CCCC 13
2 BBBB 12
3 CCCC 13
4 DDDD 14
3 CCCC 13

Emp info:
Total no of recs:7
And on the first duplicate dropped (2 BBBB 12), what is the key?
whole records are actual in th file.there is no key in the file.i have to avoid the duplicates in the whole line.

Re: SORT utility

PostPosted: Mon Sep 24, 2007 4:51 pm
by CICS Guy
Raja wrote:whole records are actual in th file.there is no key in the file.i have to avoid the duplicates in the whole line.
Then what is the lrecl and recfm?

Re: SORT utility to eliminate the duplicate records

PostPosted: Mon Sep 24, 2007 5:38 pm
by Raja
LRECL=120,RECFM=V

Re: SORT utility to eliminate the duplicate records

PostPosted: Mon Sep 24, 2007 6:23 pm
by CICS Guy
There is only one thing holding you back, and that is "How can you tell is a record is one you want to eliminate dups on or not?"
Take, for instance, the string of asterisk on either side of "* EMP reports"?
And the blank lines after "No Name Age" and "Emp info:"?

Re: SORT utility to eliminate the duplicate records

PostPosted: Mon Sep 24, 2007 9:48 pm
by Frank Yaeger
Raja,

This is rather tricky. Assuming that you want to remove duplicates only for the records with a numeric value in position 5 (e.g. 5, 1, 2, 3 and 4 in your example) and you want to keep all of the records in their original order, here's a DFSORT/ICETOOL job that will do what you asked for:

//S1    EXEC  PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG  DD SYSOUT=*
//IN DD DSN=...  input file (VB/120)
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//T2 DD DSN=&&T2,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//OUT DD DSN=...  output file (VB/120)
//TOOLIN   DD    *
COPY FROM(IN) TO(T1) USING(CTL1)
SORT FROM(T1) TO(T2) USING(CTL2)
SORT FROM(T2) TO(OUT) USING(CTL3)
//CTL1CNTL DD *
  INREC IFTHEN=(WHEN=INIT,
     BUILD=(1,4,5:SEQNUM,8,ZD,21:SEQNUM,8,ZD,29:5)),
        IFTHEN=(WHEN=(29,1,FS,EQ,NUM),
                OVERLAY=(5:C'00000001',21:SEQNUM,8,ZD)),
        IFTHEN=(WHEN=NONE,
                OVERLAY=(13:SEQNUM,8,ZD,
                         5:5,8,ZD,SUB,13,8,ZD,M11,LENGTH=8))
/*
//CTL2CNTL DD *
  OPTION EQUALS,VLSHRT
  SORT FIELDS=(5,16,BI,A,29,116,CH,A)
  SUM FIELDS=NONE
/*
//CTL3CNTL DD *
  OPTION EQUALS
  SORT FIELDS=(5,8,ZD,A,21,8,ZD,A)
  OUTREC BUILD=(1,4,5:29)
/*

Re: SORT utility to eliminate the duplicate records

PostPosted: Tue Oct 28, 2008 12:23 am
by jaganmoni
Hi,

I want to eliminate dulpicate records from my input file. The output file should contain only unique records with the input file order. In simple what I want exactly is 'I want to eliminate the duplicate records without sorting'.
My Input file is FB with LRECL=101. It should consider the entire record (i.e. 101 bytes as key) while eliminating the duplicates.

Re: SORT utility to eliminate the duplicate records

PostPosted: Tue Oct 28, 2008 1:30 am
by Frank Yaeger
Here's a DFSORT/ICETOOL job that will do what you asked for. In the future, please start a new topic rather than asking a different question in an existing topic.

//S1   EXEC  PGM=ICETOOL
//TOOLMSG   DD  SYSOUT=*
//DFSMSG    DD  SYSOUT=*
//IN DD DSN=...  input file (FB/101)
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(,PASS)
//OUT DD DSN=...  output file (FB/101)
//TOOLIN DD *
SELECT FROM(IN) TO(T1) ON(1,101,BI) FIRST USING(CTL1)
SORT FROM(T1) TO(OUT) USING(CTL2)
/*
//CTL1CNTL DD *
  INREC OVERLAY=(102:SEQNUM,8,ZD)
/*
//CTL2CNTL DD *
  SORT FIELDS=(102,8,ZD,A)
  OUTREC BUILD=(1,101)
/*