INCLUDE lines of variable multiline message IEC205I



IBM's flagship sort product DFSORT for sorting, merging, copying, data manipulation and reporting. Includes ICETOOL and ICEGENER

INCLUDE lines of variable multiline message IEC205I

Postby claudioz » Tue Jan 08, 2013 4:26 pm

Hi all,
is there a way to filter/reformat input (e.g. syslog) with multiline messages as if they were single lines?

for example, IEC205I has variable length messages (I have some ones with two and others with a set of 5 lines); is there a way to interpret the "continuation code" ("126" and "791" in the example) that is added at the end of the first line and on the beginning of the others?
I found ways to manage multiline messages if they have a fixed number of lines of if the lines are grouped by an header and a trailer but still nothing in this case.
Thanks a lot!

Follows an example of my input:
 J0014673 00000090  IEC205I OUTFIL01,GCCJ0101,S020,FILESEQ=1, COMPLETE VOLUME LIST, 674       
      674 00000090  DSN=IBN0P.GCCF0001.STORICO.DN.G1695V00,VOLS=123731,TOTALBLOCKS=1768       
 J0014668 00000290  -S010     P001        00    384     59       .10       .00     .1         
                       128882  WBATCH       0     0     0     0                               
 J0230314 00000090  IEC205I OUT3,FC165104,S011,FILESEQ=1, COMPLETE VOLUME LIST, 126     
      126 00000090  DSN=TMP.T3592.CRY.PR13.DS,VOLS=600919,                       
      126 00000090  LISTED VOL(S) HAVE BEEN DATA ENCRYPTED,KL1CD:L,KL2CD:L,             
      126 00000090  KL1=MY.TKLM.TAPES,KL2=MY.TKLM.TAPES,     
      126 00000090  TOTALBLOCKS=1          *HAS                                         
 J0230445 00000090  IEC205I SYSCPY1,NSEJBK99,P010,FILESEQ=7, COMPLETE VOLUME LIST, 791 
      791 00000090  DSN=D2SBKP0P.BNSE0.USFFPR.D130107.RF38WG.DN,VOLS=127050,           
      791 00000090  TOTALBLOCKS=356          *H                                         
 J0014673 00000281  IEF404I GCCJ0101 - ENDED - TIME=06.15.23                                 
 J0014673 00000290  -GCCJ0101 ENDED.  NAME-GEST.I-O             TOTAL TCB CPU TIME=           
                    .00 TOTAL ELAPSED TIME=    .0                                             


ps I just tried with "IFTHEN=(WHEN=GROUP ... RECORDS=5", then RESIZE ICETOOL command, as in "Create large records from small records" from "Smart DFSORT Tricks", but this is not the correct solution in my case. It may works, as I tested, but it is not the solution for this type of messages.
claudioz
 
Posts: 4
Joined: Tue Jan 08, 2013 1:40 pm
Has thanked: 0 time
Been thanked: 0 time

Re: INCLUDE lines of variable multiline message IEC205I

Postby claudioz » Tue Jan 08, 2013 5:38 pm

I am just adding a note: a partial work is done with the use of BEGIN END on the first column field of the syslog, that I was not looking at! It delimits the message.
OK, I just understand how to isolate relevant messages from the others.
Anyway, I am not understanding how to read one message as if it were "one line", i.e. with the usual column or field position of dfsort.

In the syslog I see that you can watch at 'M' for the start of a multiline message, and the 'E' for the end...
S                                                              2752  WBATCH       0     0     0     0                       
M 0020000 MVSP0    2013007 12:16:23.21 J0230445 00000090  IEC205I SYSCPY1,NSEJBK99,P010,FILESEQ=7, COMPLETE VOLUME LIST, 791
D                                           791 00000090  DSN=D2SBKP0P.BNSE0.USFFPR.D130107.RF38WG.DN,VOLS=127050,           
E                                           791 00000090  TOTALBLOCKS=356          *H                                       
N 0020000 MVSP0    2013007 12:16:23.24 S0140163 00000281  IGD103I SMS ALLOCATED TO DDNAME SYS12164                           
claudioz
 
Posts: 4
Joined: Tue Jan 08, 2013 1:40 pm
Has thanked: 0 time
Been thanked: 0 time

Re: INCLUDE lines of variable multiline message IEC205I

Postby dick scherrer » Tue Jan 08, 2013 10:26 pm

Hello and welcome to the forum,

You will need to "stack" (read about PUSH) the records and then format and write the new output as you want it.

You will need some way to define a "group". Is the number of lines per group always the same?

It will help if you show the desired output from the input you posted - and additional samples as appropriate.
Hope this helps,
d.sch.
User avatar
dick scherrer
Global moderator
 
Posts: 6268
Joined: Sat Jun 09, 2007 8:58 am
Has thanked: 3 times
Been thanked: 93 times

Re: INCLUDE lines of variable multiline message IEC205I

Postby skolusu » Tue Jan 08, 2013 11:04 pm

claudioz wrote:Hi all,
is there a way to filter/reformat input (e.g. syslog) with multiline messages as if they were single lines?

for example, IEC205I has variable length messages (I have some ones with two and others with a set of 5 lines); is there a way to interpret the "continuation code" ("126" and "791" in the example) that is added at the end of the first line and on the beginning of the others?
I found ways to manage multiline messages if they have a fixed number of lines of if the lines are grouped by an header and a trailer but still nothing in this case.
Thanks a lot!
ps I just tried with "IFTHEN=(WHEN=GROUP ... RECORDS=5", then RESIZE ICETOOL command, as in "Create large records from small records" from "Smart DFSORT Tricks", but this is not the correct solution in my case. It may works, as I tested, but it is not the solution for this type of messages.


claudioz,

It is quite easy. if you look at the syslog , you will notice that the continuation messages doesn't have a time stamp value and it is spaces. So use that value for Group to Push the string IEC205I. Use the following DFSORT JCL which will give you the desired results. I assumed your input is FBA and LRECL=133. The timestamp field starts at pos 27 for a length of 11 bytes and the IEC205I message starts at pos 58 for a length of 8 bytes.
//STEP0100 EXEC PGM=SORT                                             
//SYSOUT   DD SYSOUT=*                                               
//SORTIN   DD DISP=SHR,DSN=Your input FBA 133 byte SYSLOG dataset
//SORTOUT  DD SYSOUT=*                                               
//SYSIN    DD *                                                     
  OPTION COPY                                                       
  INREC IFTHEN=(WHEN=GROUP,BEGIN=(27,11,CH,NE,C' '),PUSH=(134:58,8))
  OUTFIL BUILD=(1,133),INCLUDE=(134,8,CH,EQ,C'IEC205I')             
//*   
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times

Re: INCLUDE lines of variable multiline message IEC205I

Postby claudioz » Wed Jan 09, 2013 5:38 pm

Well... as in my previous post, I'll found how to group that records, even if the number of lines per group is not fixed, thanks to the mark in the first column.
I've just used WHEN=GROUP with begin/end limits, instead of the number of records
BEGIN=(1,1,CH,EQ,C'M',AND,59,7,CH,EQ,C'IEC205I') and END=(1,1,CH,EQ,C'E'),
and identifing interesting records with PUSH=(134:ID=8,142:SEQ=1))

At this point, I need to rewrite all multiline IEC205I messages as if they were single line messages (with a big LRECL), regardless of the number of rows they are composed of (ok, I will assume 5 as the maximum).

So, as an example, consider the following as a compacted syslog (a multi-line log):
----+----1----+----2----+----3----+----4----+----5
N 11:01:55.22 J0227540 I123421 FASDFSA
M 11:01:58.14 J0227280  IEC205I SYSUT2
E                  929  DSN=TMP.FC1,TO
N 11:02:55.22 J0227542 I123421 FASDFSA
M 11:02:58.14 J0227290  IEC205I SYSUT2
D                  929  DSN=TMP.FC2   
D                  929  LISTED VOL(S)
D                  929  KL1=TKLM.TAPE0
E                  929  TOTALBLOCKS=1
N 11:02:58.23 J0227290 IEF234E K 9207,
N 11:02:59.00 J0227291 IEF234E K 9205,


My goal is to rewrite multiline messages to produce a single-line-per-message log (keeping IEC205I messages only):
----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----0
M 11:01:58.14 J0227280  IEC205I SYSUT2  DSN=TMP.FC1,TO                                             
M 11:02:58.14 J0227290  IEC205I SYSUT2  DSN=TMP.FC2    LISTED VOL(S)  KL1=TKLM.TAPE0 TOTALBLOCKS=1


If the number of rows were fixed (i.e. alway 5) it will be easier, using a DFSORT RESIZE-TOLEN with the correct LRECL (multiple of the initial).
But in this case the PUSH or the SPLICE could be helpful?

Thanks for your suggestions.
claudioz
 
Posts: 4
Joined: Tue Jan 08, 2013 1:40 pm
Has thanked: 0 time
Been thanked: 0 time

Re: INCLUDE lines of variable multiline message IEC205I

Postby skolusu » Wed Jan 09, 2013 11:33 pm

claudioz,

You can use GROUP to push the records individually and then use sections to get 1 large record. Alternatively you can just strip the records and then use splice. I will show you the GROUP/SECTIONS option. I assumed your LRECL=133 and RECFM=FBA. I am showing you a way to club a max of 10 records.
//STEP0100 EXEC PGM=SORT                                 
//SYSOUT   DD SYSOUT=*                                   
//SORTIN   DD DISP=SHR,DSN=Your input FBA SYSLOG
//SORTOUT  DD SYSOUT=*                                   
//SYSIN    DD *                                           
  OPTION COPY                                             
  INREC IFTHEN=(WHEN=INIT,OVERLAY=(150:1330X)),           
  IFTHEN=(WHEN=GROUP,BEGIN=(58,8,CH,EQ,C'IEC205I'),     
        END=(2,1,CH,EQ,C'E'),PUSH=(134:ID=8,SEQ=1)),
     
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,1),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0150:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,2),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0283:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,3),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0416:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,4),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0549:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,5),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0682:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,6),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0815:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,7),               
        END=(2,1,CH,EQ,C'E'),PUSH=(0948:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,8),               
        END=(2,1,CH,EQ,C'E'),PUSH=(1081:1,133)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,9),               
        END=(2,1,CH,EQ,C'E'),PUSH=(1214:1,133)),         
  IFTHEN=(WHEN=(142,1,ZD,EQ,10),OVERLAY=(1347:1,133))     
                                                         
  OUTFIL INCLUDE=(134,1,CH,GT,C' '),REMOVECC,NODETAIL,   
  SECTIONS=(134,8,TRAILER3=(150,1330))
//*
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times

Re: INCLUDE lines of variable multiline message IEC205I

Postby claudioz » Thu Jan 10, 2013 9:41 pm

Hi skolusu,
Well done :) the last code is what I was expecting.
Thanks for the work of collecting all the details and producing those statements!
claudioz
 
Posts: 4
Joined: Tue Jan 08, 2013 1:40 pm
Has thanked: 0 time
Been thanked: 0 time

Re: INCLUDE lines of variable multiline message IEC205I

Postby skolusu » Fri Jan 11, 2013 1:15 am

Claudioz,

Glad it worked for you. However I apologize for making a mistake. I showed you to handle 10 occurrences, however I only had 1 byte sequence number on the WHEN=GROUP. Sorry about that. The updated control cards are below.
//SYSIN    DD *                                         
  OPTION COPY                                           
  INREC IFTHEN=(WHEN=INIT,OVERLAY=(150:1330X)),         
  IFTHEN=(WHEN=GROUP,BEGIN=(58,8,CH,EQ,C'IEC205I'),   
        END=(2,1,CH,EQ,C'E'),PUSH=(134:ID=8,SEQ=2)),   
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,1),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0150:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,2),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0283:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,3),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0416:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,4),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0549:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,5),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0682:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,6),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0815:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,7),             
        END=(2,1,CH,EQ,C'E'),PUSH=(0948:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,8),             
        END=(2,1,CH,EQ,C'E'),PUSH=(1081:1,133)),       
  IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,9),             
        END=(2,1,CH,EQ,C'E'),PUSH=(1214:1,133)),       
  IFTHEN=(WHEN=(142,2,ZD,EQ,10),OVERLAY=(1347:1,133))   
                                                       
  OUTFIL INCLUDE=(134,1,CH,GT,C' '),REMOVECC,NODETAIL, 
  SECTIONS=(134,8,TRAILER3=(150,1330))                 
//*
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
skolusu
 
Posts: 586
Joined: Wed Apr 02, 2008 10:38 pm
Has thanked: 0 time
Been thanked: 39 times


Return to DFSORT/ICETOOL/ICEGENER

 


  • Related topics
    Replies
    Views
    Last post