Page 2 of 2

Re: Performance of SDSF REXX

PostPosted: Wed Jun 06, 2012 6:22 pm
by Viswanathchandru
Thanks NicC,

I cant understand.
DO IX=1 TO ISFLINE.0
This when gets into TRACE ?R i can find the number of lines that the entire syslog(the moment the script starts) and the script after passing this line should write only upto the limit ISFLINE.0 if i'm not wrong. then even if the log grows it should write only the line contained in isfline. so i'm not prett much sure whether the growing log is a show stopper. Please advice.


Regards,
Viswa

Re: Performance of SDSF REXX

PostPosted: Wed Jun 06, 2012 6:55 pm
by Pedro
the script after passing this line should write only upto the limit ISFLINE.0

More or less. Except...

DO IX=1 TO ISFLINE.0                                       
"EXECIO * DISKW ISFOUT (STEM ISFLINE. FINIS"               
END                                                       


Say, for example, that you have 20,000 records

What your code does:
1. do this loop 20,000 times
2. in each iteration, write 20,000 lines.
you are trying to write 20,000 x 20,000 = 40,000,000 lines. Which is way more than what you really want.

But actually, the FINIS probably causes the second and subsequent iterations to fail. What does your trace show? You need to issue EXECIO only once. And check the return code to make sure it works.

I'm afraid i'm not sure how to get the timestamp ? Please advice.

I expect each line of syslog to have a timestamp at the start of each record. What I was asking for was the first part of the first record and the first part of the last record.

ADDRESS SDSF "ISFLOG READ TYPE(SYSLOG)"

You are reading the active syslog. But as NicC mentioned, it is constantly being updated. It keeps growing, except that it is periodically offloaded to an archive. Say that your first READ is at 9:59, you may get 20,000 records. And then the system archives the syslog at 10:00. When your second READ occurs at 10:01, you will get vastly different results than the first READ.

Re: Performance of SDSF REXX

PostPosted: Wed Jun 06, 2012 7:23 pm
by steve-myers
I have written programs to analyze the data in SYSLOG. Because of the volume of data, these are Assembler programs, not Rexx scripts. SYSLOG timestamps are particularly difficult: not all records have them, and their format, especially the time of day, is difficult to process. It took careful programming, even in Assembler, to keep CPU time reasonable. Most of the data is in variable length fields at varying positions in the data. All too often, the data for a single event spans multiple SYSLOG records.

No attempt is made to process the data in real time; the SYSLOG is extracted to a flat file and the flat file is analyzed. I archive enormous amounts of SYSLOG data, because sometimes what you really need is days or weeks old.

Re: Performance of SDSF REXX

PostPosted: Wed Jun 06, 2012 10:05 pm
by Viswanathchandru
Dear all, I'm happy :) With all the people around here in this forum i was able to achieve the requirement. Special thanks to Pedro, Steve, NicC,mongan and to all who atleast visited the post in a mind set of helping. Appreciate the help and thanks a ton for giving your views and experienced comments. It values a lot as always i'm not going with empty hands. I'm going with what i expected. Thanks a lot guru's. Appreciate your help and advice.


Regards,
Viswa

Re: Performance of SDSF REXX

PostPosted: Thu Jun 07, 2012 6:43 am
by Pedro
SYSLOG timestamps are particularly difficult: not all records have them, and their format, especially the time of day, is difficult to process.
I think the
ADDRESS SDSF "ISFLOG READ TYPE(SYSLOG)" 
must do the difficult time formatting on our behalf. I finally tried the program earlier in this thread and the data produced looked exactly like the data shown by SDSF option LOG. No timestamp conversion is required.

Re: Performance of SDSF REXX

PostPosted: Thu Jun 07, 2012 7:44 am
by steve-myers
Pedro wrote:
SYSLOG timestamps are particularly difficult: not all records have them, and their format, especially the time of day, is difficult to process.
I think the
ADDRESS SDSF "ISFLOG READ TYPE(SYSLOG)" 
must do the difficult time formatting on our behalf. I finally tried the program earlier in this thread and the data produced looked exactly like the data shown by SDSF option LOG. No timestamp conversion is required.
SDSF LOG translates the 12158 it sees in the SYSLOG records to the current date (06/06/2012) and displays it in a title bar and page headers when you do a PRINT command. I don't think the time of day (20:30:09.42) is processed by SDSF. My program tests each numeric area and the : and . separators to make sure it has a good date & time, and forms a TIME DEC equivalent it uses for its purposes, though it makes no further use of the time of day.

Re: Performance of SDSF REXX

PostPosted: Thu Jun 07, 2012 9:19 am
by Pedro
SDSF LOG translates the 12158 it sees in the SYSLOG records to the current date (06/06/2012) ... I don't think the time of day (20:30:09.42) is processed by SDSF.

I am not sure what you are trying to say. The ISFLOG function returns syslog records in this format (right truncated):
NR0000000 PSVMVS1A 12158 16:00:58.17 INTERNAL 00000090  IKJ591I  DATA SET CANNOT BE USED,
N 8000000 PSVMVS1A 12158 16:01:00.02 S0095676 00000090  DSNW123I  #DSNA- DSNWVSMF - TRACE
N 4000000 PSVMVS1A 12158 16:01:00.03 S0095596 00000090  ARC0520I PRIMARY SPACE MANAGEMENT
M 4000000 PSVMVS1A 12158 16:01:01.19 S0095596 00000090  ARC0522I SPACE MANAGEMENT STARTING
E                                         444 00000090  ARC0522I (CONT.) SM4201(SMS) AT
M 4000000 PSVMVS1A 12158 16:01:01.57 S0095596 00000090  ARC0522I SPACE MANAGEMENT STARTING