Need only the last 15 minutes worth of records



JES, JES2, JCL utilities, IDCAMS, Compile & Run JCLs, PROCs etc...

Need only the last 15 minutes worth of records

Postby Compscigui » Fri Aug 10, 2018 10:17 pm

Hello everyone,

I am fairly new to JCL and REXX, and was handed off a project to send RACF log data to a program called SPLUNK. They handed me a bunch of code, said it worked, and told me to set up a job in ESP to submit. I did that, but when looking at the output, I am sending a lot of redundant data that I don't want to keep sending every 15 minutes, every day. I am looking to try and implement some sort of time variable so the only records grabbed would fall within the last 15 minutes (since the job runs every 15 minutes). I'll post the code and show what they gave me:

This is the JCL:
//DXZ4JOBC JOB (SCRUBBED),'COMPSCIGUI',    
// MSGLEVEL=1,MSGCLASS=T,PRTY=10,TIME=5                        
//*EXEC CNTR3                                                  
//*MAIN ORG=CNTR3,CLASS=BATCH                                  
//OPERSCAN EXEC OPERLOGP                                      
//OPRDATA  DD DISP=SHR,DSN=DX.D945.OPERLOG3                    
//PRTOLOG  DD  DSN=&&OLOGOUT,UNIT=DISK,SPACE=(CYL,(100,100)),  
//         DISP=(,PASS),DCB=LRECL=133                          
//OPRPARMS DD *                                                
 REPORT  PRTOLOG                                              
 TIME    0000 2359                                            
 MESSAGE ICH70001I                                            
 MESSAGE ICH408I                                              
//*                                                            
//* PARSE THE OLOG OUTPUT TO COMBINE ICH408I MESSAGES ONTO    
//* A SINGLE LINE.                                            
//*                                                            
//PARSE    EXEC PGM=IRXJCL,DYNAMNBR=50,REGION=6000K,        
//            PARM='AUTHPARS'                                
//SYSTSPRT  DD SYSOUT=*                                      
//MESSAGES  DD DSN=&&OLOGOUT,DISP=SHR                        
//OUTDD     DD DSN=&&AUTHOUT,UNIT=DISK,SPACE=(CYL,(100,100)),
//          DISP=(,PASS),DCB=LRECL=500                      
//SYSTSIN   DD DUMMY                                        
//SYSEXEC   DD DSN=SCRUBBED.REXX.EXEC,DISP=SHR                
//*                                                          
//* UNCOMMENT TO WRITE A SECOND COPY TO THE JOB SYSOUT */    
//*                                                          
//IEBCOPY  EXEC PGM=IEBGENER                                
//SYSUT1    DD DSN=&&AUTHOUT,DISP=(SHR,PASS)                
//SYSUT2    DD SYSOUT=(,)                                    
//SYSPRINT  DD SYSOUT=(,)                                    
//SYSIN     DD DUMMY                                        
//*                                                          
//* TRANSMIT THE MESSAGES TO THE SPLUNK SERVER USING SYSLOG
//*                                                        
//TRANSMIT EXEC PGM=IRXJCL,DYNAMNBR=50,REGION=6000K,      
//            PARM='SPLUNKER IP ADDRESS SCRUBBED'          
//SYSTSPRT  DD SYSOUT=*                                    
//MESSAGES  DD DSN=&&AUTHOUT,DISP=SHR                      
//SYSTSIN   DD DUMMY                                      
//SYSEXEC   DD DSN=SCRUBBED.REXX.EXEC,DISP=SHR              
 


Here is the REXX:

TRACE N     /* TRACE I - verbose, R - intermediate, N - normal, E - errors */
                                                                       
/* Parse and display the parameters passed by the batch job */          
parse arg ipaddress port                                                
say 'Endpoint -' ipaddress':'port                                      
                                                                       
/* Read all records from the MESSAGES DD into a stem variable */        
"EXECIO * DISKR messages (STEM records."                                
return_code = RC                                                        
SAY records.0 ' records read'                                          
                                                                       
/* Initialize */                                                        
call Socket 'Initialize', 'SPLUNKER'                                    
if src=0 then initialized = 1                                          
else call error 'E', 200, 'Unable to initialize SOCKET'                
if server='' then do                                                    
  server = Socket('GetHostId')                                          
  if src¬=0 then call error 'E', 200, 'Cannot get the local ipaddress'  
end                                                                    
                                                                       
/* Initialize for receiving lines sent by the server                  */
s = Socket('Socket')          /* Would add ,DATAGRAM type here if UDP */
if src¬=0 then call error 'E', 32, 'SOCKET(SOCKET) rc='src              
hostname = translate(Socket('GetHostName'))                            
if src¬=0 then call error 'E', 32, 'SOCKET(GETHOSTNAME) rc='src        
call Socket 'Connect', s, 'AF_INET' port ipaddress                      
if src¬=0 then call error 'E', 32, 'SOCKET(CONNECT) rc='src            
call Socket 'SetSockOpt', s, 'SOL_SOCKET', 'SO_ASCII', 1                
                                                                       
crlf="0D"X          /* store a literal for the end of record character */
                                                                       
/* Loop to write records to the server   */                            
DO i = 1 to records.0                                                  
    sendstring = strip(records.i)                                      
    call Socket 'Write', s, sendstring                                  
    if src¬=0 then call error 'E', 32, 'SOCKET(WRITE) rc='src          
    call Socket 'Write', s, crlf                                        
    if src¬=0 then call error 'E', 32, 'SOCKET(WRITE) rc='src
END                                                                    
                                                                       
/* Terminate and exit                                                 */
call Socket 'Terminate'                                                
exit 0                                                                  
                                                                       
EXIT return_code                                                        
                                                                       
socket: procedure expose src                                            
  a0 = arg(1)                                                          
  a1 = arg(2)                                                          
  a2 = arg(3)                                                          
  a3 = arg(4)                                                          
  a4 = arg(5)                                                          
  a5 = arg(6)                                                          
  a6 = arg(7)                                                          
  a7 = arg(8)                                                          
  a8 = arg(9)                                                          
  a9 = arg(10)                                                          
  parse value 'SOCKET'(a0,a1,a2,a3,a4,a5,a6,a7,a8,a9) with src res      
return res                                                              
                                                                       
/* Syntax error routine                                               */
syntax:                                                                
  call error 'E', rc, '==> REXX Error No.' 20000+rc                    
return                                                                  
                                                                       
                                                                       
/* Halt processing routine                                            */
halt:                                                                  
  call error 'E', 4, '==> REXX Interrupted'                            
return                                                                  
                                                                       
/* Error message and exit routine                                     */
error:                                                                  
  type = arg(1)                                                        
  retc = arg(2)                                                        
  text = arg(3)  
  ecretc = right(retc,3,'0')                                          
  ectype = translate(type)                                            
  ecfull = 'RXSCLI' || ecretc || ectype                                
  say '===> Error:' ecfull text                                        
  if type¬='E' then return                                            
  if initialized                                                      
     then do                                                          
       parse value Socket('SocketSetStatus') with . status severreason
       if status¬='Connected'                                          
          then say 'The status of the socket set is' status severreason
     end                                                              
  call Socket 'Terminate'                                              
exit retc                                                                                                                                
 


Right now is grabs everything every time the job submits, and I can't keep doing that. Any help would be greatly appreciated, as I am new to this and stuck.

Thanks
Compscigui
 
Posts: 2
Joined: Fri Aug 10, 2018 10:04 pm
Has thanked: 1 time
Been thanked: 0 time

Re: Need only the last 15 minutes worth of records

Postby willy jensen » Sat Aug 11, 2018 1:59 pm

Use the TIME function to get the current time, then compute now-15 mins. TIME('S') is probably the best choice, but beware of midnight.
Use PARSE or the SUBSTR function to pull the date/time stamp from the records.
willy jensen
 
Posts: 455
Joined: Thu Mar 10, 2016 5:03 pm
Has thanked: 0 time
Been thanked: 69 times

Re: Need only the last 15 minutes worth of records

Postby NicC » Sat Aug 11, 2018 2:17 pm

Welcome to our big world here!
As I said on the other forum, probably the best way to do this without changing the Rexx code is to send a delta data set:
run 1: save the log data set, send the saved log data set
run 2: save the latest log data set, compare against the previous log data set (joinkeys of your sort product) to create the delta data set, send the delta
subsequent runs: same as run 2

The reason for doing it this way is:
a) from previous discussions I know that not all your records have the time on them
b) you will not have the midnight problem
The problem I have is that people can explain things quickly but I can only comprehend slowly.
Regards
Nic
NicC
Global moderator
 
Posts: 3025
Joined: Sun Jul 04, 2010 12:13 am
Location: Pushing up the daisies (almost)
Has thanked: 4 times
Been thanked: 136 times

Re: Need only the last 15 minutes worth of records

Postby expat » Mon Aug 13, 2018 11:42 am

Can you not just send the data once a day

Or even forget about splunk altogether and use the report writer to generate reports either scheduled or ad hoc
expat
 
Posts: 459
Joined: Sat Jun 09, 2007 3:21 pm
Has thanked: 0 time
Been thanked: 8 times

Re: Need only the last 15 minutes worth of records

Postby NicC » Wed Aug 15, 2018 9:47 pm

This is your exec modified with new code and corrections/perfromance improvements made to your original code. None of this has been tested, of course. I have added midnight processing but I realize I have omitted the check for those odd records that do not have a timestamp. How you deal with those will depend on whether they are required or not. I have also left one 'quirk' from your original code and commented it. Up to you which EXIT statement you want to use. Check out lines with <===== on them (3 of them).
/* Rexx program to send messages to SPLUNKER */
Trace N     /* TRACE I - verbose, R - intermediate, N - normal, E - errors */
                /* TRACE O - no tracing at all */
/*----------------*/
/* Some variables */
/*----------------*/
record_count = 0
recs_per_batch = 1000   /* Estimated records generated in 30 minutes */
time_difference = 30        /* Length of time to go back (in minutes) */
t_start  = 22               /* Start position of time field (or is it 26?) */
t_length = 8                /* Length of time field */
forever  = 1                /* Loop control */

/* Parse and display the parameters passed by the batch job */
Parse Arg ipaddress port
Say 'Endpoint -' ipaddress':'port
/* Initialize */
Call Socket 'Initialize', 'SPLUNKER'
If src=0
Then initialized = 1
Else Call error 'E', 200, 'Unable to initialize SOCKET'

If server=''
Then Do

  server = Socket('GetHostId')
  If src^=0
  Then Call error 'E', 200, 'Cannot get the local ipaddress'
 
End
 
/* Initialize for receiving lines sent by the server                  */
s = Socket('Socket')          /* Would add ,DATAGRAM type here if UDP */
If src ^= 0
Then Call error 'E', 32, 'SOCKET(SOCKET) rc='src

hostname = translate(Socket('GetHostName'))
If src ^= 0
Then Call error 'E', 32, 'SOCKET(GETHOSTNAME) rc='src

Call Socket 'Connect', s, 'AF_INET' port ipaddress
If src ^= 0
Then Call error 'E', 32, 'SOCKET(CONNECT) rc='src

Call Socket 'SetSockOpt', s, 'SOL_SOCKET', 'SO_ASCII', 1
 
crlf="0D"X          /* store a literal for the end of record character */

current_time_in_minutes = Time('M')                             /* Current time in minutes */

/* Read a batch of records from the MESSAGES DD into stem variable 'records.'*/
"EXECIO "recs_per_batch" DISKR messages (STEM records."
If rc <> 0
Then Call error 'E' 400 "EXECIO error rec = "rc   /* 400??? Whatever <===== */
Do forever            
 
    /* Loop to write records to the server   */
    Do batch_ix = 1 To records.0
   
        record_count = record_count + 1
       
       time_in_record = Substr(record.batch_ix, t_start,t_length)
       record_time = Time('M', time_in_record)  /* Convert to same format as stop_time */

        /* Processing just after midnight */
        /* We could have records from 2330 previous day. If so, we need to add
           a day's worth of minutes to the time now. */
        If record_time < current_time_in_minutes        /* Today'
s record */
        Then earliest_time = Max(current_time_in_minutes - time_difference,0)
        Else earliest_time = current_time_in_minutes + 1440 - current_time_in_minutes /* Add a day of minutes (24*60)*/
       
        If record_time > earliest_time
        Then Do
       
        sendstring = strip(records.batch_ix)
   
        Call Socket 'Write', s, sendstring||crlf        
            If src ^= 0
        Then Call error 'E', 32, 'SOCKET(WRITE) rc='src
        
        End
        Else Do /* We have finished */
       
            batch_ix = records.0 + 1
            forever = 0
           
        End
   
    End
   
    Drop records.
    If forever
    Then "EXECIO "recs_per_batch" DISKR messages (STEM records."
    If rc <> 0
    Then Call error 'E' 400 "EXECIO error rec = "rc   /* 400??? Whatever <===== */
   
End
Say "Records read: "record_count
 
/* Terminate and exit                                                 */
Call Socket 'Terminate'
Exit 0
/*------------------------------------------------
This line of code never gets executed because of
the Exit 0 above:
Exit return_code <=====
------------------------------------------------*/
 
socket: Procedure Expose src
  a0 = Arg(1)
  a1 = Arg(2)
  a2 = Arg(3)
  a3 = Arg(4)
  a4 = Arg(5)
  a5 = Arg(6)
  a6 = Arg(7)
  a7 = Arg(8)
  a8 = Arg(9)
  a9 = Arg(10)
  Parse Value 'SOCKET'(a0,a1,a2,a3,a4,a5,a6,a7,a8,a9) With src res

Return res
 
/* Syntax error routine                                               */
syntax:
  Call error 'E', rc, '==> REXX Error No.' 20000 + rc

Return
 
/* Halt processing routine                                            */
halt:
  Call error 'E', 4, '==> REXX Interrupted'
 
Return
 
/* Error message and exit routine                                     */
error:
  type = Arg(1)
  retc = Arg(2)
  text = Arg(3)
  ecretc = Right(retc, 3, '0')
  ectype = translate(type)
  ecfull = 'RXSCLI' || ecretc || ectype
  Say '===> Error:' ecfull text
  If type ^= 'E'
  Then Return
 
  If initialized
  Then Do
 
     Parse Value Socket('SocketSetStatus') With . status severreason
     If status^='Connected'
     Then Say 'The status of the socket set is' status severreason
     
  End
 
  Call Socket 'Terminate'

Exit retc
 
The problem I have is that people can explain things quickly but I can only comprehend slowly.
Regards
Nic

These users thanked the author NicC for the post:
Compscigui (Wed Aug 15, 2018 11:22 pm)
NicC
Global moderator
 
Posts: 3025
Joined: Sun Jul 04, 2010 12:13 am
Location: Pushing up the daisies (almost)
Has thanked: 4 times
Been thanked: 136 times

Re: Need only the last 15 minutes worth of records

Postby Compscigui » Wed Aug 15, 2018 11:23 pm

Thank you Nic, that definitely got me close to where I needed to be. With a few value tweaks I was able to get exactly what I needed. You are awesome, and I thank you for your time and effort!
Compscigui
 
Posts: 2
Joined: Fri Aug 10, 2018 10:04 pm
Has thanked: 1 time
Been thanked: 0 time


Return to JCL

 


  • Related topics
    Replies
    Views
    Last post