Storage-Group utilization during a certain time-window



Help for IBM's record-oriented filesystem VSAM, ESDS, KSDS, RRDS, LDS and Storage management Subsystems

Storage-Group utilization during a certain time-window

Postby Aki88 » Thu Dec 22, 2016 8:32 pm

Hello,

z/OS: 2.2
DFSMS: 2.2

We have a dedicated storage group for temporary/'work' datasets. The maximum usage observed over a period of ~4 years was of 40% overall during batch run (i.e. free= ~60%).
Though very recently a %usage of 63% was observed during the batch run, this was there for a period of approximately 15 minutes post which the storage was released and the usage patterns were back to normal.
Have checked all the batch jobs that were executing within +/- 30 minutes time-window of this fluctuation. Verified the JESYSMSG for the usage of the volumes in questions, haven't been able to spot an abnormality there. Have verified the user activity, again by verifying the user-ids JESYSMSG, nothing fishy there as well.

Wanted to know if there is a way to extract the below information for that day's activity (SMF maybe?? Types 72/74?!? unsure):
a. The datasets that were allocated to the storage group in question for the time-interval in question
b. Source of these datasets, job or user
c. Storage used by the individual datasets

Thank you.
Aki88
 
Posts: 381
Joined: Tue Jan 28, 2014 1:52 pm
Has thanked: 33 times
Been thanked: 36 times

Re: Storage-Group utilization during a certain time-window

 

Re: Storage-Group utilization during a certain time-window

Postby Robert Sample » Thu Dec 22, 2016 10:05 pm

SMF 19 records, if collected, might provide you data. The type 72 / 74 records are typically associated with the WLM service class and not the storage group. I'd look at the type 30 records (assuming your site collects all 6 subtypes) for the interval in question.
Robert Sample
Global moderator
 
Posts: 3649
Joined: Sat Dec 19, 2009 8:32 pm
Location: Dubuque, Iowa, USA
Has thanked: 1 time
Been thanked: 268 times

Re: Storage-Group utilization during a certain time-window

Postby Aki88 » Fri Dec 23, 2016 4:49 pm

Thank you Mr. Sample that helps; I'll work on this. IIRC 19 is not getting logged; will check again.
Aki88
 
Posts: 381
Joined: Tue Jan 28, 2014 1:52 pm
Has thanked: 33 times
Been thanked: 36 times

Re: Storage-Group utilization during a certain time-window

Postby vasanthz » Sat Dec 24, 2016 2:19 am

Hello,

You could also look at Type 14, 15 during that interval. It would give information about the list of datasets, size, storage group etc.. that were allocated during the problematic interval.
Along with Type 61 and Type 65.
User avatar
vasanthz
 
Posts: 22
Joined: Thu Aug 05, 2010 2:53 pm
Has thanked: 6 times
Been thanked: 0 time

Re: Storage-Group utilization during a certain time-window

Postby vasanthz » Sat Dec 24, 2016 2:25 am

It would be tricky to find the spike in storage group usage from type 14,15, 61 and 65. But they would provide dataset level information and the jobnames that created the datasets which would be vital.

Hypothetically I would take 10 minutes of type 14,15,61,65 then add all the allocated space, and subtract all the deleted space for datasets with STORAGEGROUP = "XXXXX". This would not be accurate, but we can identify the job/s which contribute to the sudden spike in storage group.

P.S - we are not considering renamed dataset SMF records.
User avatar
vasanthz
 
Posts: 22
Joined: Thu Aug 05, 2010 2:53 pm
Has thanked: 6 times
Been thanked: 0 time

Re: Storage-Group utilization during a certain time-window

Postby Aki88 » Sat Dec 24, 2016 4:35 am

Thank you Vasanthz, that helps. The naming convention of the work storage DS has the exact date/time stamp of creation within itself, so even if I get the list of DS (with allocation) on the storage group during the said time-frame, will be able to pull-out enough information to build an analogy for storage utilization. DS source name makes my life simpler. Will salvage the SMF logs.
Aki88
 
Posts: 381
Joined: Tue Jan 28, 2014 1:52 pm
Has thanked: 33 times
Been thanked: 36 times

Re: Storage-Group utilization during a certain time-window

Postby Aki88 » Fri Dec 30, 2016 9:09 pm

Hello,

Before I jump into the thick of things, apologies, even though I have some experience with formatted-SMF-records, I am a beginner of beginners in raw-SMF-data processing.

I wrote a few quick-crude DFSORT steps to extract the various data types and format them.

Type-61:


//STEP002  EXEC PGM=SORT                          
//SYSOUT   DD SYSOUT=*                            
//SORTIN   DD DISP=SHR,DSN=<MAN DATASET>          
//SORTOUT  DD DSN=MY.DS,                          
//            DISP=(NEW,CATLG,DELETE),            
//            SPACE=(TRK,(2,5),RLSE)              
//*                                              
//SYSIN   DD *                                    
 SORT FIELDS=COPY                                
 INCLUDE COND=(6,1,CH,EQ,X'3D')   --> TYPE 61    
/*                                                
//*                                              
//STEP003  EXEC PGM=ICETOOL                      
//DFSMSG   DD SYSOUT=*                            
//TOOLMSG  DD SYSOUT=*                            
//SMF      DD DISP=SHR,DSN=MY.DS                  
//REPORT   DD SYSOUT=*                            
//*                                              
//TOOLIN  DD *                                    
 DISPLAY FROM(SMF) LIST(REPORT)           -      
 HEADER('JOB NAME') ON(51,8,CH)           -      
 HEADER('TIME') ON(59,4,TM1,E'99:99:99')  -  
 HEADER('DATE') ON(63,4,DT1) -                
 HEADER('DS NAME') ON(121,44,CH)              
/*                                            
 

Unless I haven't messed up the extraction/formatting, I didn't find a temporary dataset in this list.

Post this I browsed the unformatted SMF data in the MAN dataset, I could see that type 14/15/42 had entries for temporary datasets (x'0E', X'0F' and X'2A' at column-2 with HEX ON).

Second job run on type-14, which by the description looked closer to my current requirement:

//STEP002  EXEC PGM=SORT                          
//SYSOUT   DD SYSOUT=*                            
//SORTIN   DD DISP=SHR,DSN=<MAN DS>                
//SORTOUT  DD DSN=MY.DS,                          
//            DISP=(NEW,CATLG,DELETE),            
//            SPACE=(TRK,(2,5),RLSE)              
//*                                                
//SYSIN   DD *                                    
 SORT FIELDS=COPY                                  
 INCLUDE COND=(6,1,CH,EQ,X'0E')     *SMF TYPE- 14  
/*                                                
//*                                                
//STEP003  EXEC PGM=ICETOOL                        
//DFSMSG   DD SYSOUT=*                            
//TOOLMSG  DD SYSOUT=*                            
//SMF      DD DISP=SHR,DSN=MY.DS                  
//REPORT   DD SYSOUT=*                            
//*                                                
//TOOLIN  DD *                                    
 DISPLAY FROM(SMF) LIST(REPORT)                    
 TITLE('SMF TYPE 14')                              
 HEADER('JOB NAME') ON(19,8,CH)                                 -    
 HEADER('TIME') ON(27,4,TM1,E'99:99:99')                        -    
 HEADER('DATE') ON(31,4,DT1,E'9999/99/99')                      -    
 HEADER('OWNER') ON(35,8,CH)                                    -    
 HEADER('REC INDICATOR') ON(43,2,BI,E'9999')                    -    
 HEADER('TIME DS OPEN') ON(49,4,TM1,E'99:99:99')                -    
 HEADER('DD NAME') ON(57,8,CH)                                  -    
 HEADER('DSN') ON(69,44,CH)                                     -    
 BLANK                                                              
/*                                                                  
 


Output for this appears as below:

JOB NAME       TIME         DATE   OWNER      REC INDICATOR   TIME DS OPEN   DD NAME    DSN          
--------   --------   ----------   --------   -------------   ------------   --------   --------------------------------------------
XXXXXXXX   13:24:35   2016/12/27   STCUSR              8196       13:24:43   SYS00001   SYS16362.T132442.RA000.XXXXXXXX.R0167641
XXXXXXXX   13:24:35   2016/12/27   STCUSR              8196       13:24:43   SYS00001   SYS16362.T132442.RA000.XXXXXXXX.R0167641
XXXXXXXX   13:24:35   2016/12/27   STCUSR              8196       13:24:43   SYS00002   SYS16362.T132442.RA000.XXXXXXXX.R0167642
XXXXXXXX   13:24:35   2016/12/27   STCUSR              8196       13:24:43   SYS00002   SYS16362.T132442.RA000.XXXXXXXX.R0167642
 


The point where I am now stuck is:
a. How do I add the sub-types to this extraction? If I've understood it right, I'll have to add the length of each type to reach the starting position of the type I am looking for. And from that point onwards, add the offset of the field I am looking for to extract the exact data. For example, to extract the UCB section data, I'll have to add the lengths of all the preceding sections and then add the offset of the field I am looking for.
b. SMF14RIN which is reflecting as 8196 here, I am unsure if my data conversion was correct; though DFSORT has basically done what I asked it to do- convert binary to displayable data, but I am sure the code is messed up.

Any guidance is much appreciated, this is one opportunity for me to really get in and finally learn basics of SMF data processing.

PS: Other record types were also extracted in a similar manner, with similar challenges.
Aki88
 
Posts: 381
Joined: Tue Jan 28, 2014 1:52 pm
Has thanked: 33 times
Been thanked: 36 times

Re: Storage-Group utilization during a certain time-window

Postby Robert Sample » Fri Dec 30, 2016 9:40 pm

SMF record type 61 is ICF defines -- a temporary data set is not cataloged so it wouldn't show up here, normally.

SMF record type 14 and 15 have identical layouts and usually should be treated together (14 is input, 15 is output data sets).

a. SORT doesn't really handle SMF data very well -- you may wind up needing to use some other utility or write a program (if SAS is installed there, it works really well with SMF records). The TIOT and JFCB are fixed-length, and you get the DCB/DEB section length from offset 44 (SMF14SDC is the field name). Offsets 45 and 46 give you the number of UCB sections and the size of each. Once you calculate the offset to the first UCB section, merely add the size to get the starting offset for the next UCB section; continue for the number of UCB sections. And a terminology note: the 14 and 15 record types don't really have subtypes; they have sections. Record type 42 has subtypes and you'll see at offset 22 field SMF42STY which is a 2-byte binary subtype indicator for the record. Starting at offset 36, the number of triplets depends upon the subtype.
b. 8196 is correct -- SMF14RIN is a 2-byte set of bit flags - you will probably want to keep the SMF manual open while working with the record types as there's a lot of bit-level data in them. 8196 is B'0001 0000 0000 0000' which indicates that the data set is a temporary data set. It is possible for some of the other bits to be set as well (but not likely for just a temporary data set).

These users thanked the author Robert Sample for the post:
Aki88 (Fri Dec 30, 2016 11:22 pm)
Robert Sample
Global moderator
 
Posts: 3649
Joined: Sat Dec 19, 2009 8:32 pm
Location: Dubuque, Iowa, USA
Has thanked: 1 time
Been thanked: 268 times

Re: Storage-Group utilization during a certain time-window

Postby Aki88 » Fri Dec 30, 2016 10:19 pm

Thank you for the pointers and the correction Mr. Sample.
Sadly, we do not have SAS or a SMF data processing tool as MXG; the only options that I have- all revolve around building a logic with a programming language as COBOL/REXX or a tool as DFSORT.

Aside, please correct me if I have again misunderstood the data format; 8196 (DEC) to binary comes as: 0010 0000 0000 0100, so from the SMF manual, bit 3 tells me that DS is a temporary dataset, and the 14th bit tells me that 'Task termination closed the DCB'.
Aki88
 
Posts: 381
Joined: Tue Jan 28, 2014 1:52 pm
Has thanked: 33 times
Been thanked: 36 times

Re: Storage-Group utilization during a certain time-window

Postby Robert Sample » Fri Dec 30, 2016 10:36 pm

You'll have quite a time with the data, then -- especially if you need any of the records with subtypes.

Yes, you interpreted the bits correctly -- I forgot to add back the other bit when I was explaining earlier.
Robert Sample
Global moderator
 
Posts: 3649
Joined: Sat Dec 19, 2009 8:32 pm
Location: Dubuque, Iowa, USA
Has thanked: 1 time
Been thanked: 268 times

Next

Return to VSAM/SMS

 


  • Related topics
    Replies
    Views
    Last post