Reading Type80 SMF Records file



All about SAF, RACF, encryption, Firewall, Risk assessment and integrity concepts

Reading Type80 SMF Records file

Postby cortex » Mon Dec 26, 2011 7:04 pm

Hello,

First of all, sorry to bothering you in this forum, as I am *really* not an IBM guy, but I need your help, since I found this forum.

The question is simple:

I need to compute a file full of type80 records. I know the file is binary and I was able to find his structure (http://publib.boulder.ibm.com/infocente ... e80fmt.htm)

So I wrote a simple C program just to look at the file content. Looking at the record structure I noticed that at the 6th byte, its contents *must* be 0x50 (because it's type80). But the weird thing is that I can't notice any 0x50 on the file, when I print the bytes to the stdout. So I guess I'm looking in the bad way to that file.

So, how can I correctly see the file contents? Is there any mismatch between little-endian or big-endian (as I'm working under a little-endian architecture, and I guess the file was written under big-endian)?

The Mainframe admin said that this file is the original binary one.

Sorry again to bothering you guys, and I hope you understand the problem.

Kind Regards
cortex
 
Posts: 11
Joined: Mon Dec 26, 2011 5:53 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Reading Type80 SMF Records file

Postby BillyBoyo » Mon Dec 26, 2011 7:08 pm

How was the file transferred?

If you are not seeing the x'50' then the file is probably irredeemably garbled. If you look at the data with something, can you "read" any of it, ie can you see any reasonable amount of normal characters? If Yes, then something somewhere has converted it to ASCII, and you have the effect of changing all the letter e's in a document to a's and then changing all the a's to e's.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Re: Reading Type80 SMF Records file

Postby cortex » Mon Dec 26, 2011 7:19 pm

First of all, thanks for your help BillyBoyo.

If I look at the data with a text editor or print it to stdout, I can't read anything so I guess the fill is still in binary.

Just for the record and if it helps, I can show the first couple of bytes of the file:


(HEX – BIN)
1e-00011110
02-00000010 (1E in ASCII it's "Record Separator character" and 02 is "Start of Text"..weird or coincidence?)
00-00000000
28-00101000
7b-01111011
bc-10111100
01-00000001
11-00010001
31-00110001 (Somewhere behind this...the 0x50 had to appear, right?)
5f-01011111
d4-11010100
e5-11100101
e2-11100010
c1-11000001
de-11011110
1e-00011110
00-00000000
00-00000000
4b-01001011
3c-00111100
01-00000001
11-00010001
31-00110001
5f-01011111
d4-11010100
e5-11100101
e2-11100010
c1-11000001
d1-11010001
c5-11000101
e2-11100010
c1-11000001
00-00000000
01-00000001
00-00000000
00-00000000
00-00000000
b0-10110000
00-00000000
26-00100110
00-00000000
01-00000001
00-00000000
00-00000000
00-00000000
d6-11010110
00-00000000
ba-10111010
00-00000000
01-00000001



I don't know if I'm dealing with big/little endian issues and, as I said, I don't know if I'm understand the file in the right way.


Regards,
cortex
 
Posts: 11
Joined: Mon Dec 26, 2011 5:53 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Reading Type80 SMF Records file

Postby Robert Sample » Mon Dec 26, 2011 7:29 pm

Showing the bit patterns is not required -- we understand ASCII and EBCDIC.

The file you have displayed is EBCDIC -- as evidenced by the MVSA in bytes 11 through 14 of your record (the X'D4' through X'C1'). However, the header does not reflect an SMF 80 type record -- are you sure that the record extraction routine only selected record type 80 for your file? My first thought is that you have a raw SMF file which has many, many, many other record types besides what you want to look at.

There is no little endian - big endian issue that I'm aware of. And the 1E / 02 is pure coincidence -- the fields you are looking at are part of the system data, not application data and have nothing to do with ASCII.
Robert Sample
Global moderator
 
Posts: 3719
Joined: Sat Dec 19, 2009 8:32 pm
Location: Dubuque, Iowa, USA
Has thanked: 1 time
Been thanked: 279 times

Re: Reading Type80 SMF Records file

Postby cortex » Mon Dec 26, 2011 7:41 pm

Hi Robert Sample, and thanks for your reply.

Robert Sample wrote:Showing the bit patterns is not required -- we understand ASCII and EBCDIC.

Yes, absolutely. I just copy-pasted a print of my test routine.

Robert Sample wrote:The file you have displayed is EBCDIC -- as evidenced by the MVSA in bytes 11 through 14 of your record (the X'D4' through X'C1'). However, the header does not reflect an SMF 80 type record -- are you sure that the record extraction routine only selected record type 80 for your file? My first thought is that you have a raw SMF file which has many, many, many other record types besides what you want to look at.


Well...at least I've asked the mainframe admin to give me a file with just Type80 Records. I've trusted him :mrgreen:

Robert Sample wrote:There is no little endian - big endian issue that I'm aware of. And the 1E / 02 is pure coincidence -- the fields you are looking at are part of the system data, not application data and have nothing to do with ASCII.


Yes, I was almost sure of that. Now I'm 100% sure.

But you are saying that those aren't Type 80 Records. Are you saying that because the lack of that X'50' that I was expecting too? Or can I look to some other clues to conclude that?


Thank you very much, again.

Kind Regards.
cortex
 
Posts: 11
Joined: Mon Dec 26, 2011 5:53 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Reading Type80 SMF Records file

Postby Robert Sample » Mon Dec 26, 2011 8:32 pm

I just created a file of nothing but type 80 recordsd from yesterday's SMF data at our site. Here's what the data in the file starts off with:
0RECORD SEQUENCE NUMBER - 1
 000000  1E020036 3A370111 360FE2E8 E2F1

0RECORD SEQUENCE NUMBER - 2
 000000  1E500000 67A20111 359FE2E8 E2F12800   0200D7C4 C3E2E7E7 F040D7C4 C3E2F9
 000020  F040005E 000A8010 00000000 00000000   0000C4C3 F3F6F1F1 C4400000 651401
 000040  359F4040 40404040 40EE0800 F7F7F3F0   00000000 00000000 01030000 000001
 000060  E2E8E2F1 4BE2C2D7 E7C5E7C5 C3030110   04011005 01000F06 D7D9E9F8 D9F111
 000080  C4C1E3C1 E2C5E340 210700E2 E8E2F14B   5C2608E2 E8E2F140 40404031 14C4C9
 0000A0  C3E4E240 D7D9D6C4 4B40E4E2 C5D9C9C4   40355050 01000700 03C00000 000000
 0000C0  000000D5 F1404040 404040D7 E4C3F7E7   E7F040D5 F1404040 404040E3 C5C3C8
 0000E0  F0F040C9 D5E3D9C4 D9404000 00000000   000000D7 C4C3E2E7 E7F040D7 C4C3E2
 000100  F0F040
0RECORD SEQUENCE NUMBER - 3
 000000  1E500000 68670111 359FE2E8 E2F12800   0200D7C4 C3E2E7E7 F040D7C4 C3E2F9
 000020  F040005E 000A8010 00000000 00000000   0000C4C3 F3F6F1F1 C4400000 651401
 000040  359F4040 40404040 40EE0800 F7F7F3F0   00000000 00000000 01030000 000001
 000060  E2E8E2F1 4BE2C2D7 E7C5E7C5 C3030110   04011005 01000F06 D7D9E9F8 D9F111
 000080  C4C1E3C1 E2C5E340 210700E2 E8E2F14B   5C2608E2 E8E2F140 40404031 14C4C9
 0000A0  C3E4E240 D7D9D6C4 4B40E4E2 C5D9C9C4   40355050 01000700 03C00000 000000
 0000C0  000000D5 F1404040 404040D7 E4C3F7E7   E7F040D5 F1404040 404040E3 C5C3C8
 0000E0  F0F040C9 D5E3D9C4 D9404000 00000000   000000D7 C4C3E2E7 E7F040D7 C4C3E2
 000100  F0F040
I think that it is obvious by comparing this to your data that you do not have a file of just type 80 records.
Robert Sample
Global moderator
 
Posts: 3719
Joined: Sat Dec 19, 2009 8:32 pm
Location: Dubuque, Iowa, USA
Has thanked: 1 time
Been thanked: 279 times

Re: Reading Type80 SMF Records file

Postby BillyBoyo » Mon Dec 26, 2011 8:49 pm

cortex wrote:[...]
(HEX – BIN)
1e-00011110
02-00000010 (1E in ASCII it's "Record Separator character" and 02 is "Start of Text"..weird or coincidence?)
00-00000000
28-00101000
7b-01111011
bc-10111100
01-00000001
11-00010001
31-00110001 (Somewhere behind this...the 0x50 had to appear, right?)
5f-01011111
d4-11010100
e5-11100101
e2-11100010
c1-11000001



Likely record ends here.

The MVS1 Robert mentions is not in the "correct" position, it is four bytes too early - looks likely that the RDW (Record Descriptor Word) is not present in your data that you have shown us. This would make this the SMF record type "02", which you are not expecting.

The 0111315F is a date, the 315th day of the 11th year of the the 2nd millenium, ie 315th day of 2011. The "F" is the sign indicator (unsigned) in a "packed" decimal field. The four bytes in front of that are a binary number indicating hundredths of a second since midnight.

Following that, again with no RDW, it looks like an SMF type 30 record.

cortex wrote:de-11011110
1e-00011110
00-00000000
00-00000000
4b-01001011
3c-00111100
01-00000001
11-00010001
31-00110001
5f-01011111
d4-11010100
e5-11100101
e2-11100010
c1-11000001
d1-11010001
c5-11000101
e2-11100010
c1-11000001
00-00000000
01-00000001
00-00000000
00-00000000
00-00000000
b0-10110000
00-00000000
26-00100110
00-00000000
01-00000001
00-00000000
00-00000000
00-00000000
d6-11010110
00-00000000
ba-10111010
00-00000000
01-00000001



I don't know if I'm dealing with big/little endian issues and, as I said, I don't know if I'm understand the file in the right way.


Regards,


So, problem number one, without the RDW you won't know where one record starts and finishes, as this contains the record-length.

Problem number two, you are getting records other than those you expect. Maybe you have your 80s somewhere, but you'd have to work very hard to find them, and it might not even be possible.

You have to go back to those who supplied the data, again requesting only the data that you require. The type 80s are variable-length records. You'd be better off by having the RDW included, as it is perfectly possible that actual data within the record could look like any delimiting character(s) for a record you are expecting (the time since midnight could give you any byte value possible). You'd have to read the entire file byte-by-byte, or in chunks of the file, to not be tripped by the time.

Even better would be for the IBM guys to give you readable data, ie all textual, then your problems probably go away.

EDIT to correct "thousandths" to "hundredths" for the time since midnight.
Last edited by BillyBoyo on Mon Dec 26, 2011 8:59 pm, edited 1 time in total.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Re: Reading Type80 SMF Records file

Postby enrico-sorichetti » Mon Dec 26, 2011 8:57 pm

I wonder why such a thoughtless requirement to process SMF data on a PC :roll:
cheers
enrico
When I tell somebody to RTFM or STFW I usually have the page open in another tab/window of my browser,
so that I am sure that the information requested can be reached with a very small effort
enrico-sorichetti
Global moderator
 
Posts: 2994
Joined: Fri Apr 18, 2008 11:25 pm
Has thanked: 0 time
Been thanked: 164 times

Re: Reading Type80 SMF Records file

Postby cortex » Mon Dec 26, 2011 9:19 pm

Hi to all, and thanks for the explanations and examples.

So..I'm sure that I was wasting my time with this file. I'll have to talk with the admin. And to be honest, I had a bad feeling about this when I didn't find that x'50'.

But looking to Robert Sample's example, I have some simple questions for you guys:

What is that smaller record in the beginning? (Seems I have it to on my file too).

I'm trying to read and understand that example by mapping the values to the correspondent field (Based on the format of type 80). I can see there the x'50'. Do I need to read that from left to right or top to down? Because x'50' it's suppose to be the 6th byte (before, it must be the RWD (4 bytes) and 1 byte for flag, right?).


Finally, I still don't know the main goal of this task of processing SMF records on a PC. I just know that we have to visualize those records, and based on its contents produce a textual report (i.e, ASCII). I heard they want to do this with many other records (102, for instance) and in the mainframe they aren't able to produce this textual representation of every record types.


Thanks again, you all!

Regards.
cortex
 
Posts: 11
Joined: Mon Dec 26, 2011 5:53 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Reading Type80 SMF Records file

Postby BillyBoyo » Mon Dec 26, 2011 9:22 pm

Big meeting. Information is needed. Boss(es) on IBM side say they have no programming capacity. Boss of unix/pc says he has a good guy twiddling his thumbs, just give us the data. Everyone agrees. Wheels start falling off instantly. Getting the correct data to the other system takes four times as long as producing a report on the mainframe and ends with a report being produced and downloaded to the unix/pc to be processed... er, printed. Ah... someone wanted it in a spreadsheet...

Curious, as I was posting this, I see the latest response...
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 265 times

Next

Return to Mainframe Security