IBM Mainframe Forum

We run a transaction processing system that uploads files from a unix server to a mainframe using NDM (Network DataMover).

The file uploads to a GDG base using a (+1), then it kicks off a job that copies it to another GDG base and then removes the initial one that was uploaded.

We are having a problem where we have two files get uploaded at the exact same second - in this case it appears that one file overwrites the other file.

One would think that the mainframe OS would see that one file was writing to the GDG base and hold the other one out or kick back a message that the GDG base is in use, and that does appear to be happening in some cases. In others it appears that it doesn't know and overwrites one file with the other.

Any suggestions on how to handle this. We have considered doing a round robin format and having more than one GDG base to upload to, but are looking for something less invasive as that would be a large coding effort.

Thanks!

The bases do not get held - the datsets themselves should be locked if they have DISP=(NEW,CATLG,DELETE). If you are deleting the upload after copying why are you using a GDG?
If your JCL is really setup correctly then the simplest solution is to have each upload use its own file. This need not be deleted after copying as the next upload will simply overwrite the contents.

GDGs and long running servers like an FTP server, and I presume an NDM server, that use dynamic allocation do not mix very well.

What happens is the first time a dynamic allocation that references a GDG the connection between the relative generation and the absolute generation is locked in for the duration of the long running function.

In other words, if you have

GDG.G0235V00
GDG.G0236V00
GDG.G0237V00

when you start

GDG(+1)

will translate to

GDG.G0238V00

If another GDG(+1) request comes in 30 minutes later (or 24 hours later, for that matter) it will still be GDG.G0238V00, not GDG.G0239V00 if the same FTP server (or NDM server) are still running.

To the best of my knowledge, though I'm not an expert on this subject, I believe that NDM places an enqueue on the GDG base when the process starts, using the same method that JES does, and keeps that enqueue until the dataset is closed. Logically, I would think that any subsequent processes using that same GDG base would be placed in the hold queue until the enqueue was released, so I find it hard to understand how this situation could happen.

But, I do agree with Nic. If you aren't using the GDG's for their real purpose as data repositories, then why use a GDG? You might as well use just regular, uniquely-named datasets. Maybe use a template where the dataset names are the same, except add a node of the process number (that's unique) to make them different enough.

I don't think that's entirely correct, MrSpock. There shouldn't be an enqueue against the GDG base, because a GDG base is nothing more than a catalog entry.

I think what's happening here is that one request is being sent to the FTP server, which dynamically allocates a new generation. Before dynamic deallocation is complete, the FTP server receives an identical request, and it interprets that request as being for the same generation it's already using.

OR... and this is what I think is more likely... the new generation isn't cataloged until the dataset is deallocated. Therefore, if you have two different threads running, the first one is dynamically allocated as GDG.G0238V00, and the next request hits the catalog, and since GDG.G0238V00 hasn't been cataloged yet, that's the DSN the catalog returns for the next generation. If that's the case, you might have to run VTOC queries to see where the uncataloged version is, because chances are it didn't get overwritten, it's just orphaned somewhere.

I don't make any claims to expertise in this area, either. But I have had some experiences with this sort of thing.

I do agree that the best solution would be to key off of some other unique property and abandon the use of GDGs for this process. Date and time stamps are usually a good place to start.

From the JCL Reference manual, section 12.19 (empahsis added):

| DISP and ENQ for generation data sets: The way the initator issues an ENQ
| to control generation data sets can be different than with other data
| sets. The initiator only issues the ENQ for the GDG base name for a
| generation data set that is referenced by either:


| Their relative GDG names (for example, DSN=TEST.GDG.DATASET(0)).

| As GDG ALLs (for example, DSN=TEST.GDG.DATASET)

| For example, the initiator issues the ENQ for the GDG base name,
| TEST.GDG.DATASET for the generation data sets shown in the list above.
| Generation data sets referenced by either their relative GDG names or as
| GDG ALLs are processed this way because the initiator does not know which
| specific absolute generation names will be required. This is because the
| conversion from relative generation name to absolute generation name is
| done during the allocation for the step referencing the data set. The ENQ
| for the specific, absolute, generation (G0000V00) data set name or names
| is issued at the start of the step requesting the relative GDG or GDG ALL.

| Note that the initiator does not issue an ENQ for the GDG base name for a
| generation data set that is referenced by its absolute GDG name. Instead
| it issues an ENQ for that specific G0000V00 data set name.

MrH wrote:... OR... and this is what I think is more likely... the new generation isn't cataloged until the dataset is deallocated.

Dynamic allocation and SMS allocation update the catalog at the time of allocation.