Looping of transaction reading messages from MQ



Support for CICS/ESA, CICS/TS & Transaction Gateway, CICS Configuration Manager and CICS Performance Analyzer

Looping of transaction reading messages from MQ

Postby Nara_513 » Thu Jun 06, 2013 11:35 pm

Hi All,

I have a production transaction issue, need suggestions for the fix :

We have transaction QQQQ which gets triggered from MQ as soon as the MQ receives a new message in it.
The transaction QQQQ reads the message using the below options :

COMPUTE MQGMO-OPTIONS = MQGMO-ACCEPT-TRUNCATED-MSG
                      + MQGMO-SYNCPOINT
                      + MQGMO-WAIT

CALL MQGET USING   MQ-CONNECTION-HANDLE,
                   MQ-OBJECT-HANDLE,
                   MQMD,
                   MQGMO,
                   MQ-MSG-LENGTH,
                   WS--PACKETS,
                   MQ-DATALEN,
                   MQ-COMPLETION-CODE,
                   MQ-REASON-CODE.



So here it clearly states that the messahe will get deleted when teh tran QQQQ executes SYNCPOINT .

After it reads the messages it do some processing of the data and it will update in the production VSAM files .

Now here comes the issue :

The transaction is looping quite often nowadays consuming high CPU . when analysed what we found is :

The transaction will be calling a program ' AAAAA' whcih updates the VSAM files, part of the data is sent by the QQQQ by placing some of the data into the Transient Storage queue and other data is sent by reference to the called program 'AAAAA' .

At times the tran is going into infinite loop as it was not able to update the VSAM files thru AAAAA .
Any update the tran do has to be done to all the files or no update should be done .So if concurrent update happens , then SYNCROLLBACK issued to undo the changes to the files.

But at atimes the SYNCROLLBACK is not functioning properly and in one of the file the record update is not undone and its getting commited . Since Rollback issued the tran again processes the message , but unable to update the files since one of the file alreadys has got a record updated in previous try , which results in duplicate record insertion .

So the AAAAA program returns control back to tran QQQQ, the code at tran QQQQ is define din such a way that , it will try for ten times to update files in case of concurrent update or duplicate record found :

PERFORM C150-UPDATE-REC. 
* This paragraph will call AAAAA which will update the MQ message details in the production VSAM files.


* Concurrent update flag is set when 
DB-ACCESS-RETURN-STATUS = CONCURRENT-UPDATE
DB-ACCESS-RETURN-STATUS = DUPLICATE-RECORD-FOUND
***File status returned by the AAAAA program

IF  WS-CONCURRENT-UPDATE-FLAG = YES  
***If concurrent update try ten times to write the data into the files . 

PERFORM D115-UPDATE-FILES
       VARYING WS-TEMP
       FROM +1 BY +1
       UNTIL WS-TEMP> WS-NUM-RETRIES
       OR    WS-CONCURRENT-UPDATE-FLAG = NOPE
ELSE
    NEXT SENTENCE.

*** After ten tries it goes to the below IF LOOP

IF  WS-CONCURRENT-UPDATE-FLAG = YES 
***If the lag is still set , then it will log the error message in another file and writes the message into error queue
    MOVE DB-ACCESS-RETURN-STATUS
      TO TRM-RETURN-STATUS
    PERFORM E900-HANDLE-ERROR
     ........
     ..........
     PERFORM LOG-ERR-MESSAGE
ELSE
   NEXT SENTENCE.

*** After that the tran is executing the SYNCPOINT command, it means that after ten tries , the message has to be removed from the message queue , but in our case the message is not removed from the queue . The transaction  is defined in such a way that it will run until it completes the processing of messages in the queue. Since the tran is reading the same message , its looping infinetely reading the same message causing high CPU utilization .

ONE REPLY IS ONE UNIT OF WORK, SO COMMIT UPDATES

EXEC CICS SYNCPOINT
          END-EXEC..




here i have few queries :

1. How a record written into the file is not undone when ROLLBACK command is issued .
2. is their a chance to write same record twice , because the problematic file is getting data from Transient storage queue .
3. Why a SYNCPOINT is not able to remove the message from the MQ . IS it possible ?

I do came up with aplan to fix this , but i feel its not optimal solution, so can u let me know what i can do to fix the issue . Please let me know in case you need any other details regarding this

Thanks for your time !!
Nara
Nara_513
 
Posts: 44
Joined: Fri Dec 02, 2011 1:11 am
Has thanked: 0 time
Been thanked: 0 time

Re: Looping of transaction reading messages from MQ

 

Re: Looping of transaction reading messages from MQ

Postby dick scherrer » Fri Jun 07, 2013 1:32 am

Hello,

What have you come up with? If you explain, we can offer comments/suggestions.

I suspect your problem is caused by increased volume causing more bottlenecks. The current design is quite prone to this happening.

There may be some alternative ways to get around this, but they depend on a better knowledge of "what is really going on". How many of these transactions will run concurrently? How many users might need to use this transactikno in a day? How many messages need to be dealt with in one iteration?
Hope this helps,
d.sch.
User avatar
dick scherrer
Global moderator
 
Posts: 6308
Joined: Sat Jun 09, 2007 8:58 am
Has thanked: 3 times
Been thanked: 90 times

Re: Looping of transaction reading messages from MQ

Postby Nara_513 » Fri Jun 07, 2013 2:55 pm

Hi Dick,

Please find the details :

The transaction QQQQ runs in four CICS regions in production(all sharing same set of files) , we have transaction TTTT (Runs in four CICS regions) which sends the request to vendor and the transaction QQQQ processes response sent by vendor . Since we have four regions , we have four specific response queues each one specific to a region where the vendor can place the response . Based up on the queue , the tran will get triggered in the corresponding region.

So , the tran QQQQ can run simultaneously in more than one region but processing different data .

My issue is the tran QQQQ is looping quite often ...where it is processing the same message from the queue and is not able to update the message details in the VSAM files .
When checked what we found is : Two issues , when tran executes ROLLBACK and SYNCPOINT --> the changes we expect as per the design is not happening.

The tran QQQQ will run untill all the messages in the queue(which triggers the tran) are processed . On an average 30,000 users will be using this tran.


Thanks Nara
Nara_513
 
Posts: 44
Joined: Fri Dec 02, 2011 1:11 am
Has thanked: 0 time
Been thanked: 0 time

Re: Looping of transaction reading messages from MQ

Postby Nara_513 » Mon Jun 17, 2013 6:47 am

Hi All,

We have worked with our IBM team and further looked into the CICS trace . What we have found is the transaction QQQQ calls a program to update two files . So because of some issue , the first file is getting updated and the second file is not getting updated so a ROLLBACK is issued as we are not able to update second file , when ROLLBACK is issued the first file content is not getting rolled back instead it is gettign committed .
Since ROLLBACK is issued, control comes to the calling tran QQQQ and the tran tries to process the message from the queue for ten times , each time it tries , a ROLLBACK is issued by the UPDATE program as it tries to write a duplicate record into the file (Cos the first file data is not undone when ROLLBACK is issued at the first instance). So after TEN tries the tran QQQQ issues a SYNCPOINT , but since a ROLLBACK happened(Update program) beofre the SYNCPOINT, the current message we are trying to process is not deleted when a SYNCPOINT is issues .

hence as per the design of the tran QQQQ, it will again read the same message from the MQ and tries to process it untill ten times(with TEN ROLLBACKS issues by update program ) and then SYNCPOINT , which is causing the infinite loop of teh transaction QQQQ.

I just have a question over here ...what exactly happens when we issue a SYNCPOINT COMMIT after a ROLLBACK .

As a work Around, we are setting up the " Harden Back Counter " for teh MQ, and doing the code changes to the tran QQQQ to read the back count value , so that in case if a ROLLBACK happens, for a couple of times, the tran QQQQ will check the back count value and remove the message from the queue from the MQ .

This solves our problem (even though the real root cause is not fixed : ROLLBACK unable to remove the inserted record into first file )

Thansk Nara
Nara_513
 
Posts: 44
Joined: Fri Dec 02, 2011 1:11 am
Has thanked: 0 time
Been thanked: 0 time

Re: Looping of transaction reading messages from MQ

Postby Ed Goodman » Mon Jun 17, 2013 7:50 pm

This is ringing a very faint little bell from way back in 1990. We had something like this and it turned out one of the program was not marked as...logged? Logable? log-using? un-logged? Something that meant it just accepted the fate of the run even if it didn't work.

This let things get out of sync. Maybe your program that is not backing out changes has this flag set. Maybe it's not really a subroutine, but a separate message with the CICS 'commit right-away' option set. (In IMS we call it EXPRESS). This would move it outside the influence of the main program's commit/rollback process.
Ed Goodman
 
Posts: 342
Joined: Thu Feb 24, 2011 12:05 am
Has thanked: 3 times
Been thanked: 17 times


Return to CICS

 


  • Related topics
    Replies
    Views
    Last post