Job having more steps



IBM's Command List programming language & Restructured Extended Executor

Job having more steps

Postby Swamy1 » Sun Oct 14, 2012 9:09 am

Hi All,
I am facing an issue while executing a job having 100 steps, Kindly please help and suggest the solution for the below performence issue. Thanks in advance for you help.
My job will have 100 steps and some of the steps using GDG files as input and output. If the job got abended we need to verify failed step and setup gdg files previous versions in the subsequent steps and other fixes, we were taking minimum 30 min to fix and restart the job from the abended step every time.
some one suggest how to over come the suitvation to reduce the amount of time to fix this issue.
1. Any REXX procedure to automate the process to start from the abended step and to use the correct files
2. Any other alternative processes or solutions for this issue.this issue.
Swamy1
 
Posts: 2
Joined: Wed Oct 10, 2012 9:52 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Job having more steps

 

Re: Job having more steps

Postby enrico-sorichetti » Sun Oct 14, 2012 11:20 am

100 steps was a poor design decision to start with
but why the abends? on a production systems the abends should be what the name implies ...
ABNORMAL situations
handled by human intervention and proper analysis

from your description they seem like a common occurrence, which implies a design/implementation flaw
wouldn' t it be wiser to fix the process rather than automating the recovery ?

the best approach would be
1) fix the <cause> of the abend
2) (1) + split the job in smaller units each one with the recovery unit

smal work units is what is commonly done in well oiled automated system
since the/any scheduler can trigger ( rough description ) ...
a job for success
a different one for failure
then, proper planning will give the desired results without writing a single line of coding

anyway the topic should be in the JCL section, not the REXX one
cheers
enrico
When I tell somebody to RTFM or STFW I usually have the page open in another tab/window of my browser,
so that I am sure that the information requested can be reached with a very small effort
enrico-sorichetti
Global moderator
 
Posts: 2643
Joined: Fri Apr 18, 2008 11:25 pm
Has thanked: 0 time
Been thanked: 130 times

Re: Job having more steps

Postby prino » Sun Oct 14, 2012 1:03 pm

  • Use a scheduler, they normally do this automagically, or
  • Pre-roll your GDG's, i.e always use (-1) as input and (0) as output, or
  • Add an initial step that copies all GDG's to normal files and finish with one step that updates all GDG's

Like Enrico said, the design is bad!
Robert AH Prins
robert.ah.prins @ the.17+Gb.Google thingy
User avatar
prino
 
Posts: 536
Joined: Wed Mar 11, 2009 12:22 am
Location: Oostende, Belgium
Has thanked: 3 times
Been thanked: 21 times

Re: Job having more steps

Postby dick scherrer » Mon Oct 15, 2012 7:43 am

Hello and welcome to the forums,

While our major intention is to help, we often must post replies that are negative. . .

Echoing the observations of the others, someone really needs to address the design of this process.

The "best" way to do this (imho) is thru the system scheduling software. One successful process should cause the next to run and so on. Properly setup, restart processes are handled as well.

More importantly, someone needs to fix the existing problems so they do not recur. There is NO good reason to have regular abends / restarts. This speaks to poor design, poor implementation, and very poor testing and promotion practices.

If you / your organization decide to fix this, people will be here that will have suggestions.
Hope this helps,
d.sch.
User avatar
dick scherrer
Global moderator
 
Posts: 6304
Joined: Sat Jun 09, 2007 8:58 am
Has thanked: 3 times
Been thanked: 91 times

Re: Job having more steps

Postby Swamy1 » Wed Oct 17, 2012 9:53 pm

Hi,
I am totally not agreeing with your comments. My project is very huge application and will have multiple flavors, before promoting any code to production will go thru different test cycles in different environments. For every release we need to process thru the job(100 steps) in the test environments. I am expecting your inputs how to automate the process and to reduce the time to fix the issues occurring during the batch cycles. Always welcome for your valuable thoughts and suggestions.
Thanks
Swamy
Swamy1
 
Posts: 2
Joined: Wed Oct 10, 2012 9:52 pm
Has thanked: 0 time
Been thanked: 0 time

Re: Job having more steps

Postby BillyBoyo » Wed Oct 17, 2012 9:55 pm

Well, I think you'll find it pretty unanimous. I can count on the fingers of one nose the number of times I've seen a JOB with 100 steps. And there are reasons for that, and they've been set out.

If you want to sail against the wind, so be it, but I don't think you'll find so much good advice going in that direction.
BillyBoyo
Global moderator
 
Posts: 3804
Joined: Tue Jan 25, 2011 12:02 am
Has thanked: 22 times
Been thanked: 264 times

Re: Job having more steps

Postby enrico-sorichetti » Wed Oct 17, 2012 10:05 pm

(1) I am totally not agreeing with your comments.
My project is very huge application and will have multiple flavors, before promoting any code to production will go thru different test cycles in different environments. For every release we need to process thru the job(100 steps) in the test environments.
(2) I am expecting your inputs how to automate the process and to reduce the time to fix the issues occurring during the batch cycles.
(3) Always welcome for your valuable thoughts and suggestions.


(1) since you asked we replied with the best of out multidecennal IT experience on large projects
if You do not agree on /You not like on the answers You get, just ask somewhere else , we do not care :evil:
if You want to receive only the answers You like/agree on hire a paid consultant who for the proper fee
will be glad to feed with any horse manure you are willing to pay for .

(2) You are not in the position to expect anything
we reply on our time and free of charge
and above all based on experience that You seem to lack
so You can NOT tell us how/what to reply

statement (3) is in contrast with statement (1)
cheers
enrico
When I tell somebody to RTFM or STFW I usually have the page open in another tab/window of my browser,
so that I am sure that the information requested can be reached with a very small effort
enrico-sorichetti
Global moderator
 
Posts: 2643
Joined: Fri Apr 18, 2008 11:25 pm
Has thanked: 0 time
Been thanked: 130 times

Re: Job having more steps

Postby Robert Sample » Wed Oct 17, 2012 10:35 pm

What you are asking for is considerably beyond the scope of this -- or any -- forum. You need to convince your organization to hire a consultant to help, or you need to do enough reading / learning to do the task on your own. Depending upon your site and the tools available there, it may -- or may not -- even be possible to accomplish what you asked; a consultant could help you determine what is feasible.
Robert Sample
Global moderator
 
Posts: 3367
Joined: Sat Dec 19, 2009 8:32 pm
Location: East Dubuque, Illinois
Has thanked: 1 time
Been thanked: 222 times

Re: Job having more steps

Postby Peter_Mann » Wed Oct 17, 2012 11:23 pm

While I'll agree 100% that the process is flawed, it's not impossible to handle restarts without a scheduling system, since I'll assume the 100 step job you run is NON-PRODUCTION, thus prolly cannot run from a production scheduler, but most sites, with a scheduler, have a restart managment tool that is tightly coupled with the scheduler and can, in most instances be used stand-alone, like CA-11 (RMS) for instance, you''ll need to question the support folks at your site
1) is there a RMS tool like CA-11?
2) are you authorized to use the tool?
3) how do I invoke such a tool?
do some research at your site, ask questions there first.
Peter
Peter_Mann
 
Posts: 139
Joined: Fri Jun 24, 2011 7:37 pm
Location: Lowell,AR
Has thanked: 14 times
Been thanked: 3 times

Re: Job having more steps

Postby dick scherrer » Wed Oct 17, 2012 11:32 pm

Hello,

I am totally not agreeing with your comments
Well, OK . . . You don't have to agree.

However, most of us have worked on systems that are most likely exponentially larger than your system, it would be well for you to consider how we have successfully done this kind of thing and have not gotten to where you are now.

For every release we need to process thru the job(100 steps) in the test environments
Yup, be a very bad thing is each release was not completely tested. But there is NO business (or technical) reason to run 100 step jobs.

Indeed, if something smaller than a complete release is to be promoted (problem fix, legal requirement, etc) it would be far more efficient to only test/promote what was changed. If "everything is being released, by all means, promote all of the individual processes.

And, yes, most of us "old guys" were implementing "monster" batch jobs when there was no scheduling software. All of the restarts had to be accomodated either in the job itself and/or with special restart process. Which is why any of us who can, use the scheduling systems whenever possible. This also reduces/eliminates the calls in the middle of the night because the :monthly" went down.

That being said, my main concern is still the number of "things" that "go bump in the night" and require all of this crisis intervention.
Hope this helps,
d.sch.
User avatar
dick scherrer
Global moderator
 
Posts: 6304
Joined: Sat Jun 09, 2007 8:58 am
Has thanked: 3 times
Been thanked: 91 times


Return to CLIST & REXX