Performance improvement is based upon the idea that a computer program either runs I/O-bound (waiting for I/O to complete), or CPU-bound (waiting for CPU time to complete). A CPU-bound program can have I/O reduced almost to nothing and the elapsed time changes little, if any, as the I/O is not the bottleneck. An I/O-bound program can be reduced to almost no CPU time without changing the elapsed time. If the program is CPU-bound, you improve performance by making code changes because the code is almost always what is taking the time -- tables and data base access are often causative agents for CPU consumption. If the program is I/O-bound, you improve performance by making changes to reduce I/O -- increasing buffers (30 to 50 per sequential file, appropriate numbers for other file types), moving tape processing to disk, using cache to replace disk access.
If you do not know whether the program is I/O-bound or CPU-bound, all you can do is guess when attempting to improve its performance. Start by adding buffers and making other I/O changes -- if there is no significant change, then assume you're going to have to dig into the code to figure out where the program is spending the most time. If you can shift the program run time to an less busy part of the day, that may be a major improvement.
However, first you must determine what is acceptable run time. This can vary by site and sometimes even by application. You cannot expect a program to be able to read 25,000,000 records, for example, in 2 minutes -- but neither should it take 13 hours (although I have worked on systems where that was true).