Taming Batch Operations Using Perl: A Tale of One Application, a Group of Users, Two Languages, and a Few Tools

• By Philipp Janert
• Apr 13, 2007

Like this article? We recommend
Art of UNIX Programming, The

A while back, I did some consulting with a group of technical specialists working for a medium-sized company. This group performed certain calculations for the company—or rather, the calculations were performed by an old, old piece of software which we will call BRP/P. This is a story of BRP/P and some of the people working with it. (Some details have been changed to protect the innocent—and the guilty.)

Working with BRP/P

Although it was now running on Unix, BRP/P originated on a mainframe and never seemed to have adapted fully to its new environment. For instance, its data-input format was 80-character, FIXED-FORMAT (that is, the month will be encoded in columns 17–19, and so on), but at least it no longer required paper punch cards.

Stemming from a mainframe, BRP/P had no notion of an "interactive session," of course: all processing was batch, period—but the classic kind of batch job, in which you deliver your deck of punch cards the evening before to the system operator and (hopefully) pick up your results the next morning at the window. This being the 21st century, we did away with the punch cards and the sysops, but the spirit of the ’60s was still very much alive and well.

BRP/P read its instructions from program files, which ended (reasonably enough) in the extension .BRP. However, rather than printing to the console, it generated not one, but two output files: one contained the results (if any) of the calculation; the other one contained mostly logging information. Any error messages and warnings were also printed to this file, together with the (copious!) logging info. Both of these files were automatically named by BRP/P: the basename was the same as the basename of the program (that is, the .BRP) file, but the extension was different: the results file had the extension of .OUT, while the logging file had the extension of .OUCH.

These two files were a source of enduring mirth. First of all, BRP/P would not overwrite (and thereby clear) the .OUT file on startup; it would generate a new .OUT file only when it actually had results to show. In case it encountered an error, it would whisper this to the .OUCH file while leaving the old .OUT file standing around. Checking only the .OUT file after a run was therefore not enough—you never knew whether your program had actually completed! So, to examine the result of any program modification required first a careful scan of (several pages of) .OUCH, before one would trust to look at the actual results themselves. (Interactive shell users will note the particularly pernicious naming convention, too, rendering tab-completion almost maximally useless: the first tab completes the basename; the second tab gets us to myfile.OU?, requiring another key stroke and tab to complete!)

Coming from the Unix environment, I experienced working with BRP/P as incredibly painful (somebody described it as "scraping the paint off the walls with your fingernails"), but it took a few days to realize precisely what made it so painful. Once I recognized the lack of interactivity, the path to a solution was clear: provide a little Perl wrapper, taking the name of a BRP/P program file as parameter. The wrapper would run BRP/P and then scan the log file for errors, printing them to the screen. Only if no errors were found would it read the .OUT file, paging its contents. The first version of the wrapper (called doBRP) was up within a couple of hours. Although in this early incarnation it was flaky and offered few creature comforts, it was met with enthusiasm by my coworkers—enough to encourage further development!