Mac OS X Unleashed

Mac OS X Unleashed

By John Ray and William C. Ray

Automating Tasks with Shell Scripts

With as many times as we've mentioned how powerful shell scripting can be and how much time and effort it can save you, you might be expecting that writing shell scripts is going to require dealing with some additional level of complexity on top of what you've already learned. Shell scripts are simple programs that you write in the language of the shell, and if you've made it this far in the book, you've been learning and working in the language of the shell for a few chapters now. If you consider this fact, and the notion that Unix, by design, attempts to abstract the notion of input and output so that everything looks the same to the OS, you might have a good guess at what we'll say next: That's right—you already know how to write shell scripts. There are a few more shell techniques that you can learn to enhance your ability to program the shell, but Unix itself doesn't care whether it's you typing at a command prompt or commands being read out of a file on disk. Everything you've typed so far in working with the shell could have been put in a file, and the computer could have typed it to itself—voila, a shell script.

At its most trivial, a shell script can be exactly what you type at a prompt to accomplish some set of tasks. If you find that you have a need to repeatedly execute the same commands over and over, you can type them once into a file, make that file executable, and forever after execute them all just by typing the name of the file.

It really is as simple as it sounds, but just in case it's not quite clear yet, an example should help. Consider the following situation: Let's say that every day when you log in to your computer, you like to check the time (with date), check to see who's online (using the who command), check to see how much space is left on the drive with your home directory (with df), and finally check who's most recently sent you mail (with from).

You could type in each of these things to a command prompt when you log in to your machine, or you could put them in a file, make it executable, and let the file "type" them for you.

soyokaze ray 226> date

     Mon Jun 18 23:35:55 EDT 2001

soyokaze ray 227> who

     joray    ttyp0   Jun 14 18:22   (140.254.12.151)
     ray      ttyp1   Jun 18 21:49   (24.95.74.211)
     ray      ttyp2   Jun 15 10:00   (rodan.chi.ohio-s)
     radman   ttyp3   Jun 18 23:33   (ac9d3e22.ipt.aol)

soyokaze ray 228> df . 

     Filesystem            kbytes    used   avail capacity  Mounted on
     /dev/sd2g             953619  846078   12180    99%    /priv

soyokaze ray 229> from | tail -10

     From vanbrink@home.ffni.com  Mon Jun 18 16:20:23 2001
     From billp@abraxis.com  Mon Jun 18 17:28:33 2001
     From douglas_mille70@hotmail.com  Mon Jun 18 18:34:28 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 19:23:42 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 20:42:53 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 21:24:00 2001
     From buckshot@wcoil.com  Mon Jun 18 22:02:15 2001
     From jray@poisontooth.com  Mon Jun 18 22:28:56 2001
     From jray@poisontooth.com  Mon Jun 18 23:15:28 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 23:34:43 2001

soyokaze ray 230> cat > imhere
#!/bin/csh

date
who
df .
from | tail -10
soyokaze ray 231> chmod 755 imhere
soyokaze ray 232> imhere

     Mon Jun 18 23:36:51 EDT 2001
     joray    ttyp0   Jun 14 18:22   (140.254.12.151)
     ray      ttyp1   Jun 18 21:49   (24.95.74.211)
     ray      ttyp2   Jun 15 10:00   (rodan.chi.ohio-s)
     Filesystem            kbytes    used   avail capacity  Mounted on
     /dev/sd2g             953619  846078   12180    99%    /priv
     From billp@abraxis.com  Mon Jun 18 17:28:33 2001
     From douglas_mille70@hotmail.com  Mon Jun 18 18:34:28 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 19:23:42 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 20:42:53 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 21:24:00 2001
     From buckshot@wcoil.com  Mon Jun 18 22:02:15 2001
     From jray@poisontooth.com  Mon Jun 18 22:28:56 2001
     From jray@poisontooth.com  Mon Jun 18 23:15:28 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 23:34:43 2001
     From owner-c-r-ffl@serge.shelfspace.com  Mon Jun 18 23:36:48 2001

As you can see, executing the file imhere, containing my commands, produces essentially the same output, with much less typing. (The output has a few changes because one user has left the system, and new mail has arrived between the by-hand runs and the execution of the shell script.)

The only part of the imhere script that might be confusing is the first line, #!/bin/csh. The shell interprets the first line of a shell script in a special manner. If a pattern such as this is found #! <path to an executable file> , the executable file named in that line is used as the shell for executing the contents of the script.

Note

The C shell, csh, and the Bourne shell, sh, are probably the best shells for you to write shell scripts in. Some shells offer somewhat more power in their scripting ability, but csh and sh are the only shells that can be considered ubiquitous. This might not be a concern, if you never intend your scripts to run anywhere but on your own personal machine. But if you think you might ever use another machine, or are interested in distributing your scripts to other people, you can't rely on any specialty shells being available.

Tip

In many scripting languages—tcsh/csh shell scripting being no exception—anywhere that a # appears indicates that the rest of the line is a comment. A #! on the first line is a special comment to the system, indicating which program is the intended interpreter for the script contained in the file.

Single-Line Automation: Combining Commands on the Command Line

Before we go too far with the notion of storing collections of commands in files, however, let's look at what can be done at just the command-line level. You already know about using pipes and variables. These concepts can be combined to produce very powerful expressions directly at the command line, without any need to store the collection of commands in a file.

Consider for a moment the netpbm collection of graphics manipulation programs that was installed in Chapter 17, "Troubleshooting Software Installations, and Compiling and Debugging Manually." Included in the capabilities of the suite are a number of conversions among various file formats, as well as a range of manipulations of the image content itself. With Mac OS, if you want to convert a GIF file into a PICT file, and convert it to four-color grayscale along the way, you have a number of options. You could fire up PhotoShop or GraphicConverter, and perform the changes there, and save thefile. Alternatively, you could program a conversion filter in DeBabelizer to perform this manipulation for you. With netpbm, you can perform the manipulation from the command line:

soyokaze ray 277> ppmtogif < john.ppm > john.gif

     ppmtogif: computing colormap...
     ppmtogif: 192 colors found

soyokaze ray 278> giftopnm < john.gif > john.pnm
soyokaze ray 279> ppmtopgm < john.pnm > john.pgm
soyokaze ray 280> ppmquant 4 < john.pgm > john.pgm2

     ppmquant: making histogram...
     ppmquant: 120 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...

soyokaze ray 281> ppmtopict < john.pgm2 > john.pict

     ppmtopict: computing colormap...
     ppmtopict: 4 colors found

Figure 18.1 shows a comparison of the original image john.gif, and the four-color grayscale image, john.pict.

18fig01.jpg

Figure 18.1 A comparison of an original file and the result of processing it through one of a number of different netpbm filters.

Note

The netpbm suite provides most of its manipulation facilities on an internal format, variably named .ppm, .pgm, or .pnm. It provides a number of filters that can read data into this format, a number of filters that can act upon and modify the contents of files in this format, and a number of filters that can output into other file formats. Together, these facilities allow a wide range of image formats to be read, manipulated, and written. Additionally, the file format is well documented and simple for a programmer to write code to parse. This makes it easy for a programmer to write her own filters to perform any manipulations that the provided software cannot. See the netpbm man pages, if you've installed this suite, for considerably more information on the use of netpbm.

From the brief discussion at the beginning of this section, you should already have an idea of how you could combine all that into a single file, if for some reason you wanted to perform that conversion to the john.gif file over and over and over.

This seems to be not very useful a thing to automate, and quite a bit of typing to boot (although, frankly, not nearly as much work as starting up Photoshop to do something this simple!). Let's see what we can do with pipes and shell variables to cut down on the amount of typing.

First, observe that all the programs are taking the input files on STDIN, and are producing output on STDOUT. Unix command-line programs are frequently like this, and it's a very good thing. Using the power of pipes to connect one program's STDOUT to another program's STDIN, we can shorten that collection of commands to a single command line:

soyokaze ray 287> giftopnm < john.gif | ppmtopgm | ppmquant 4 | ppmtopict > john.pict

     ppmquant: making histogram...
     ppmquant: 120 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...
     ppmtopict: computing colormap...
     ppmtopict: 4 colors found

I'll let you verify that the output is graphically identical on a file of your own.

You might think that it's probably not very likely that you'll want to perform this single manipulation repeatedly to the same image. However, there are many times when you'd like to be able to perform a collection of manipulations like that on a number of different images. With what you know about shell variables, you might be able to come up with a way to abstract that command line so that it could be reused for any GIF file. You might try something like this:

soyokaze ray 288> set infile=john.gif
soyokaze ray 289> giftopnm < $infile | ppmtopgm | ppmquant 4 | ppmtopict > $infile:r.pict

     ppmquant: making histogram...
     ppmquant: 120 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...
     ppmtopict: computing colormap...
     ppmtopict: 4 colors found

Note

Note how I use the :r modifier to the shell variable $infile to remove the .gif suffix, and replace it with my new .pict suffix.

Now, you could simply use new values for $infile, and you'd have a reusable command that could perform the same manipulation on any GIF image.

It's still too much work though, right? Well, remember aliases? We can further automate things by using an alias command to compact that large command-line expression into something more manageable.

soyokaze ray 290> alias greyconvert 'set infile=\!#:* ; giftopnm < $infile | ppmtopgm |

      ccc.gif
    ppmquant 4 | ppmtopict > $infile:r.pict '
soyokaze ray 291> greyconvert john.gif

     ppmquant: making histogram...
     ppmquant: 120 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...
     ppmtopict: computing colormap...
     ppmtopict: 4 colors found

Note

Notice how I've used the history substitution \!#:* to get the arguments passed into the alias, and then put those arguments (although I only expect one, a filename) into $infile?

That command is getting pretty long, isn't it? The good news is that for almost any task in Unix, you can figure out how to build up to an expression like this, just as shown here. Start by figuring out how to do it one step at a time on the command line, and work your way up to an elegant solution that solves the problem for you with as little repetitive work as necessary.

After you've invented useful aliases such as this one for yourself, remember to store them in your .cshrc or .tcshrc file in your home directory, or in your aliases.mine file in ~/Library/init/tcsh/, so that you can use them again, whenever you log in to your computer.

Multi-Line Automation: Programming at the Prompt

Creating customized commands that perform special functions like the greyconvert command built in the previous sections is useful, but it still doesn't address the need to automate tasks. For that, we need some sort of looping command, and the tcsh shell offers two: the foreach command and the while command. Both commands repeat a block of shell commands. The first executes it "for each" of its arguments, and the second executes it "while" some condition is true.

The foreach command and while command of the shell are unlike other shell commands that you've become familiar with in that they require additional information beyond the first command line. For example, the syntax for the foreach command is

foreach <variablename> ( <item list> )
  <first command to execute>
  
   <second command to execute>
  .
  .
  .
  <nth command to execute>
end

The while command, on the other hand, has the syntax

while ( <comparison> )
  <first command to execute>
  <second command to execute>
  .
  .
  .
  <nth command to execute>
end

In the foreach command, the <item list> can be a space-separated list of items, or a command that produces a space-separated (or return-separated) list of items, or a command-line wildcard that matches a list of files. As a demonstration, consider a situation in which we want to execute our previous greyconvert command on every GIF file in a directory containing many files. This can be accomplished in several ways by the use of the foreach command. The following example demonstrates the wildcard match to all the files of interest:

localhost amg 246% ls

     AMG_cal-cover.gif         AhMyGoddess-v05.gif       AhMyGoddess-v10-f1.gif
     AhMyGoddess-v01-f1.gif    AhMyGoddess-v06-f1.gif    AhMyGoddess-v10-i1.gif
     AhMyGoddess-v01.gif       AhMyGoddess-v06.gif       AhMyGoddess-v10-i2.gif
     AhMyGoddess-v02-f1.gif    AhMyGoddess-v07-f1.gif    AhMyGoddess-v10-i3.gif
     AhMyGoddess-v02.gif       AhMyGoddess-v07.gif       AhMyGoddess-v10.gif
     AhMyGoddess-v03-f1.gif    AhMyGoddess-v08-f1.gif    amg-nt0694_cover.gif
     AhMyGoddess-v03.gif       AhMyGoddess-v08.gif       amg-nt0694_i1.gif
     AhMyGoddess-v04-f1.gif    AhMyGoddess-v09-f1.gif    amg-nt0694_i2.gif
     AhMyGoddess-v04.gif       AhMyGoddess-v09.gif
     AhMyGoddess-v05-f1.gif    AhMyGoddess-v10-b.gif

localhost amg 247% foreach testfile ( *.gif )
foreach -> greyconvert $testfile
foreach -> end

     ppmquant: making histogram...
     ppmquant: 165 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...
     ppmtopict: computing colormap...
     ppmtopict: 4 colors found
     ppmquant: making histogram...
     ppmquant: 159 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...
     ppmtopict: computing colormap...
     ppmtopict: 4 colors found
     .
     .
     .
     ppmquant: making histogram...
     ppmquant: 163 colors found
     ppmquant: choosing 4 colors...
     ppmquant: mapping image to new colors...
     ppmtopict: computing colormap...
     ppmtopict: 4 colors found

localhost amg 248% ls

     AMG_cal-cover.gif         AhMyGoddess-v05-f1.pict   AhMyGoddess-v10-b.gif
     AMG_cal-cover.pict        AhMyGoddess-v05.gif       AhMyGoddess-v10-b.pict
     AhMyGoddess-v01-f1.gif    AhMyGoddess-v05.pict      AhMyGoddess-v10-f1.gif
     AhMyGoddess-v01-f1.pict   AhMyGoddess-v06-f1.gif    AhMyGoddess-v10-f1.pict
     AhMyGoddess-v01.gif       AhMyGoddess-v06-f1.pict   AhMyGoddess-v10-i1.gif
     AhMyGoddess-v01.pict      AhMyGoddess-v06.gif       AhMyGoddess-v10-i1.pict
     AhMyGoddess-v02-f1.gif    AhMyGoddess-v06.pict      AhMyGoddess-v10-i2.gif
     AhMyGoddess-v02-f1.pict   AhMyGoddess-v07-f1.gif    AhMyGoddess-v10-i2.pict
     AhMyGoddess-v02.gif       AhMyGoddess-v07-f1.pict   AhMyGoddess-v10-i3.gif
     AhMyGoddess-v02.pict      AhMyGoddess-v07.gif       AhMyGoddess-v10-i3.pict
     AhMyGoddess-v03-f1.gif    AhMyGoddess-v07.pict      AhMyGoddess-v10.gif
     AhMyGoddess-v03-f1.pict   AhMyGoddess-v08-f1.gif    AhMyGoddess-v10.pict
     AhMyGoddess-v03.gif       AhMyGoddess-v08-f1.pict   amg-nt0694_cover.gif
     AhMyGoddess-v03.pict      AhMyGoddess-v08.gif       amg-nt0694_cover.pict
     AhMyGoddess-v04-f1.gif    AhMyGoddess-v08.pict      amg-nt0694_i1.gif
     AhMyGoddess-v04-f1.pict   AhMyGoddess-v09-f1.gif    amg-nt0694_i1.pict
     AhMyGoddess-v04.gif       AhMyGoddess-v09-f1.pict   amg-nt0694_i2.gif
     AhMyGoddess-v04.pict      AhMyGoddess-v09.gif       amg-nt0694_i2.pict
     AhMyGoddess-v05-f1.gif    AhMyGoddess-v09.pict

In this example, the foreach testfile ( *.gif ) line could have been replaced, with the following variants, with identical results:

foreach testfile ( `ls *.gif` )

foreach testfile ( AMG_cal-cover.gif AhMyGoddess-v01-f1.gif ... amg-nt0694_i2.gif )

Note

The second variant should be understood to contain a list of all GIF files in the directory, not just the three shown. The presence and direction of the single backquotes in the first line are also critical. Single backquotes around a command on the command line, if you remember, cause that command to be executed and its results substituted into the current command line in the place of the quoted command.

The while command works similarly, executing its code block while some <condition> holds:

localhost ray 225> set x = 10
localhost ray 226> while ( $x > 0 )
while -> echo $x
while -> @ x = ( $x - 1 )
while -> end

     10
     9
     8
     7
     6
     5
     4
     3
     2
     1

localhost ray 227> echo $x

     0

This example obviously doesn't have much day-to-day applicability. Most while expressions that are actually useful do things like watch for particular events to occur, such as the existence of temporary files, or disk space usage of more than or less than some value. None of these is particularly easy to demonstrate in a text-only format such as a book, but we expect that you'll get the idea fairly quickly. Table 18.7 shows the conditional operators that can be used to construct the <condition> part of while loops and if conditional statements.

Table 18.7. Logical, Arithmetical, and Comparison Operators Comparison Operators

Operator or Symbol Function
|| Boolean OR arguments.
&& Boolean AND arguments.
| Bitwise Boolean OR.
^ Bitwise Exclusive OR.
& Bitwise Boolean AND.
== Equality comparison of arguments ($x == $y is true if the value in $x equals the value in $y).
  Compares arguments as strings.
!= Negated equality comparison of arguments ($x != $y is true if the value in $x is not equal to the value in $y).
  Compares arguments as strings.
=~ Pattern-matching equality comparison (matches shell wildcards).
  Compares arguments as strings.
!~ Pattern-matching negated equality comparison.
  Compares arguments as strings.
<= Less than or equal to.
>= Greater than or equal to.
< Less than.
> Greater than.
<<

Bitwise shift left. To avoid the shell interpreting this as redirection, it must be in a parenthesized subexpression.

For example,

set y = 32;
@ x = ( $y << 2 )
>> Bitwise shift right. See preceding comment.
+ Add arguments.
- Subtract arguments.
* Multiply arguments.
/ Divide arguments.
% Modulus operator (divide and report remainder).
! Negate argument.
~ Ones complement of argument.
( Open parenthesized subexpression for higher-order evaluation.
) Close parenthesized subexpression for higher-order evaluation.

Another common use for the while command is to create infinite loops in the shell. This is a way to do things such as cause a shell command to execute over and over, potentially creating something like a "drop directory" that automatically processes files that are copied to it. For example, if we want to create a directory into which we could copy GIF files, and any files copied into it would have the greyconvert process run on it automatically, and then the GIF files would be deleted, we might try something like the following:

localhost Pictures 243> while (1)
while -> foreach testfile (*.gif)
while -> greyconvert $testfile
while -> rm $testfile
while -> end
while -> sleep 60
while -> end

Note

The first four lines, with the while-> prompt, are actually part of the foreach command, but it doesn't display its prompt while the while-> prompt is active.

This while command attempts to loop perpetually (the value of 1 is true for the purposes of a comparison expression), and to internally execute our previous foreach loop to convert any files that match the *.gif pattern in the current directory. An rm command has been added to the foreach loop to remove the GIF file after it has been converted. The sleep 60 command after the foreach command's end causes the while loop to pause for 60 seconds before going on to its end statement and re-looping to the top of the while.

Unfortunately, this does not quite work properly, as when there are no files that match *.gif, the foreach line fails without creating its loop, and the end on line 4 is mistaken as intended to end the while loop.

Thankfully, you can work around this problem by applying the final topic in our discussion of shell scripts.

Storing Your Automation in Files: Proper Scripts

With all the power available to you directly at the command line, the move to putting shell scripts in files should seem almost anticlimactic in its lack of complexity. As mentioned at the beginning of this chapter, anything that you can type on the command line, you can put in a file, and the system will quite happily execute it for you, if you make the file executable. It really is that simple. There'd be little more to say, except that putting your script in a file allows you to conveniently separate parts of the execution into separate shells, preventing conflicts such as those just demonstrated earlier with the while and foreach loops.

Any script that is put in a file and directly executed (rather than source, in your current shell) creates its own shell in which to execute. To use this to make the previous example function properly, we can put the foreach section of the command into its own file:

localhost Pictures 244>cat > greyconv.csh

     #!/bin/csh
     foreach testfile (*.gif)
     greyconvert $testfile
     rm $testfile
     end
localhost Pictures 245> chmod 755 greyconv.csh
localhost Pictures 246> ls -l greyconv.csh

     -rwxr-xr-x  1 ray  staff  75 Jun 23 01:58 greyconv.csh

localhost Pictures 247> cat greyconv.csh

     #!/bin/csh
     foreach testfile (*.gif)
     greyconvert $testfile
     rm $testfile
     end

Then our perpetual while command can be run as

localhost Pictures 248> while (1)

     while -> ./greyconv.csh
     while -> sleep 60
     while -> end

Note

Remember to chmod the script file so that it's executable. The shell won't let you run the script if it's not set to be executable.

This command will loop perpetually in the current directory, executing the greyconv.csh shell script every 60 seconds. Any file with a .gif extension that is placed in the directory will be passed through the greyconvert alias that we created earlier, and then the original file will be deleted. This will run perpetually in the directory, allowing any file dropped in to be converted (within 60 seconds) automatically. Because the end of the foreach is in a completely separate shell, it can't accidentally end the while, and everything will work as expected. Of course, if there were a reason to do this on a regular basis, you could put that while loop into its own shell script file. It could be stored and executed as a single command from the command line just like any other command.

Note

Particularly astute readers will observe that the script does not attempt to make sure that the whole GIF file has been copied into the directory before it executes upon it. You might also expect that the behavior of the script could be rather unpredictable if the greyconv.csh script takes longer than a minute to execute. Shell scripts are generally an exercise in successive refinement to eliminate potential problems such as these, and these examples should be considered nothing more than the first step to a final application. You can get quite a bit of functionality out of simple scripts such as this. But if you're inclined to tackle the more complicated "correct" solutions, we can't recommend reading the man pages and checking out some books specifically on shell scripting strongly enough.

Two final things to note because we've now introduced independent shells invoked as the result of placing scripts in files: the $argv[1]...$argv[ n ] and corresponding $1...$ n command-line argument variables. (Refer to the table of shell variables and the table of alternative variable addressing methods for a refresher on these.) This also allows us to introduce the if ( <comparison> ) shell command, allowing conditional execution of code blocks.

If we'd like to make the greyconv.csh script somewhat more general, we can make use of the ability to pass arguments into the script. For example, it would be nice to be able to use greyconv.csh on the GIF files in a directory, without actually having to be in the directory when we run the while loop. This is easily accomplished by using the shell argument variables. The modified version of greyconv.csh is as follows:

#!/bin/csh

if ( $?1 == 1 ) then
cd $1
endif

foreach testfile (*.gif)
greyconvert $testfile
rm $testfile
end

This version of greyconv.csh demonstrates both an if conditional expression, and the use of a command-line argument. The if statement checks whether the variable $1 is set (using the $? <variablename> alternative variable addressing to check for existence). If it is set, the script assumes that the value in $1 is the name of a directory, and it cds into that directory before executing the foreach loop to convert and delete the GIF files. gre y conv.csh can now be called with a directory, to operate in that directory, or without a directory specified, to operate in the current directory.

We hope that gives you a few ideas for how powerful shell scripts can be, and how you can use them to make your use of OS X much more productive. We've only just scratched the surface in this chapter, and have trivialized some explanations to their simplest case to avoid a chapter that takes half the book. What's here can get you quite a way into shell scripting, and many Unix users with years of experience don't use more than a fraction of what's covered here. Still, if you're looking for more power and more capabilities, don't hesitate to go to the man pages and shell programming–specific reference books.

Share ThisShare This

Informit Network