- Got Storage?
- Getting Ready to Receive the Files
- Learning && Trying Rsync
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Learning && Trying Rsync
It’s time to get to the good stuff: those CD images for Whitebox! Let’s begin by reviewing the manpage for rsync. Let’s create a copy:
´man rsync | col –b > rsyncmanpage.wri´
This command takes the heavily formatted manpage information, strips obnoxious characters, and then sends the text to a new file. The wri extension tells Windows to open the file with WordPad/write.exe, an editor that understands UNIX line endings. In fact, because the Cygwin shell can execute Windows binaries, run this command:
´write.exe rsyncmanpage.wri´
There! Now you have a copy of the documentation available as you work through the complex commands. What? You didn’t get this far? Missing rsync or col? At this point, you need to run the Cygwin setup.exe and ensure you have the tools you need.
As you review your rsync manual, you see it supports both rcp and URL syntax for connections. Because very few people know rcp syntax, given it is an accursed tool of uncertain and troubled security lineage, not unlike Sauron’s ring, let’s focus on using rsync in a simpler way.
The whiteboxlinux.org website points out the images can be retrieved a few rsync repositories. I’m going to use rsync://ftp.esat.net/mirrors/whiteboxlinux.org. Let’s start by listing files, much like an FTP site. We need to descend into directories, too. I will use the --list-only parameter to see the files and directories within the site name. I will end the URL with a directory marker / to indicate I want to see directory contents.
rsync --list only rsync://ftp.esat.net/mirrors/whiteboxlinux.org/ drwxr-xr-x 38 2005/05/04 18:44:22 . drwxr-xr-x 15 2003/12/15 03:05:06 3.0 drwxr-xr-x 15 2005/05/04 18:44:42 4 drwxr-xr-x 21 2005/05/05 19:58:45 contrib
You now see the directories for the versions. Let’s try version 4. That means we add a 4, the directory name, at the end of our command.
rsync --list-only rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/ drwxr-xr-x 15 2005/05/04 18:44:42 . drwxr-xr-x 91 2005/05/04 18:45:27 en
Now it’s time to select the en or English version, so add en/ to your command.
rsync --list-only rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/en/ drwxr-xr-x 91 2005/05/04 18:45:27 . drwxr-xr-x 59 2005/05/24 15:15:36 debuginfo drwxr-xr-x 35 2005/05/24 15:15:02 extras drwxr-xr-x 43 2005/05/05 19:59:11 iso drwxr-xr-x 6 2005/05/04 18:45:01 obsolete-updates drwxr-xr-x 30 2005/05/04 18:45:57 os drwxr-xr-x 95 2005/05/24 17:03:38 updates
We want the ISO/CD images, so add that and proceed:
$ rsync --list-only rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/en/iso/ drwxr-xr-x 43 2005/05/05 19:59:11 . drwxr-xr-x 4096 2005/05/05 08:33:52 i386 drwxr-xr-x 4096 2005/05/05 20:00:24 source drwxr-xr-x 4096 2005/05/05 20:00:01 x86_64
Got a dual-core chip? Want to use gcc to compile from source? Confused by my questions and just want to run a simple copy of WhiteBox? Just navigate into the i386 directory:
$ rsync --list-only rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/en/iso/i386/ drwxr-xr-x 4096 2005/05/05 08:33:52 . -rw-r--r-- 651507712 2005/05/04 19:56:25 manifestdestiny-binary-i386-1.iso -rw-r--r-- 668731392 2005/05/04 19:57:23 manifestdestiny-binary-i386-2.iso -rw-r--r-- 656910336 2005/05/04 19:58:20 manifestdestiny-binary-i386-3.iso -rw-r--r-- 364388352 2005/05/04 19:58:51 manifestdestiny-binary-i386-4.iso -rw-r--r-- 508 2005/05/04 19:58:51 manifestdestiny-binary-i386-md5sums
Ah! There they are, the ISO images we need. We need to transfer these to our repository. Now let’s run the command needed to transfer the md5sum file.
grep 1.iso manifestdestiny-binary-i386-md5sums && md5sum ./manifestdestiny-binary-i386-1.iso 39131c4e570ac8368b7819b3fe783274 manifestdestiny-binary-i386-1.iso aedb34d89f388540d33a4bf4ff1dd072 *./manifestdestiny-binary-i386-1.iso
This MD5sum file is a great test of neat command-line parameters we’ll set. Speaking of which, spend some time learning from the many great examples rsync provides in the manpage document.
As you review rsync’s pages of options, you’ll find that some are perfect for picking up a very large file, possibly in several tries. The -n option lists what actions rsync will take for a complicated set of options, without performing the transfer. This is great for testing a lengthy transfer will work as planned. The --partial option is very nice. Be default, rsync deletes a partially retrieved file. By adding this option, we preserve the existing data and let rsync add to it. I like the --progress option because it lets me know when the transfer is stalled due to some issue, like an ISP throttling my bandwidth. Wait a minute! A simple -P implements the two P’s of Progress and Partial! So, I should be able to try download the file with a simple -P!
So let’s try a transfer. I type rsync, the -P option, the full URL, the path, the filename I want, and last, the destination directory, or ’.’ because I want it placed right where I am—My Storage Depot.
Me? I just type away and ignore the wrapping. So first, let’s get the md5sums file:
$ rsync -P rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/en/iso/i386/manifestdestiny-binary-i386-md5sums .
manifestdestiny-binary-i386-md5sums
508 100% 496.09kB/s 0:00:00 (1, 100.0% of 1)
sent 168 bytes received 648 bytes 96.00 bytes/sec
total size is 508 speedup is 0.62
That was fast and easy. Look at my download speed of 496.09KB/sec. That’s due to compression my ISP provides and the fact it seems few are using rsync. The evil plot works! Now for CD 1:
$ rsync -P rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/en/iso/i386/manifestdestiny-binary-i386-1.iso . manifestdestiny-binary-i386-1.iso 17692144 2% 3.79MB/s 0:02:43
So, look at the bandwidth we’re achieving now! That’s more than 3MB/sec! We haven’t even turned on compression for rsync itself!
Ok, I’ll be honest. I stopped the first transfer by pressing Ctrl-C, the "stop the application NOW!" keystroke. This illustrates the resumed file transfer moves very fast over previously downloaded content.
So it won’t be long. My bandwidth restriction will kick in, and I’ll go from a speedy connection at 100KB/sec to a lousy 3 or 4KB/sec. I’ll use the control-c sequence to stop the transfer. I will resume the transfer later. This shows what a good Ctrl-C produces:
$ rsync -P rsync://ftp.esat.net/mirrors/whiteboxlinux.org/4/en/iso/i386/manifestdestiny-binary-i386-1.iso . manifestdestiny-binary-i386-1.iso rsync error: received SIGUSR1 or SIGINT (code 20) at /home/lapo/packaging/tmp/rs ync-2.6.6/rsync.c(163) rsync error: received SIGUSR1 or SIGINT (code 20) at /home/lapo/packaging/tmp/rs ync-2.6.6/rsync.c(163)
This article could go on and on. Unfortunately, it cannot. For now, though, download those CDs, in stages if you must. You know the command and the techniques to do so quickly and easily. Well, as quickly as you ISP allows, that is. From there, burn the ISO images to a blank CD.
Hmmm, there was one last thing I wanted to add. Something about typing all those very long command line items. Those very long items that would bring odd errors if you so much as missed a dash. I bet you had a rough time retyping all that stuff. What am I forgetting? If you press the Up arrow key, Cygwin should recall your last command perfectly, keeping you from endless retyping and fumbling. Wow, and to think you went through all that typing when one simple keystroke would help you out...