E-Mail Server Administration 101
Date: May 7, 2004
Each TCP/IP network service has its own quirks and lingo. For the harried administrator who isn't responsible for a particular service (such as E-mail), trying to troubleshoot issues that could be related to that service ("Hey, I can't check my E-mail!") can become a royal pain, especially when it seems to take forever to get across what you're trying to say to the administrator of that particular service ("Run a what??").
In this article, you'll learn today's version of how E-mail works—it has changed somewhat since the early Internet days—along with the jargon, issues, and needs that go along with E-mail administration. By the end, you should feel much more competent when talking to your E-mail administrator, with the lovely side effect that the two of you will finish your interactions sooner, making your days less frustrating and wasteful.
How E-mail Works
E-mail has become such a common tool today that most of us use it without knowing or caring how it actually works. Everyone who uses E-mail knows that people have E-mail addresses. All E-mail users are also aware, at least vaguely, of the E-mail program they use to compose and send the message. However, between hitting Send and receiving the "You've got mail" notice, a lot goes on behind the scenes.
Let's follow a typical E-mail message along its journey. Alice wants to send E-mail to her friend Bob. She starts her favorite E-mail program, such as Eudora or Outlook, known in mail system jargon as a mail user agent (MUA). The MUA's job is more complex than it looks. First, it gives Alice an interface in which type her E-mail and enter the delivery information in the To: and From: boxes. Then, when she clicks Send, it attaches headers to the E-mail with the information needed for that message to get where it needs to go, and for the recipient to be able to respond: a message ID code, the date, and the contents of the To: and From: boxes. Once this packaging is completed, the MUA submits the E-mail to the mail system for delivery.
But what is this "mail system," anyway? Alice's MUA doesn't connect directly to Bob's MUA, after all. Bob's computer might not even be turned on at the time, so there's no guarantee that his E-mail client could receive the mail the moment it was available. Besides, most end-user machines on the Internet don't have a static IP address with a name mapped to them through DNS. How would Alice's MUA know where to send Bob's E-mail in the first place? Firewalls and other network routing games could also cause problems even if Bob has a static IP and name.
Fortunately, the mail system is designed to avoid all of these problems. Rather than trying to send directly from MUA to MUA, the mail system uses a network of mail servers that hand the E-mail off to one another, storing it until the handoff can take place. When Alice clicks that Send button in her MUA, the E-mail client sends the E-mail to the mail submission agent (MSA) that Alice configured the E-mail client to use when she first set it up—or that her helpful administrator set the machine to use.
The MSA's job is to validate the mail being submitted, making sure that it's coming from someone who is allowed to send mail through that particular server. A competent ISP's MSA typically checks the sender's IP address to make sure that the mail is being sent by one of its own customers, and rejects mail submitted from other addresses. Think of the MSA as the "gated on-ramp" to the E-mail superhighway—it's the only part of the system that can verify that the sender was indeed authorized to send that mail. The MSA might even require Alice to authenticate with a username and password to send out that mail.
Once the MSA at Alice's ISP approves her E-mail submission, it hands off the E-mail to the ISP's mail transfer agent (MTA). The MTA's job is to handle the long-haul delivery of the mail to other MTAs. However, how the E-mail gets from Alice to Bob isn't always as straightforward as you might think.
Usually the MTA at Alice's ISP talks directly to the MTA at Bob's ISP, handing off the E-mail in one nice, neat move. At other times, the E-mail can't travel directly from the source MTA to the destination MTA. For example, a large corporation or university may have offices in many buildings, possibly in different cities, so there might be a hierarchy of MTAs at the head office, the branch offices, and then in each department. If Bob works for such a corporation, Alice's ISP's MTA will probably relay the mail to the corporation's "main" MTA, which will then use its knowledge of the organization's structure to decide which branch MTA to relay the E-mail to. That branch MTA can then use its knowledge of the branch's structure to relay the mail to the MTA that services Bob's department.
Along the way, every MTA adds a Received: header to the top of the E-mail as a kind of postmark: "This mail item passed through this MTA at this time, from (previous MTA's hostname or IP address), destined for (recipient's E-mail address)." This setup makes it possible to trace the route that the mail took through the network of MTAs. Here's an example of a single Received: header; you'll probably have more than one of these in the headers of most of your E-mail messages:
Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30])
by michelangelo.renaissoft.com (8.12.10/8.12.10/1.1)
with ESMTP id i2T7sDtD013132
for <dee@renaissoft.com>; Sun, 28 Mar 2004 23:54:14 -0800
When the E-mail reaches the MTA at Bob's ISP or the final MTA at his company (the departmental MTA, in the case of our corporate example), it has nearly reached its ultimate destination. Next, the E-mail normally is handed off to a mail delivery agent (MDA). The MDA is like a secretary, responsible for delivering the E-mail to local mailboxes for storage. If the ISP does any content filtering, such as scanning for spam and/or viruses, the MTA may send the E-mail to the filtering program first—to be consistent, we can call it a mail filtering agent (MFA)—which then sends the filtered result to Bob's MDA.
On reaching the MDA, the E-mail sits in storage in Bob's mailbox at the ISP. When he starts up his own MUA and clicks Check Mail, his MUA connects to the ISP's MDA and downloads any waiting mail, including the E-mail that Alice sent so many words ago.
The whole process, in a nutshell, ends up looking something like this:
MUA->MSA->MTA->...->MTA->(MFA)->MDA->MUA
In practice, some of these functions are combined on the same servers, sometimes even within the same software. Most MTA software can also be configured as an MSA; for instance, handling both the "pickup" function from the sending E-mail clients and the passing-along function from server to server. However, keeping these various roles separate in an explanation helps illustrate the process better; if you're an administrator at a larger ISP, each of these E-mail system functions may in fact be handled separately.
These days, maintaining a separate MTA and MSA is becoming popular even among smaller ISPs, due to the added control it gives administrators over both processes. If an MSA is handling all the mail submissions and verifying the authorization of the senders, the MTA can limit itself to talking only to other registered MTAs, which prevents spammers from connecting directly to your MTA to send their mass mailings. There are a number of potential security and spam-attracting drawbacks in using a combined MTA/MSA solution.
Pity the Modern Mail Administrator
In a sense, the mail administrator's job hasn't changed much since the first days of E-mail. The mail system has worked more or less the same way, after all, for many years. Issues such as E-mail address management, mail routing, mail storage management, and dealing with mail queues and delivery failures are nothing new to the experienced E-mail admin.
On the other hand, the once-trusting E-mail system has had to become much more paranoid, thanks to both viruses and spam. Ten years ago it was common for MTAs to relay mail for anyone and everyone—it was a courtesy provided by well-connected sites to their poorer cousins who couldn't afford a full-time connection to the Internet. As spammers began taking advantage of these open relays to spew their mailings at the expense of others, that relaying practice quickly fell out of favor. Today, an open relay is considered a security problem, and it's enough to get a site blacklisted until the relay is properly secured. Oh, how times change.
The spam problem has reached such epidemic proportions that it's now a necessity to do some sort of E-mail blocking or filtering. That responsibility is a relatively new one to E-mail administrators, who had previously been concerned mostly with connectivity and delivery issues. Today, an E-mail administrator is expected to understand what a naïve Bayesian classifier is, and in general terms how it works. She's expected to know what DNS Block Lists (DNSBLs) are, how they work, and how to judge which ones are appropriate to use for her organization's needs. A modern mail administrator needs to be aware of emerging anti-spam technologies such as Sender Policy Framework (SPF), Microsoft's Caller-ID for E-mail, and Yahoo!'s Domain Keys.
Content filtering is another relatively new task for mail administrators to worry about. At one time it was considered a gross invasion of privacy for a mail administrator to poke his nose into users' E-mail. The same filters that quarantine spam and viruses, though, are now being used to enforce corporate E-mail policies regarding profanity and sensitive content. In fact, in certain industries it's now a legal requirement that all E-mail be archived for a certain amount of time, in case company auditors or federal regulators need to review it as part of an investigation.
On the flip side, today's E-mail administrators often have to deal with other E-mail administrators to work out routing issues. Aside from answering common user questions ("Why did my mail bounce?"), mail administrators have to turn to one another to ask "Why is mail from my network being blocked by yours?" Perhaps your E-mail server's IP address or even your entire domain or address block has been listed in a DNSBL for some reason that you now have to investigate—and resolve, usually by proving your innocence to the DNSBL's maintainer. Or perhaps the receiving site just has a particularly aggressive spam filter that's getting in the way, and you need to figure out what aspect of your users' mail is setting it off.
Understanding Common Mail Errors
As a user trying to deal with the E-mail administrator, one useful thing to do is to actually read through bounce notices when you receive them. This practice not only lets you understand what's going on better, but helps you to send an informed E-mail that speeds up the technical support process—or avoid paging your administrator frantically for something that doesn't actually need her intervention.
Suppose you see text such as this:
********************************************** ** THIS IS A WARNING MESSAGE ONLY ** ** YOU DO NOT NEED TO RESEND YOUR MESSAGE ** ********************************************** The original message was received at Fri, 05 Mar 2004 19:29:27 -0600 from mail.example.net [192.168.1.50] ----- The following addresses had transient non-fatal errors ----- <user@example.net> ----- Transcript of session follows ----- <user@example.net>... Deferred: Connection refused by mail.example.com. Warning: message still undelivered after 14 hours Will keep trying until message is 5 days old
You don't need to do anything; there could be just a temporary glitch in the system and the MTAs will continue trying to deliver the mail. However, you may see something similar to this instead:
The original message was received at Tue, 13 Jan 2004 14:12:29 -0700 from mail.example.net [192.168.1.50] ----- The following addresses had permanent fatal errors ----- <user@example.com> ----- Transcript of session follows ----- ... while talking to mail.example.net.: >>> RCPT To:<user@example.com> <<< 550 <user@example.com>... user unknown 550 <user@example.com>... user unknown
You probably just mistyped the E-mail address. Sometimes you'll see this response if a virus is forging your E-mail address and trying to blast out E-mail to everyone and their dog.
If someone's mailbox at the MDA is full, the following response may be returned to you:
The original message was received at Tue, 1 Feb 2004 15:23:21 -0800 from [192.168.4.15] ----- The following addresses had permanent fatal errors ----- <user@example.net> (reason: Mailbox Full ) ----- Transcript of session follows ----- maildrop: maildir over quota.
And now, the one that tends to confuse people most: If you tried sending E-mail to someone but your server is blocked for whatever reason by a DNSBL, among the text in the bounce E-mail there will be a Reason: field, with terms such as rejected or refused, sometimes with an explanation and often even a URL to the particular DNSBL that blocked your message. These responses are handy to send to your E-mail administrator so he or she knows who to contact to solve the problem.
When you need to share a genuine problem with your E-mail administrator, be sure to forward the entire E-mail, including the headers, so the administrator can see the full error message.
Administrators as Teachers
E-mail administrators are not just high-tech postal janitors, although it sometimes must feel that way to them and you. A big part of their job has always been to support users, which, like it or not, often means teaching users as well. As the E-mail administrator's job becomes more complex, so does the end user's, and this growing craziness shifts more emphasis to the administrator's teaching skills on top of everything else.
Users need to be made aware of E-mail-borne worms and viruses, and taught not to open suspicious attachments—along with how to recognize what's suspicious and what isn't. They need to know what to do with any spam that happens to slip past the filters, and how to report it. They need to know why spam is a serious problem, and why buying spam-advertised products and services is a bad idea. They need to understand "phishing" schemes, so that they can spot them on their own when these scams inevitably arrive in the inbox.
All of these "best practices" for using E-mail are vital nowadays, with all the E-mail-borne hazards we face. In a world where it takes only one careless mistake by one user to spell disaster for a company's network and/or reputation, no business can afford to do without some user education. And yet, E-mail administrators are expected to take on this new duty on top of becoming security and content-processing experts—without additional time, staff, training, or other resources.
Throwing the administrator increasingly into the role of educator has led to some interesting problems. Perhaps the biggest mistake E-mail administrators make when dealing with users is underestimating them. Many IT personnel talk down to their users, treating them like four-year-olds who can't do anything properly. Get a group of administrators together and you can always start a conversation with, "I've got this user who's so stupid..." This may be a byproduct of the fact that these administrators are often more comfortable around machines than around real human beings; they were more than likely hired for their technical expertise more than their interpersonal skills. So how is this person supposed to leap into teaching without at least some sort of guidance?
Among other things, teachers need patience and high expectations for their students to live up to. Administrators who hold their users to low expectations will invariably find that their users don't aim any higher. Those who act like godlike wizards and treat users as simple fools who can't be trusted, create users who will never try to do anything on their own. The moment these users find themselves outside their comfort zone, they'll page the administrator for help, not wanting to "break anything." This setup makes more work for the administrator, naturally, and doesn't encourage users to learn anything or develop new skills. By all means, provide your users with the help they need, but never forget your obligation as a teacher—you're there to help them learn, so they can eventually do things on their own.
NOTE
One option is to take some of the teaching burden from the E-mail administrator's shoulders, if for no other reason than so she can spend her time making sure all the filters and connections are running smoothly. Many organizations have technical writers on staff. These specialists can be the perfect go-betweens, learning from the E-mail administrators and then generating the initial instructional documentation or day classes used to help get people up to speed on E-mail issues. Then the administrator can use the company's or ISP's web site to keep people up to date on the latest issues.
Wrapping Up
The next time you see your E-mail administrator pulling her hair out while muttering about spam, viruses, worms, blocks, and more, hopefully you'll have a bit more sympathy. This is a major time of change for experienced E-mail administrators, and many people running E-mail servers these days were just thrown into the job rather than really having the background in the first place. Having to learn this vast array of new skills can be both stressful and rewarding, but in the meantime there's still a job to do.
At least now you can look forward to the look on the E-mail administrator's face when you tell her that your users are having problems connecting to the MSA, but the MDA is working just fine.