Internet Email Protocols: A Developer's Guide

Description

• Copyright 2000
• Edition: 1st

• Book
• ISBN-10: 0-201-43288-9
• ISBN-13: 978-0-201-43288-6

Internet Email Protocols gathers, in one comprehensive and convenient resource, the key technologies that form the underlying mechanics of Internet email. It provides developers and networking professionals with in-depth explanations of essential concepts and a framework for understanding how and why these technologies fit together to provide cohesive email services. This book shows how to troubleshoot and solve email problems, how to evaluate different implementations of the standards and protocols, and how to be better equipped when faced with the challenge of writing programs that process email.

For each chapter, the author presents an overview of the material, detailed information, and any important extensions and advanced uses of the protocols being discussed. You will find in-depth descriptions of:

• Message formats, covering the structure of message headers, the various fields, and the intricacies of email addresses, as well as a detailed treatment of MIME
• Mail transport technology, focusing on the SMTP protocol
• The most current versions of the POP and IMAP protocols
• The mechanics of filtering email, with an emphasis on applying this knowledge to blocking unsolicited bulk email
• The mechanics of mailing list processing, with an emphasis on applying information from earlier chapters
• Security details, covering authentication, digital signatures, encryption, and anonymity

In addition, the appendices provide example folder formats, a discussion on UNIX folder locking, and an overview of the languages commonly used for email development. The accompanying CD-ROM contains all the RFCs referenced in the book, as well as the source code for several popular email packages, including Sendmail and Procmail. Bringing together information that is scattered among many RFCs, software packages, and lore, this one detailed book is an important resource for anyone developing email applications.

0201432889B04062001



Sample Content

Table of Contents

(Each chapter concludes with a Summary.)

List of Figures.


List of Tables.


Preface.


Acknowledgments.


1. Overview.

The View from 10,000 Feet.

Mail Transfer Agents.

Mail Delivery Agents.

Mail User Agents.

The Standards.

The Standards Process.

Augmented Backus-Naur Form.

A Brief History of Internet Email.

2. Mail Messages.

The Message.

The Body.

The Header.

Inside Fields.

Whitespace.

Atoms.

Comments.

Quoting.

Field Folding.

Case Sensitivity.

Backspaces.

Standard Fields.

Originator Fields.

Receiver Fields.

Reference Fields.

Trace Fields.

Resent Fields.

Other Fields.

Extension Fields.

Nonstandard Fields.

Some Examples.

A Simple Message.

Redistributing.

Forwarding.

Replying.

Email Addresses.

Mailboxes.

Group Lists.

Received Fields.

Field Elements.

Comments.

Machine Parsing.

Diagnosing Problems.

Date-Time Fields.

3. Mail Transport.

Interface Model.

Commands and Replies.

SMTP Sessions.

Protocol States.

Standard Commands.

Quit.

Help.

Noop.

Verify.

Expand.

Hello.

Mail From.

Recipient.

Data.

Reset.

Send.

Send or Mail.

Send and Mail.

Turnaround.

Local Extension Commands.

Mail Transactions.

Mailing.

Relaying.

Forwarding.

Operational Considerations.

Protocol Nonconformance.

Reliability and Responsibility.

Queuing.

Sending.

Receiving.

Timeouts.

Bounces.

Loop Detection.

Security.

Message Submission.

SMTP Extensions.

Message Size Declaration.

Command Pipelining.

8-bit MIME Transport.

Remote Message Queue Starting.

Enhanced Status Codes.

Delivery Status Notifications.

LMTP.

4. MIME.

Predecessors to MIME.

Overview of MIME.

MIME Headers.

MIME-Version.

Content-Type.

Content-Transfer-Encoding.

Content-ID.

Content-Description.

Content-Disposition.

Content-MD5.

Content-Language.

MIME Extension Fields.

MIME Media Types.

Text.

Image, Audio, and Video.

Application.

Message.

Multipart.

Other Media Types.

Message Header Extensions.

Charsets.

Encoding.

Using Encoding.

Mailcap Files.

General Syntax.

Fields.

Flags.

5. POP.

Interface Model.

Commands and Replies.

Protocol States.

Standard Commands.

User.

Password.

APOP.

Quit.

No Operation.

Status.

List.

Retrieve.

Top.

Delete.

Reset.

Unique-ID Listing.

Example POP Dialogs.

Basic Dialog.

Maildrop Summary.

Keeping Recent Messages on the Server.

POP Extensions.

Extended Service Offerings.

POP3 Extension Mechanism.

6. IMAP.

Interface Model.

Protocol States.

Commands and Responses.

Status Responses.

Server Data.

Command Continuation Request Responses.

More on Multiple Pending Commands.

Data Objects.

NIL.

Atoms.

Quoted Strings.

Literals.

Parenthesized Lists.

Mailbox Names.

Internal Date.

Size.

Message Numbers.

Unique Identifiers.

Message Flags.

Envelope Structure.

Body Structure.

Standard Commands.

Capability.

Noop.

Logout.

Login.

Authenticate.

List.

Create.

Delete.

Rename.

Status.

Select.

Examine.

Close.

Subscribe.

Unsubscribe.

Lsub.

Copy.

Append.

Check.

Search.

Fetch.

Store.

Expunge.

UID.

Experimental Commands.

Example IMAP Dialogs.

Replacing POP.

Folder Summary.

IMAP Extensions.

Nonsynchronizing Literals.

Idle.

Namespace.

Access Control Lists.

7. Filtering.

A Model for Filtering.

Available Inputs.

Condition Matching.

Filter Actions.

Filtering in Mail Agents.

MTA Filtering.

MDA Filtering.

MUA Filtering.

Basic Filtering.

Simple Personal Filtering.

Vacation.

Email Robots.

Unsolicited Email.

Relaying.

Known Offenders.

Originator and Recipient Fields.

Field Contents.

Telltale Fields.

Header Errors.

Message Bodies.

Opt-Out Lists.

Whitelists.

Prevention.

Advanced Filtering.

Scoring.

Collaborative Filtering.

Dynamic Email Addresses.

8. Mailing List Processing.

Precursors to Mailing Lists.

Personal Address Lists.

Group Lists.

MTA Alias Lists.

Basic List Processing.

Exploders.

Advanced List Processing.

Bounce Handling.

Administrative Commands.

Digests.

Archives.

Distributed List Processors.

9. Security.

Security Issues.

Eavesdropping.

Impersonation.

Denial of Service Attacks.

System Integrity.

Core Security Technologies.

Pseudo-Random Number Generation.

Cryptographic Hash Functions.

Symmetric Encryption.

Asymmetric Encryption.

ElGamal.

Security Services.

CRAM.

OTP.

Kerberos.

GSS-API.

OpenPGP.

Security Frameworks.

TLS.

SASL.

Protocol Security Extensions.

IMAP Extensions.

POP Extensions.

SMTP Extensions.

MIME Security.

Security Multiparts.

PGP.

S/MIME.

Anonymous Remailers.

Appendices.


A - Example Folder Formats.


B - UNIX Folder Locking.


C - Programming Languages.


D - IMSP.


E - ACAP.


F - LDAP.


Acronyms.


Index. 0201432889T04062001

Preface

This book is about the various protocols and standards that make up the technical side of Internet email. It describes the formatting of mail messages, including MIME--the SMTP, POP, and IMAP protocols, filtering and mailing list technologies, and security. The focus is on understanding how and why Internet email works and how it has evolved from its early forms into the "killer app" many people use on a daily basis.

Two types of applications dominate the Internet: the Web and email. Go to the computer section of any bookstore and count the number of titles devoted to the Web, including those for end-users, as well as those for developers and administrators. Now, count the titles dedicated to Internet email. You'll find the number significantly smaller, particularly for technical ones. This book attempts to remedy this, at least in some small way.

Learning how email works is not necessarily straightforward. The information is spread across a large number of documents written over a period of many years, and most of the protocols have undergone several iterations, resulting in corrections and new versions. Thus, one of the difficulties is understanding the relationship of an email document with its predecessors and successors. The information is presented here in a way that cuts across established standards by providing a current view of email technology. After reading this book, it will be much easier to put a particular RFC into perspective.

Another difficulty is that most standards aren't intended to help people learn particular protocols; they are intended to provide, at least theoretically, an exact definition of a protocol. This book presents the information in an order better suited to learning the material.

To add to these issues, there are differences and errors in the interpretation of standards and proposed standards, as well as ambiguities and errors in the documents themselves. This has resulted in various email packages implementing the standards in different ways. Key interpretations of some if the problem areas are provided as appropriate.

The bottom line is that it can be difficult to get a detailed picture of how email works. The goal here is to solve that problem.

Audience

This book is for developers, administrators, and possibly power-users, or more generally anyone with the need to understand the technical side of Internet email. Developers will find the information organized to make it easier to digest the actual standards documents. Even those developers not creating dedicated email applications should find the information useful for adding email functionality to their programs.

System administrators can use the information to help them troubleshoot and manage email systems. A system administrator's life is easier when he or she has a detailed understanding of how things are put together and how they work. Power-users can use the information to help them use email tools to the utmost. This is particularly true for the chapters that deal with mail messages, MIME, filtering, and security.

Organization

There's a natural progression to the book's structure, in which each chapter builds on information in a previous chapter or chapters. Readers new to the technical aspects of email should start at the beginning and proceed chapter by chapter. Advanced readers, of course, can read the chapters in any order.

Be aware that Chapters 2, 3, 4, and 9 contain information that is fundamental to an understanding of email. Other chapters can be safely skipped according to reader interest.

Chapter 1, Overview, provides a high-level overview and introduction to Internet email.

Chapter 2, Mail Messages, provides a detailed walk-through of the structure of email messages.

Chapter 3, Mail Transport, describes SMTP and various extensions to the protocol, and how they are used to transport email.

Chapter 4, MIME, describes MIME and how it's used to encapsulate structured or nontext data in email messages.

Chapter 5, POP, describes the POP protocol, its use, and its various extensions.

Chapter 6, IMAP, describes the IMAP protocol, its use, and several protocol extensions.

Chapter 7, Filtering, describes how email filters work, as well as various tasks that can be performed with them. The emphasis is on applying knowledge of how email works to make filtering more effective, particularly with unsolicited bulk email.

Chapter 8, Mailing List Processing, provides a detailed walk-through of mailing list processors and how email headers are manipulated to perform mailing list functions.

Chapter 9, Security, describes various aspects of email security, including authentication, digital signatures, encryption, and anonymity.

Appendix A, Example Folder Formats, describes the format of several folder formats, with an emphasis on understanding the performance implications of their various design features.

Appendix B, UNIX Folder Locking, describes various issues related to folder locking in UNIX, which can be a particularly troublesome problem.

Appendix C, Programming Languages, surveys several common programming languages, and some of their interesting characteristics related to email.

Appendix D, IMSP, provides a brief overview of the IMSP protocol. Although it was never widely implemented and is being replaced by ACAP, its historical importance makes it worth mentioning.

Appendix E, ACAP, provides an overview of the ACAP protocol. Although not exclusively an email protocol, ACAP provides several features of interest to email developers.

Appendix F, LDAP, provides a brief overview of the LDAP protocol, which provides network-based querying and updating of directory services information.

What This Book Does Not Cover

Email is a broad topic, particularly when adjacent technologies are factored in. No book can hope to cover it all. More specifically, this book does not describe how to use, configure, or manage specific email packages. There are plenty of books out there to fulfill that need. The information here is meant to help readers learn the underlying technology implemented by email software.

Some areas of email technology are not covered--for example, the relationship between Internet email and X.400. Several advanced uses of MIME, such as transmitting voice and FAX data, are also not discussed. These areas should be easier to learn, however, with a solid understanding of the underlying technology provided here.

In addition, this book does not provide detailed information about related protocols, such as TCP/IP and DNS. However, when knowledge about these protocols is important in understanding a particular topic, information is provided as necessary. Network protocols are designed in layers. Although understanding how they work is important, most Internet email can be understood without learning every other protocol in the picture.

Finally, several algorithms related to security are not described in detail. Security is a complicated topic, requiring much more coverage than can be provided in this book.

Conventions

Examples directly related to the topic of a chapter are inset, enclosed in a box frame, and rendered with a monospace font, as shown here.

like this

If an example involves a dialog between either a client and a server or a user and a computer, the client or user inputs are set in bold. Items in a dialog that need particular attention are shown in bold italic. Summarization of information is represented in italics.

the client said this

the server responded with this

followed by a large amount of other information

Other textual examples and syntax descriptions are inset and rendered with a monospace font, with no framing box,

like this.

Most email protocols use a carriage-return (CR, US-ASCII 13, '\r') character followed by a line-feed (LF, US-ASCII 10, '\n'), character to terminate lines of text. For readability, the examples in this book do not show the line terminators, except in a few instances. Each protocol has specific line termination requirements, which should be considered as implied in the examples and syntax descriptions.

While on the topic of syntax descriptions, it should be noted that some do not exactly match the ones provided in the RFCs, particularly some of the older ones, such as RFC822. In the interest of clarity, the author has chosen to provide syntax descriptions that might be easier to understand or that might resolve an ambiguity. In particular, this is the case with the use of whitespace in RFC822. The differences are generally cosmetic, but the "improved" syntax descriptions shown should be treated not as reference material, but merely as an aid for learning the material.

When referenced in the body of the text, message header fields, protocol commands and parameters, and email addresses are presented in a monospace font. In a few places, '_' is used to identify a space character. It is generally limited to places where the lack of such a character would be confusing.

Many of the examples use the domain names example.com, example.edu, and example.net. Don't try sending email to any of these locations--it won't work. The example domains are set aside for use in situations in which domain names are needed but writers want to avoid the possibility of using ones that someone might use in the future.

One final note about conventions. This book uses email when referring to electronic mail. Some people might feel e-mail should be used instead; however, English has many word combinations that get shortened over time. Remember that electronic mail was initially shortened to e-mail. As this type of word becomes widely used, eventually the hyphen gets dropped. In fact, there are many instances where simply mail is used.¹ It is the author's opinion that this term is used widely enough for the hyphen to go.² In particular, this form has become the standard, as it were, in the email industry. Go ahead and drop it, it's just a hyphen; you won't invoke the wrath of the hyphen police.

¹Maybe by the time we're done, everyone will shorten it to just m.

²For a second opinion, see http://www.cs-staff.stanford.edu/knuth/email.html

A Few Notes about the Standards

The primary document describing the rules and guidelines for implementing Internet email protocols is the RFC, or Request for Comment. In fact, the RFC defines most fundamental protocols used on the Internet. There are thousands of RFCs available, ranging from information to required standards. Anyone who spends time with Internet protocols will eventually find their way to them because they are an important aspect of the technical underpinnings of the Internet. The creation of RFCs is described in further detail in Section 1.2.

Reference is made to a large number of RFCs, most of them directly related to Internet email; some of them are only indirectly related. This book does not replace these RFCs, it amplifies them. Although some readers might only have a passing need to understand the exact specifications of a protocol, there are bound to be situations in which referring to an RFC will resolve a question. Readers are strongly encouraged to read the RFCs, and not rely solely on the information in a book.

This book includes syntax descriptions for many elements of email. In places, these descriptions deviate slightly from the exact words used in the RFC syntax specifications in the interest of clarity. The author apologizes for any inadvertent errors made in the process. On a related note, if any unqualified statements made in this book conflict with the standards, the standards probably win.

One last general point about the RFCs is in order. Standards and protocols evolve. As a result, updates are published as new RFCs. Readers delving into the world of Internet standards need to be aware of the historical context of a particular RFC. For example, which RFCs does it update or replace, and which RFCs update or replace the ones in question. This book references many older RFCs, particularly in the context of newer ones. There is no doubt that current RFCs will be updated by newer ones, some while this book is still new. Readers should always check to see if an RFC has been updated.

Getting the Standards

There are two primary repositories for Internet protocol specifications--one is for RFCs, the other is for Internet drafts. The RFC repository is available at http://www.ietf.org/rfc.html This site contains an overview of the RFC repository and the standards process, answers to frequently asked questions about RFCs, and the actual repository for RFC documents.

All the RFCs referenced in this book are provided on the accompanying CD-ROM. This is intended to help people with limited bandwidth, particularly because there are so many RFCs related to Internet email.

Internet drafts (IDs) are documents intended to become RFCs eventually. They often undergo several iterations and review by many people before that happens, if indeed it ever does. The repository for Internet drafts is located at http://www.ietf.org/ID.html Because of their dynamic nature, IDs are not provided on the CD-ROM. This should not be a problem--most readers don't need to worry about them.

This Book's CD-ROM

A CD-ROM accompanies this book. It contains all the RFCs referenced here, as well as the source code for several email packages. A thin layer of HTML is provided to aide in navigating the information. The purpose of the CD-ROM is to allow readers to access the information without having to download it from the Internet.

Each RFC is presented in its published form, direct from the master source, with no alterations. With it is a companion HTML page, which presents some of the information contained in the master table of contents at the RFC repository. This information is available by accessing the top-level HTML pages on the CD-ROM.

The choice of software packages to include on the CD-ROM was a difficult one. Some were chosen because they are reference implementations. Some were chosen because they are classics that most people working with Internet email will deal with at one time or another. The absence of a package does not reflect the author's opinion of it. There are just too many software packages available; a line had to be drawn somewhere.

The software packages on the CD-ROM are the more recent versions available at the time of publication. Most software packages on the Internet are distributed as compressed archive files. The ones here have been uncompressed and expanded from their archive format, without modification. Note that they are provided as examples for viewing by readers. Neither the author nor the publisher provides support for them.

One final word about the CD-ROM. Over time, some of the information will become out of date. Readers are strongly encouraged to check the original sources to determine if this is the case.

Acknowledgments

Several people deserve special mention. The first ones are the many authors of email RFCs, those still with us, and those departed. This book is a tribute to their work.

The staff of Addison-Wesley is to be commended for helping in the creation of this book. Karen Gettman, the editor, not only believed the book was worth publishing, but provided insight and advice throughout the process. She also lined up some of the best reviewers I could have imagined. Mary Hart, the project editor, kept the process of routing draft submissions and reviewer comments clean and predictable. Their understanding and professionalism helped make this project manageable. Marilyn Rash, the production coordinator, kept the book's production phase running smoothly, even in the face of the inevitable glitches that arose. Dianne Cannon Wood, the copy editor, did an excellent job of smoothing the rough edges off the draft manuscript. In addition, the people at Interactive Composition in Portland, Oregon, are to be commended for coming through in a pinch.

I would like to thank the technical reviewers--Vic Abell, Mark Crispin of the University of Washington, Mark Ellis of Reuters, Ned Freed of Innosoft, Vic Hargrave of Endeavor Information Systems, Howard Lee Harkness, Christopher Lindsey of NCSA, Danny J. Mitzel, and Pete Wilson. Their comments and suggestions on the draft material were invaluable.

I would also like to thank several people I work with at Motorola Computer Group. Ken Ridgely, my manager, encouraged me to pursue this project and gave me flexibility with my schedule at all the right times. Being a technical manager poses special challenges; Ken is one of the best. Larry Cruz and Eric Sunderland provided feedback on early drafts of the material; their patience with these was admirable. Susan Lee Wilson provided a good sounding board for many of the problems I encountered. Her uncanny ability to ask just the right question, at just the right time, helped me solve many perplexing problems. I'd also like to thank the many people who graciously let me bend their ears about my experiences writing a book.

A few other people deserve a word of thanks. Both Randal L. Schwartz, General Manager of Stonehenge Consulting Services Inc., and Joseph Hall, author of Effective Perl Programming, encouraged me to believe my idea for a book wasn't farfetched and provided useful advice to a first-time author. In particular, Joseph recommended that I submit a proposal to Addison-Wesley. Without their encouragement and help, the book would never have been written. I'd also like to thank Meng Weng Wong, of pobox.com, for technical feedback and for chiding me on my use of my native tongue, and Sean M. Burke for opening my eyes, as it were, to the larger importance of the Content-Language field.

Last, but not least, I'd like to thank my wife, Kate, for understanding my desire to write this book, providing moral support during the project, and gallantly sacrificing time normally spent together. Marrying her was the best decision I've ever made.

I cannot imagine writing this book without the help of all these people. Their involvement has made this a better book.

0201432889P04062001



Updates

Submit Errata

