Home > Articles > Security > Network Security

  • Print
  • + Share This
Like this article? We recommend

There are far more envelope and content tests used in the spam world than can be covered in a series of articles. This limitation requires us to stick to the more popular methods, which in the content-testing world include feature recognizers, collaborative spam-reporting networks, and Bayesian learning engines. By using a combination of these three techniques, you can eliminate the vast majority of spam that is currently passing to your users.

Feature Recognizers

There are particular features that are the hallmarks of spam—although some of them are also the hallmarks of items like E-mail newsletters, which is why so often these requested E-mails are caught by spam filters (and why you should use a quarantine to let the users sort their own spam from their ham).

A "feature" can be any characteristic of an E-mail message that might tip you off that the mail is (or is not) spam. It can be something blatant, such as an extra mail header that spam software inserts for tracking purposes, or it can be something more subtle, such as the symptoms of mail header forgery.

Feature recognizers use pattern-based rules to look for a large number of features in the headers and in the body of an E-mail message. Many of these rules are contributed by mail administrators around the world, and can be downloaded in regularly-updated sets from public websites, such as the SpamAssassin Custom Rule Emporium. When a new form of spam starts making the rounds, it usually takes less than a day for someone to submit a new rule or two to accurately identify these items. Think of these as "virus signatures for spam." By keeping your rule sets up-to-date, your feature recognizer can better protect you.

One popular spam feature is HTML-formatted mail, in part due to what we discussed in Part I: spammers can embed a unique code into an image access link, allowing them to know who looked at the message, and therefore determine which addresses go to real people. Another reason for HTML mail is URL obfuscation. Scam spammers will often include URLs such as:

http://www.paypal.com:80@7527202817/secure-login.php

Take this URL apart, and you will actually find that it points to host 192.168.0.1 and not to http://www.paypal.com. This trick works because most people don't realize that the full possible format of a URL is really this:

scheme://username:password@host:port/filepath

In our example, then, the scam artist is taking advantage of the optional username and password fields to trick you into thinking you're actually connecting to http://www.paypal.com, when in fact "http://www.paypal.com is just the username, and the "80" is not a port number but instead is the optional password. The real host's IP address is encoded as a "really large number" (the decimal version of the four bytes used for a traditional IP address): 7527202817. When you convert this decimal number to an IP address, you get 192.168.0.1, but that's not obvious to human eyes, and so this value looks at a glance like some sort of "session ID" code.

This technique is a great way to trick people into thinking they are interacting with their bank, favorite online auction service, ISP, or other venue that has vital information someone might want to steal. Since legitimate senders have no reason to conceal their URLs, the presence of an obfuscated URL in an E-mail message is a strong sign that the mail is spam, and it's an easy feature for properly-designed feature recognizers to pick out.

Spammers also like HTML for its visual impact. It lets them catch your attention with bright colors, pretty pictures, and large fonts. In fact, this is one area where spammers often go overboard, making their spam even easier to pick out from legitimate mail. Most people don't use HTML to compose the mail they send, and when they do, they tend to stick to reasonable font sizes and default colors like black. When you spot 32-point red text in an HTML E-mail, then, the mail is almost always spam.

This same ability to use colors for text and background lets spammers "hide" text as well, by printing white letters on a white background, or using an extremely tiny font. Why would a spammer want to hide his message from you? He's trying to fool your content filters by stuffing the mail with a bunch of words that usually signal that the mail is legitimate. That way, if your filters see ten words that make the mail look like spam ("buy," "guaranteed," "limited time," "exclusive offer"), but 100 words that make it look like legitimate mail ("dictionary," "rhododendron," "oncology"), the mail is more likely to get through. However, this list of junk terms could distract you from the sales pitch if visible to human eyes. Using HTML font tricks, the spammer can hide them from you, without hiding them from your content filter. Since there are few legitimate reasons to print white text on a white background, this feature is usually associated with spam.

Another popular spam tactic is to display an attached image with HTML, but without any text. Technically, mail programs that send HTML-based mail are supposed to include a plain-text version of the mail as well, so the contents can be read by mail programs that aren't HTML-aware. It's easy to forget that there's no requirement that a mail client be able to read HTML, and many popular clients in the Unix world, such as elm and pine, don't support HTML without additional plug-ins. If you receive a message that contains HTML but no plain-text, you can be pretty sure that either it was sent by a poorly-written mail client (of which there are plenty, alas) or it was designed to broadcast spam.

Not all spam features involve HTML, however. Feature recognizers can also watch for the techniques popularly used to evade content filters. For example, back to the ever-popular "Viagra" pitches, a spammer might try to fool the filters by using "V|@gra," "V^I#A*G%R*A," or even "Via<!—nonsense_gra" in order to get their point across in a way that a computer won't easily catch on the server-end or in an E-mail client's mail filters.

Then there's "When Feature Recognizers Attack." No computerized tool is perfect. While often associated with spam, the phrase "multi-level marketing," for example, could appear in a legitimate context as well. Say, a newsletter about spam scams. Most rules are designed to identify spam features, rather than ham features, but the basic idea behind feature recognizers is to assign a score to each rule (positive values for spam features, and negative values for ham features) and then add up the cumulative scores at the end of the process. Features that appear almost exclusively in spam tend to be assigned higher scores than features that are found often in ham as well.

If you've got a large number of spam and ham items sorted into separate piles, a particularly clever feature recognizer can even compute these score values for you, based on how often a rule gets triggered by the mail items in each pile. For example, if you've got 1,000 items in the spam pile and 1,000 items in the ham pile, you could find out how many of the mail items in each pile contain a pattern like "MLM" or variations on the phrase "multi-level marketing." You might find that this rule triggers in 250 of the items in the spam pile (25%), but only in 10 of the items in the ham pile (1%). You could then conclude that if this rule gets triggered, there's a 250/2000 (12.5%) chance that the mail is spam, and only a 10/2000 (0.5%) chance that the mail is ham. The feature recognizer can then use these percentages to come up with a balanced score for this rule, so that it doesn't receive too much (or too little) weight.

Collaborative Spam-Reporting Networks

One aspect of spam that many forget is that it is a one-to-many broadcast. The typical spammer sends to millions of recipients at a time, so by the time you receive a copy in your mailbox, you can be sure that many thousands of others have already received theirs. Wouldn't it be handy if some of these people could come forth and say, "I've already received that mail, and it's definitely spam"? If enough people did that by the time you received your copy, you could be pretty confident that what you just received was spam. This is the beauty of collaborative spam-reporting networks.

There are a number of these collaborative networks, such as the Distributed Checksum Clearinghouse, Vipul's Razor and Pyzor. These sites maintain databases of spam reported by people all over the Internet, so that a content-filter can quickly determine whether a given E-mail has already been reported by others as spam. The more people who have reported a given piece of spam, the more the content-filter should be inclined to make the same diagnosis—again, by using a scoring system.

Spammers actually do try to make each copy of a piece of spam "unique" for each user, but these collaborative networks are typically smart enough to recognize a particular spam in spite of these "personalizations." The fact that one copy reads "Dear paul387" and another reads "Dear chris24" doesn't make it a different mail item, as far as these databases are concerned. If the rest of the mail fits the same template, a few extra words thrown in won't be enough to make the mail items seem unique.

One handy function of these networks is that they can give you an idea of how many people have reported an item as spam in real time. This factor lets you make threshold-based decisions; you might decide that if 100,000 people think an item is spam, that's good enough for you. Since this is based on human input, rather than rule-based automation, the results can be more reliable than a computer's decisions. Also, the real-time nature lets these networks respond quickly to new forms of spam that computerized filters using yesterday's rule-sets might miss.

A downside of consulting external databases, however, is that when you receive a piece of mail, your server then has to wait for a response from a server somewhere else on the Internet. On a high traffic site or a day when network traffic is overloaded or even down, the resource and time-lag cost of processing every piece of E-mail becomes prohibitive. On better days, this feature adds one or two seconds to mail processing. But, on high traffic sites, this performance hit may be more than you can tolerate.

Bayesian Learning Engines

If you hang out anywhere online where people talk about their favorite spam solutions, you have no doubt seen people bandying about the term "Bayesian filters." The Bayesian learning engine's popularity has skyrocketed over the past year, for good reason: it's a truly effective content-scanning tool.

Learning isn't just a buzzword here. Bayesian engines gobble up the spam and ham that people mark off in their quarantine folders, analyzing the frequency with which certain tokens (words, symbols, or phrases) appear in each. Over time, they (the engines, that is, not necessarily your users) actually get "smarter;" the more spam and ham the engine chews on, the more accurately it understands the big picture.

In a sense, Bayesian (often dubbed "Bayes") learning engines are automated feature recognizers. The feature recognizers we discussed earlier are based on patterns that human beings have spotted and hard-coded into rules. Bayesian learning engines try to do the same sort of thing on their own, without any initial rules to go on. You sort your mail into a spam pile and a ham pile, and let the Bayes engine scan them both to identify patterns without any further human guidance. Every message you receive thereafter can be used to add to its knowledge, just by telling the Bayes engine whether that new item is spam or ham.

For example, the fact that the word "Viagra" appears as a token in so many pieces of spam and appears so rarely in legitimate E-mail makes the Bayesian learning engine suspicious when that token shows up in new mail. Unlike ordinary feature recognizers, however, this kind of pattern recognition is highly individual. If you're a urologist, for example, and "Viagra" appears often in your legitimate E-mail, the Bayesian learning engine will regard that token with much less suspicion (possibly even as an indicator of legitimate mail!).

Most importantly, though, you never have to tell the Bayes engine to look for a specific "Viagra" pattern, as you would with ordinary feature recognizers. The Bayesian learning engine should pick out that pattern all on its own, if it's found in a lot of your E-mail. Whether that token is taken to be an indicator of spam or ham depends entirely on whether it shows up more often in your spam pile or your ham pile.

The Bayes engine uses a "confidence level" to indicate the likelihood that the mail as a whole is spam. A low confidence level (say 10%-20%) is a strong sign that the mail is ham, whereas a high condidence level (90%-100%) suggests the mail is almost certainly spam. If the mix of spam tokens and ham tokens is almost even, the confidence level ends up around 50%, which is the Bayes engine's way of saying that it couldn't make up its mind.

Due to the effectiveness of the Bayesian engines, spammers aren't fond of them. In fact, these days many spammers go to the trouble of trying to "poison" Bayes databases that chew on their mail. As we mentioned in the feature recognizers section, the spammer stuffs a long list of dictionary words together somewhere in the E-mail message, hoping that some of those words will appear more often in your ham pile, so that the mail will look more legitimate. While a feature recognizer will just be fooled or not fooled, to a Bayes this technique is meant as a "poison pill." If you try to train your Bayes engine to recognize this E-mail as spam, you force it to add all those dictionary words to its database of spam tokens. The end result is that legitimate mail that includes those dictionary words in the future can get flagged as spam.

All is not lost, however. The size of your Bayes database determines how vulnerable it is to this poisoning tactic. A gallon of poison in a bathtub of water is quite potent, but that same gallon in the Pacific ocean is diluted to the point of harmlessness. As your database grows, this tactic has less and less impact. In fact, smarter anti-spam solutions can even learn to identify these blocks of dictionary words (which are usually all lower-cased and without any punctuation) and avoid adding them to the database at all—the poison is just another pattern, or feature, after all.

  • + Share This
  • 🔖 Save To Your Account

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020