Table of Contents

1. Introduction
2. White List Mechanisms
3. Secret Address Mechanisms
4. Word Filter Mechanisms
5. Blocking Mechanisms
6. Why ISP Spam Scrubbers Don't Work
7. Summary of Spam Prevention (as of 2004.06.28)

1. Introduction

This document will describe to you the major spam prevention techniques available today. The reason I'm writing the document is to educate people, so they don't waste time on things like word filters, when they should be paying more attention to better techniques such as white lists, and aliases. Alas, any document of this type must come to terms with all the wrong ideas first, just so that the reader has assurance that these have truly been examined before any recommendations are made.

Therefore there has to be a huge chapter dealing with "word filters".

This document doesn't deal with the various new laws or industry proposals for solving the problem. It mainly concentrates on products and methods that you can implement right now to prevent spam.

2. White List Mechanisms

(Authorized senders list) How it works: A "White List" is a list of people from whom you will accept mail. For example, Ross's authorized senders list. A black list is a list of people from whom you will NOT accept mail, even if they were on your whitelist. A "Challenge/response" mechanism is a mechanism to allow people to add themselves to your whitelist. Or you can add them yourself.

When someone new sends you an email, their first message is "suspended" till they answer the challenge email. The "challenge" email just requires the sender to verify themselves by copying a word into a form. This then adds them to your whitelist, and their original email then comes through. Below is a typical "challenge" message.

  Type the following word:  School

Any human can copy the word "school", but automated programs cannot. Non-authenticated messages go into a holding tank, which you can inspect at any time. After a few weeks, they are deleted automatically.

Spam Arrest: To learn how whitelist is supposed to work, have a look at http://spamArrest.com. They don't necessarily have the best service, but at least they have the right idea, and their service definitely works. One of my customers has used it for over a year and is still using it.

The way SpamArrest works is that their server picks up your mail from our server, and then you pick up the filtered mail from their server. Your whitelist is on their server. It costs about 3.00/month. They have a special mechanism for newsletters, which is discussed below.

The limitations of their service is that it prevents our webmail from working, because they will have already picked up your mail. Another limitation is I don't think they have any mechanism to automatically add any new outgoing messages to your whitelist. These limitation could be overcome.

NewsLetters versus Newsgroups: What I mean by a "newsletter" is something with a single return address. These are no problem, because you can just enter the return address into your whitelist as soon as you sign up.

What I mean by "news groups" are things where you sign up and then there is a mechanism that broadcasts email messages to everyone in the group. One example are the Yahoo news groups. The difference between a news group and a newsletter is that with a news group, the return address can be any member of the group. There is no one "from address" that you can add to your whitelist. Therefore most challenge/response systems have a special mechanism for whitelisting the news group as a whole.

Desired Features:

a. Automatically add outbound addresses: The server based white list must be updated automatically with the "To" address of all outbound mail. This is the same idea as adding people automatically to your address book, except that since the whitelist is on the server, the server has to do it.

b. Webmail: It is desirable that any whitelist mechanism should offer their own webmail.

Bluesoft Whitelist Implementation We did a 2 month trial of a piece of software that implemented "whitelists" and "blacklists" with our existing mail server. From an operational viewpoint, it was 100% effective. Zero spam and zero mistakes. Only problem was the program itself would crash our mail server. So all we need to find is another program/mail server combination that works and all your problems will be over.

3. Secret Address Mechanisms

The most reliable method to avoid spam is to not publish your email address. By "publish" I mean making your email address available to people you don't know. The most common method of Never use it on your website, even in a "contact us" link. Instead, use blind email, or disguise your email so machines cannot read it.

Email AliasesThe basic idea is to only give your email address to trusted individuals. For any website, just enter an "alias" email address. For example, if your name is Fred Smith, and you sign up for the buy and sell, give them an address like "FredSmith_BuySell@MyDomain.ca". Just add that address to your alias list. If any spam starts to come in addressed to that address, you know where it came from, and you can cut it off by removing that domain.

The above takes care of websites. In addition, you might also want a single address that you never give to anyone other than trusted individuals. For example, FredSmith2@abc.com. Note the sequence nnumber. In the event that one of your trusted people puts your address into a spam list, you can always change the sequence number in your email alias, and then mail out to everyone in your address book the new address. Remove Generic Addresses As is described under "domain aliasing", in my mail server document, our email server defaults the forwarded domain with aliases such as "admin", "Postmaster" and "webmaster". These aliases should be erased, since they are obvious targets for spam mail.

Blind Email Blind email is a scheme for implementing "contact us" links on your website without disclosing the end address. Instead, program an email form into your website, and have a program which executes the form and sends the email. Thus the sender does not know the email address unless you reply. It is called "blind" because the sender doesn't see your email address.

I use this scheme in all my websites. If you want to see how this works, go to the front page of the Bluesoft.ca, and try and send me an email message. If we are running your website, I can program this scheme into your website in a couple of hours. (See "Blind Email" under Web Hosting Services).Below is a link to my blind email, so you can see how it works: Contact Robin Tivy via blind email

The two problems I've run into with blind email are:

1. If something goes wrong, the sender does not receive the error message. (So you have to be sure the target address is always working).

2. Anyone making cosmetic changes to the "contact us" form, has to realize they are modifying a form, and cannot just stupidly use some handy "what you see is what you get" editor to make cosmetic changes.

4. Word Filter Mechanisms

By "word filter" I mean any piece of software that claims to be able to "analyse" incoming email messages and determine which are spam. The simplest are the old filters that were offered in various email clients where you typed in certain words like "sex", and "special offer". They worked for a while, but spammers eventually beat them. The latest word filter is a "Baysian Spam Filter", which you supposedly "train" to be able to recognize good from bad.

ISP Spam Scrubbers These are services offered by various internet service providers such as Telus. In some cases, ISP spam scrubbers may also make use of anti spam organizations.

Baysian Spam Filters A "Baysian" spam filter is one that compares the message body with a database of example "spam" messages and also with a database of known "good" messages. Such systems have to be initially "trained" by telling them which messages are spam, and which are not. They then mark the message by putting a tag such as "[spam]" into the subject heading. You can then configure your email client filter to automatically put anything marked [spam] into a spam folder. Once you are confident, you can get the server itself to delete the spam. Depending on how much "training" you give them, they eventually get pretty effective, and will detect almost all spam.

As of 2004.06.27, Betsy and several of my other customers have used a free add-in called "SpamBayes". It works pretty well, until the spam gets more sophisticated, which it will. It took a while to "train" it, since you have to manually transfer numerous examples of both good and bad messages to its database. Once trained, it has folders for 2 categories: "Suspect" and "Spam", in addition to your in box. You can inspect either folder any time you wonder how it is doing. On about 50 messages a day, it occasionally made mistakes and wrongly put things into suspect, but did not often wrongly class things as spam. Betsy found she was able to delete the spam folder. It is still time consuming: after download of 500 messages, it takes about 1 second per message to figure out if they are spam.

5. Blocking Mechanisms

There are various organizations that maintain blacklists of IP addresses, and to which you can subscribe. Both mail clients and mail servers can subscribe.

The real question is: what controls are there on what gets onto the blacklist? Otherwise anyone can blacklist an innocent IP address, including competitors, and pranksters.

SpamcopFor example, we were trying to address a severe spam problem on one of our domains, and the ISP told us to simply go and list all the IP addresses on Spamcop.net. Even if this were possible, it turns out all the addresses were innocent people whose computers had a virus. So what mechanism exists at SpamCop to ensure the integrity of their database?

Spam Networks My brother is currently using a service called "SpamNet" from http://Cloudmark.com, and thinks it works. At least this week. The basic idea is that you pay 3.00/month, and that any members can tag individual messages as spam. So its individual message based, rather than IP based. That ensures that innocent people are not blacklisted. However, I think the spammers will soon find ways to vary their messages so that no two messages are exactly the same. And this "network" approach still relies on volunteers to tag the spam. So I'd say its a waste of time.

Individual User Blocking: The basic idea is to keep a list of "from" addresses that are sending out spam. No-one does this anymore, because the spammers just generate hundreds of new names per minute.

IP Blocking Since spammers cannot fake the sending IP address, this sounds promising.

Domain Blocking: The basic idea is to blacklist domain names which send spam. There are two problems:
 1. Spammers can fake the return domain, and most ISP's do not do a "reverse lookup". Although every IP address used by a mail server is supposed to be registered such that the receiver can do a 'reverse lookup'.

What needs to happen is that every mail server should

is needed is a "reverse lookup". We tried to do this for a few months, but the problem is that we kept running into

6. Why ISP Spam Scrubbers Don't Work

By "ISP Spam Scrubber" I mean a service offered by your ISP which supposedly removes all your spam, without any effort on your part. Many of them use both "word filters" and "lookup services".

Case 1: I recently experimented with an expensive spam filter called Postini, on behalf of customers. The salesman gave me the line that I didn't have to understand what it was doing, but just accept that it will work. Well it didn't. What happened was that Postini rejected a significant number of legitimate email messages, and at the same time let some junk mail through.

Beware of any Spam Scrubber that doesn't allow you to inspect what it rejects

We don't have a Spam filter on our mail service. Our mail service delivers ALL email to your account. If you want to mess around with a personal spam filter, that is your business, but at least you'll know what is going on. A much better approach is to use aliases to prevent your spam problem from getting out of hand.

7. Summary of Spam Prevention (as of 2004.06.28)

There is only one thing that really works in the long run, and that is a Whitelist/Blacklist scheme, with Challenge/Response. See the separate chapter on this. Anything else is temporary.

Lately a lot of ISP's have started offering "Spam Scrubbing" as part of their service. It sounds good - they just remove all the spam and you don't even have to think or understand what they are doing. Unfortunately they don't work. What happpens is they are deleting an increasing number of legitimate mail messages. They are based on a flawed theory. Despite whatever fancy language may be used to advertise them, what they do is look for word patterns. The spammers are well aware of this, and study how to write messages that trick them. This leads to the filters trying to look for all sorts of marginal indications of what might be spam. The more they do this, the more they make mistakes. There is no "magic bullet".

Under no circumstances use a spam scrubber service where you cannot personally inspect which messages are being rejected.

"Baysian" spam filters are a bit better than the regular spam filters, but still limited. The way they work is you "train" them by feeding them examples of "good" messages and "bad" messages. Eg: if your business is "mountaineering", then messages that talk about mountaineering are more probably not spam. One free Baysian filter, shich I describe in a later chapter is called "SpamBayes". Our conclusion is that these filters are somewhat useful right now, (except for your webmail), but they are time consuming to set up. And in the long run, they will be increasingly beaten by clever spam such as "Here is your report".

The proper long range solution is for the mail servers to properly support individual "white Lists" of people from whom you will accept email. A white list is similar to your address book, except that people emailing you for the first time can add themselves to it. The details of whitelists are explained in a later chapter.

We did a 2 month trial of a piece of software that implemented "whitelists" and "blacklists" with our existing mail server. From an operational viewpoint, it was 100% effective. Zero spam and zero mistakes. Only problem was the program itself would crash our mail server. So all we need to find is an other program/mail server combination that works and all your problems will be over.

In the meantime, my recommendation is prevention. Prevent your email address from getting onto spam lists. There are several simple techniques for this. If your email address is already on spam lists, change your address. This takes about 30 seconds on our server. Then send your new address to everyone in your address book. (phone me if you need help).

Once you have a fresh email address, only give out "alias" addresses to all but the most trusted clients. An alias is just another email address to which your email account will answer. For example, if your main account is "Fred@mydomain.com", an alias might be "Web@mydomain.com. You can add and delete aliases to your mail account whenever you feel like it. Any time one of the aliases starts to draw spam, delete it. If you have to provide some list with an address, provide them with an alias. Name your aliases such that you can trace any leakages, and independantly discontinue that particular alias.

The most important technique I know for managing your email addresses. solution is aliases. A "alias" is just another variation of your email address. The idea is you can make the aliases "temporary" and change them easily without disrupting your account. On your website, post a temporary "alias" address, and change it every few months. For example, suppose your permanent email address is Fred@MyDomain.com. Go into your email account and add a second alias to the same account called "FromWeb47@MyDomain.com. Use that address everywhere on your website. Only give your permanent address out when someone trustworthy contacts you.

So far I have just talked about "temporary aliases". You can also have permanent aliases, that you only intend to change if a problem exists. For example, if some website requires your email address, give them a permanent alias, specifically named for that website. For example, I advertise in Overture, so I give them an email address called "Overture@MyDomain.com". As far as I know, I'll never have to change that, but if it somehow wound up on a junk list, I would delete overture and replace it with Overture2.MyDomain.com.

Another standard trick you can use when listing your email address on your website is to change a couple of characters. This beats the automatic email address harvesting programs that the spammers use. For example, below is what you might list on your website:

To Email Me: Fred$MyDomain.com (Change the $ to an @)

Vulnerabilities: The only remaining vulnerability of the stuff I've talked about so far is if somebody writes a virus that gets into one of your contacts address book (in Microsoft Outlook), and gets access to their contact list. However, as soon as it happens, you can either delete their alias, or "blacklist" that address.