Achieve Email Freedom

The Problem

These days, it is easy to accumulate large amounts of email in a short amount of time. Many email clients help you manage that email, but most require you to store your mail in their own proprietary format. These programs rarely use standard formats, and even more rarely allow you to easily export the messages to other formats.

What we need is a way to separate the mail store from the client program. It is possible to use IMAP with most clients, but then the email is stored only on the server, and you can't get at it when you're not online.

The Solution

Enter Mac OS X. With Mac OS X on your computer, you can solve this problem nicely, and with enhanced filtering abilities to boot.

What you need

There are four main parts to this setup: sendmail, imapd, fetchmail and procmail. Fortunately, the only one that isn't included with Mac OS X version 10.1.2 is imapd. You can get my pre-compiled version here or you can compile it yourself.

fetchmail is what is used to retrieve email from whatever sources you usually get your mail from. It can handle POP3 and IMAP, as well as many others. The one caveat here is that you must store your passwords for these accounts in a configuration file, but proper file permissions can mitigate this problem.

sendmail acts as the mail transfer agent (MTA), and will be used to deliver the messages to the local user(s). Some configuration changes are necessary, and will be explained below.

procmail provides powerful filtering capabilities. It can sort mail into separate folders and mailboxes. This part is optional, but, as you will see, is a large part of untying yourself from any particular mail client.

imapd provides an IMAP server on your local machine. This is what your mail client uses to communicate with the computer.

Great, How Do I Set It Up?

The first thing is imapd. Well, if you downloaded my pre-built imapd, you simply put it in /usr/local/libexec/imapd. Then edit /etc/inetd.conf and remove the ## from the line that starts with ##imap4. Now you need to kick inetd so that it will recognize the imap4 line. This is done by kill -1 pid, where pid is the process ID for inetd.
For more information on building imapd yourself, see this article on Stepwise.

Next is fetchmail. Fetchmail is already installed on Mac OS X, so all we need to do is configure it. This is fairly simple. Consider the following example:

defaults
	no fetchall
	keep

poll mail.yourhost.com
	protocol POP3
	user     yourUsernameHere
	password yourPasswordHere
	smtphost localhost

The first bit defines defaults for all configurations. no fetchall tells it to only fetch new messages. keep tells it to keep messages on the server after fetching them.

The rest is fairly self-explanatory. The things to make note of are the poll keyword and the POP3 keyword. The poll keyword tells the fetchmail program that this server should be checked every time fetchmail is run. It could be replaced with skip, which stores the configuration, but will not check that server unless expressly told to do so. POP3 can also be IMAP or any of a number of other settings. These are the only two I care about.

You need to create a file called .fetchmailrc in your home directory, containing the defaults section, and one or more poll or skip sections. You will also want to create a cron job to run fetchmail periodically to retrieve your mail. I use the following line in my crontab:

# Fetch mail every 5 minutes
1,6,11,16,21,26,31,36,41,46,51,56 * * * * /usr/bin/fetchmail --verbose > /var/log/fetchmail 2>&1

Next comes sendmail. Sendmail is already installed, but you must alter the configuration a bit to make it work. First off, edit /etc/mail/sendmail.cf, find the line that reads DontBlameSendmail=safe and replace safe with GroupWritableDirPathSafe. Edit /etc/hostconfig, find the line that reads MAILSERVER=-NO- and change it to MAILSERVER=-YES-. You can now start sendmail by running /System/Library/StartupItems/Sendmail/Sendmail.

Lastly is procmail. This is the hardest part, so I will mostly leave it to this article and this one to describe how to do the filters. Here is a simple example that I use:

# The default location for mail folders
MAILDIR=$HOME/Library/Mail/IMAP
  
# It's ALWAYS a good idea to have a logfile with Procmail.
LOGFILE=$HOME/Library/Mail/procmail.logfile

# make sure DEFAULT has a value when invoking
# Procmail with a "-m" parameter and testing rcfiles.
DEFAULT=/var/mail/$LOGNAME
 
# Johnson, Dave
:0
* ^From:.*dave@.*foobar.com
My\ Friends/Johnson,\ Dave

Most of this speaks for itself, but the Dave Johnson filter is the part to pay attention to. The line beginning with a * contains a regular expression describing the From line I want to match. The :0 line has to be there, but I don't fully understand it yet. The third line is the file, under $MAILDIR, where any matching messages should be deposited. Please note that spaces have to be escaped, and any folders have to be created before messages can be stored there. I'm not sure why this is.

Store your modifications of this in a file called .procmailrc in your home directory.

One last thing. You need to connect sendmail and procmail. This is done through your .forward file. Put the line | /usr/bin/procmail in and save.

Now we need to tell sendmail to let us run procmail. Create the /usr/adm/sm.bin/ directory. In that directory, create a symbolic link to procmail with the command ln -s /usr/bin/procmail /usr/adm/sm.bin/procmail. Sendmail will only allow programs in this directory to be run from .forward files.

Wrapping It Up

Now all of the pieces are in place. We just need to configure your client to access the imap server. This process will be different for different clients. Mainly you need to create an IMAP account, use the IMAP server as 127.0.0.1 (localhost), the SMTP server as 127.0.0.1, and the username and password to those that you use to log in to your Mac OS X account. The other thing you need to do is to set the "Root Folder" (Entourage) or "Account Path Prefix" (Mail.app) to what you set MAILDIR to here. Don't include $HOME/. Now coax your mail client to retrieve messages from the server, and you've got it made!

Other advantages come from this system. Besides procmail's excellent filtering capabilities, I (since this runs on my PowerBook) can take all of my email with me for reference, without having to be connected to the internet. This is in addition to NOT being tied to one email client. When I am working on another of my computers, I can access my mail by pointing the IMAP server at my laptop's IMAP server. But possibly the biggest benefit is that my mail is stored in standard, all-text mbox-format files. I can archive them easily, and even do plaintext searches if I need to.

YMMV

Hopefully this works for you, though of course your mileage may vary. I find this solution particularly useful to gather email from my 4 accounts and distribute it all to folders on my local machine. If you have any questions, comments, or changes, please let me know! My address is tim at macwebb dot com. I can't promise that I will be able to help, but I'll give it a shot whenever I can. I apologize for not providing a link, but on a page dedicated to making email better, I didn't want to give spammers an easy grab!