Introduction

I've been using Evolution, the mail client, since 2000. I think it was the first Linux program along with Netscape that I remember using on a daily basis. What can I say about a program I've used for more than 10 years? Well I'm very grateful to it and I hope its development continues and it gets better and better every day. But now it's time for a change.

Motivation

There are two main reasons I wan to switch from Evolution to Mutt:

  • Speed. Evolution is now much faster than it used to be but when you have 5 Gigabytes of email it starts to behave slowly.
  • Knowledge. As an engineer I don't feel comfortably using something I don't totally understand and I even feel embarrased about my (lack of) understanding of email, something that I use (dammit, sometimes I even depend on) every day. That's why switching to Mutt makes sense. Its take on email forces you to do it the Unix way. This mean using several small programs together to accomplish your task and by dividing the task into small pieces you understand the big picture much better. Now I can tell the difference between a MTA, a MUA  and a MDA.

General architecture

[caption id="attachment_216" align="aligncenter" width="300" caption="Notification of email arriving"]General architecture
As you can see in the above figure, there are quite a lot of small programms (blue boxes and red circle) cooperating to deliver my mail for me. I'll try to explain what each of these ones do and how to configure them in the following sections.

As there are a lot of configuration files involved I won't paste them here but point to the Mercurial repository where I keep them versioned.

Step 1. Fetching email: getmail

getmail is a MRA or Mail Retrieval Agent. Its main task is fetch your email from a POP3 or IMAP server. Unless you have a fixed IP and you run your own SMTP and IMAP/POP3 server you probably have your email delivered to your ISP or company's own servers.

Another popular MRA is fetchmail. I use getmail because it's way simpler to configure and whetever I have to decided between two programs and one of them is written in python I go for that one. I'm very comfortable with Python so if I ever need to change something or go to the source to understand what's going on, that way is easier for me.

Anyway, this is my configuration file for getmail. You have to call the getmail program every time you want to download email from your IMAP/POP3 server. A typical run looks like this:

[lgs@ticotico ~]$ getmail
getmail version 4.20.0
Copyright (C) 1998-2009 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:
Enter password for SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:  *******
  0 messages (0 bytes) retrieved, 0 skipped

As this is a very tedious task I have set up a cron task that runs every 5 minutes. Yes, I'm that anxious about email :-) But the problem is that if I put this in a cron job it will ask for the password every time it runs. One solution is to put the password in the getmailrc configuration file but I don't feel like sharing my password with the whole world and I really want to keep my configuration files in Bitbucket. So the other solution is to use expect. This is a common Unix tool for manipulating other programs. This is my getmail-no-password expect script:

#!/usr/bin/expect -f

set timeout 600
spawn /usr/bin/getmail
expect -exact "Enter password for SimpleIMAPSSLRetriever:lgs@correo.yaco.es:993:  "
send -- "put-your-real-password-here\r"
expect eof
exit

So expect will call getmail and as soon as the "Enter password ..." phrase goes to stdin it will send my real password, followed by the '\r' character. One minor issue is I have to give expect a timeout. After this time has passed expect will stop running. 600 seconds (10 minutes) is more than enough for running this script in my cron job but the first time I synchronize with my IMAP server it takes way more time than this so I just run getmail and enter my password manually instead of using the getmail-no-password expect wrapper.

This script is not in my Bitbucket repository for obvious reasons :-)

This is my crontab for fetching email:

*/5 * * * * /home/lgs/bin/getmail-no-password > /dev/null

Note that I redirect stdout to /dev/null but not stderr so cron will email me if something goes wrong with getmail or the expect wrapper script.

Step 2. Filtering email: maildrop

As soon as getmail gets the mail it can save it to the final destination, e.g. my Maildir directory, or we can send it to the MDA (Mail Delivery Agent) to do more powerful things. In my case I went for dropmail but as always there are other options like procmail. Procmail is more mature in the Unix world but I like dropmail syntax better.

getmail fetches an email message from the server and then run dropmail with the mail message as the argument. As you can see in my dropmail configuration file (called .mailfilter), I do several things with a mail message:

  • Pipe it to through reformail to add a Lines header, which is useful in Mutt.
  • Pipe it to spambayes (sb_filter) to add antispam features to my mail system. More on this later
  • Send a notification to my GNOME desktop in case the message is not spam. More on this later.
  • Filter the message and store it in one of my mail folders depending on several regular expression rules that match agains some header values of the mail message.

Step 3. Anti spam: spam bayes

The typical antispam software in Unix system is spamassassin but I use Spam Bayes because of several reasons:

  • I setup spambayes in the previous company I worked for and I was very happy with its results.
  • In my current company, spamassassin is used and it's not very effective. This may be a configuration issue.
  • Spamassassin eats a lot of cpu.
  • Did I mention that spam bayes is written in python?

The way spam bayes work is quite simple: first you need to train it giving a bunch of spam messages and a bunch of ham (ham is the opposite of spam, in other words, good mail) messages. You do it with the following command line:

sb_mboxtrain.py -g .inbox/ -s .spam/

This will create a hammie.db database in Berkeley format. Now you should write a configuration file to tell spam bayes where this database file is located so it can use it to filter spam messages when called from dropmail. Here is my spambayes configuration file.

If you remember my maildrop configuration file this was the line that calls spam bayes:

xfilter "python -W ignore:::: /usr/bin/sb_filter.py -d ~/.hammie.db"

What's that -W ignore::: argument to Python? Well spam bayes spit some warnings when run under python 2.5 or 2.6 and that's not good for maildrop because python writes the warnings to stderr and maildrop thinks there is a problem with that process and aborts the mail delivery. So by calling python with the -W ignore::::: argument we tell it to keep the warnings quiet and don't write them to stderr. I hope spam bayes get updated soon and I can remove this hack.

The good thing about spam bayes is that it can learn with your help. Any time it fails when filtering a message you can retrain it fix its behaviour. By the way, all that spam bayes does is adding some headers to the message and then you need to add some rules to your maildrop configuration file to put the message where it belongs. The interesting thing is that sometimes a message is not spam or ham because spam bayes is not sure. You can store these messages in a folder different than the spam folder so you can look it more frequently and teach spam bayes with them.

Step 4. Reading and composing emails: mutt

From all text based email clients mutt is the most popular. Israel Herraiz told me about Sup and it looks very promising. I think I'll give it a try pretty soon. But for now, I'll stick to Mutt for a while.

Mutt is the most complex software from all the programs I mentioned in this post, so its configuration file is the biggest one. Hopefully I have added enough comments to make it easier to understand the options that I use.

Step 5. Sending emails: msmtp

The last step consist in sending emails. Until recently Mutt didn't know how to talk SMTP, the protocol for sending emails and due to popular demand that feature has been added. But in keeping the original Mutt spirit and thus, the Unix way of using one program for each task, I decided to go for msmtp, a small client for sending emails. As always, there are plenty of options in Linux land but I think msmtp does a very decent job. As usual, you can check my msmtp configuration file in Bitbucket. Remember to set the set sendmail="/usr/bin/msmtp" option in Mutt.

In these days full of spam it's usual that your SMTP server ask you for authorization information unless you are in the same private network. As in the getmail case I didn't want to write my password in plain text in the msmtp configuration file. Fortunately msmtp has support for GNOME (and MacOS X) keyring so you can simple store your password in the safe keyring of your operating system and msmtp will pick it up for you there. The only issue is that you need to be logged in a graphical GNOME session in order have access to the keyring.

Anyway, to store your password in the keyring use the following command changing your username and server for your own values:

./msmtp-gnome-tool.py --set-password --username lgs --server correo.yaco.es

Unfortunately you need the source code of msmtp to use this command since it is not in the msmtp rpm package that I installed in my Fedora system. So go ahead and grab it. Once you untar it, it should be in the msmtp/scripts/msmtp-gnome-tool/msmtp-gnome-tool.py

Bonus track 1: notifications

Mutt is great and all but one thing I miss from Evolution is GNOME notifications every time a new mail reaches my inbox. Well, with such flexible system as the one I built, that's not difficult to have. Just add this line to the maildrop configuration file:

xfilter "python /home/lgs/bin/email_notificator.py"

I added after storing spam in the spam folder since I don't want to be notified when spam arrives.

Here is the content of my email_notificator.py script:

import email
import subprocess
import sys

if __name__ == '__main__':
    data = sys.stdin.read()

    message = email.message_from_string(data)
    from_ = 'Unknown from'
    if 'From' in message:
        from_ = message['From']

    subject = 'Unknown subject'
    if 'Subject' in message:
        subject = message['Subject']

    cmd = ['/usr/bin/notify-send', '-u', 'critical', '-t', '5000',
           '-i', '/usr/share/icons/gnome/48x48/actions/mail_new.png',
           '%s' % from_, '%s' % subject]
    subprocess.call(cmd, env={'DISPLAY': ':0.0'})
    print data

No rocket science here. In the following screenshot you can see the script in action:

Notification of email arriving
Notification of email arriving

Bonus track 2: logs rotation

Several programs of my email ecosystem log information to different files. As always you should rotate your logs if you don't want your hard disk to become full really fast. Just add a configuration file for log rotate as mine and add a line to your crontab like this one:

0 9 * * * /usr/sbin/logrotate /home/lgs/mail_logs/logrotate.conf --state=/home/lgs/mail_logs/logrotate.status

Bonus track 3: reading mails from my android device

One of the nice things of being a text only program is that you can use mutt from any computer that can connect through a ssh link to the computer you have your mail and mutt installed. In Android there is the fantastic ConnectBot application which let you do just that. See this screenshot in you don't believe it :-)

Mutt in Android
Mutt in Android

The only gotchas of using Mutt from my Android phone is that the GNOME integration gets in my way: now I can't send emails since msmtp will look for the password in the GNOME keyring and there is no such thing in Android. One way to fix this issue is to configure another account in msmtp which doesn't use the keyring but ask for the password manually and switch to that account with a key combination when using it from the phone.

TODO

  • Mark the spam message as already readed so I don't get distracted every time a spam message arrives my spam folder. Then once in a while I can review them looking for false positives.
  • Addressbook integration.
  • PGP integration

References