Refiltering Mail

Once you have a set of sieve rules set up to filter your mail into folders, you probably don't want to manually file mail. However, when importing new mail into your account through a mail client (MUA), or correcting a misclassified mail from your spam filter (such as with the antispam plugin), you will have to choose a destination folder for the mail manually. This HOWTO helps you create a folder for mail that needs refiltering, and have dovecot re-deliver and re-filter that mail.

Note: It is now also possible to request re-filtering from the command line.

Prerequisites

This HOWTO assumes that you use the IMAP protocol, a Maildir mail store in ~/.maildir, and the dovecot-lda.

You will also need:

getmail
procmail (for formail only; other alternatives may exist)
moreutils (for lckdo; other alternatives exist)
incron, and a Linux kernel with inotify support (optional, but highly recommended)

Using getmail to refilter mail

First, create the directory ~/.getmail, change its permissions with chmod 700 ~/.getmail, and create ~/.getmail/getmailrc-refilter with the following contents:

[retriever]
type = SimpleIMAPSSLRetriever
server = localhost
port = 993
username = yourusername
password = yourpassword
mailboxes = ('REFILTER',)

[filter-1]
type = Filter_external
path = /usr/bin/formail
arguments = ("-a","X-YourName-Refiltered: yes")
exitcodes_drop = ()

[destination]
type = MDA_external
path = /usr/lib/dovecot/dovecot-lda
arguments = ("-e",)

[options]
delete = true
received = false
delivered_to = false

Running getmail with this getmailrc will:

Log into the imaps server on localhost (use SimpleIMAPRetriever if you do not use SSL) with the specified username and password
Retrieve mails from the folder REFILTER
Run each mail through formail, adding a header that flags refiltered mail. This header allows your sieve scripts to filter refiltered mail differently; for instance, they could ignore any spam headers on refiltered mail, so that you can refilter misclassified mail without it ending up right back where you moved it from. You should customize the header to include something unique like your name, to ensure that spammers cannot get a free pass by setting a commonly used header.
Pass each mail to Dovecot's LDA dovecot-lda for filtering and delivery. The -e ensures that dovecot will return an error if filtering or delivery fails; getmail will leave the mail alone if such an error occurs, to prevent you from losing mail.
Delete mail from REFILTER, only after successfully delivering it.
received = false and delivered_to = false prevent getmail from adding additional headers to your mail.

So, to run getmail and refilter your mail:

getmail -g ~/.getmail -r getmailrc-refilter

Refilter virtual accounts

When using getmail to refilter virtual mail accounts, more configuration is needed. Here is an example of a config file used to redeliver mail for a virtual account:

[retriever]
type = MultidropIMAPSSLRetriever
envelope_recipient = delivered-to:1
server = localhost
port = 993
username = yourusername
password = yourpassword
mailboxes = ("REDELIVER",)
move_on_delete = .Trash

[destination]
type = MDA_external
path = /usr/local/libexec/dovecot/dovecot-lda
arguments = ("-e", "-f", "%(sender)", "-d", "%(recipient)")
user = mailaccess
group = mailaccess

[options]
delete = true
recieved = false
delivered_to = false

This file use MultidropIMAPSSLRetriever in order to extract the recipient info in the mail. This is configured in the property envelope_recipient. The recipient must be passed to dovecot-lda with the -d parameter as shown with the arguments line. It's important to specify the arguments as separate strings as shown here, not "-d %(recipient)" in one string.

Another important note is that if dovecot.conf is configured to restrict allowed users (first_valid_uid), one might need to specify which user getmail should run as, as shown here with user = mailaccess.

Using cron to refilter mail every few minutes

Rather than running getmail manually after moving mail to REFILTER, you can automatically run it every few minutes with cron. Run crontab -e to edit your crontab, and put this in it:

*/5* * * * lckdo -q ~/.getmail/refilter.lock getmail -q -g ~/.getmail -r getmailrc-refilter

This will refilter your mail every 5 minutes. lckdo ensures that only one getmail runs at a time.

Using incron to refilter mail only when needed

Running getmail every 5 minutes seems wasteful, since it usually won't have anything to do; it also means you have to wait 5 minutes for mail to refilter. With incron, you can instead trigger getmail whenever you move mail into REFILTER. This means that getmail will run immediately when it has mail to refilter, and not run at all when it has nothing to do.

First, be sure to add your username to the file /etc/incron.allow:

sudo nano /etc/incron.allow

Create an executable script ~/local/bin/do-getmail with the following contents:

#!/bin/bash

dirsnotempty () {
    find "$@" -not -empty | grep -q .
}

while dirsnotempty ~/.maildir/.REFILTER/{cur,new}/ ; do
    lckdo -q -w ~/.getmail/refilter.lock \
        getmail -q -g ~/.getmail/ -r getmailrc-refilter
done

incrontab --reload # Rearm the one-shot rule

# Handle races, but don't wait
if dirsnotempty ~/.maildir/.REFILTER/{cur,new}/ ; then
    lckdo -q ~/.getmail/refilter.lock \
        getmail -q -g ~/.getmail/ -r getmailrc-refilter
fi

Run incrontab -e to edit your incrontab, and put this in it:

BEWARE: Do not add any comments nor any additional space characters or tabs! incrontab is very sensitive to this and will not run.

/home/username/.maildir/.REFILTER/cur/ IN_MOVED_TO,IN_ONESHOT /home/username/local/bin/do-getmail
/home/username/.maildir/.REFILTER/new/ IN_MOVED_TO,IN_ONESHOT /home/username/local/bin/do-getmail

This incrontab will run do-getmail whenever you add mail to the REFILTER folder. IN_ONESHOT will disable the incron rule after one event, to prevent incron from running one instance of do-getmail for every mail; otherwise, dropping a few thousand mails into REFILTER would run a few thousand do-getmail processes. do-getmail will continue running getmail as long as mail exists in REFILTER. (An instance of getmail will only process mail that existed in REFILTER when it started running, not mail added later.) do-getmail will then re-enable the one-shot incron rules. Finally, do-getmail checks one more time to see if it needs to run getmail, to avoid missing mail moved to REFILTER between the last instance of getmail and the incrontab --reload.