If you can’t beat ‘em..clean ’em. Using imapfilter for remote rules to an IMAP mailbox

I

Software engineers receive tons of e-mails. Academics do, too. I am both. As several users do, I use inbox rules to flow my incoming messages. For example, e-mails concerning call for papers go straight to an inbox folder called CFP. E-mails related to editorial concerns go to a folder called EDITORIAL. E-mails sent to several distribution lists go to..the trash.

And so on.

When using systems such as Gmail and Exchange, creating inbox rules (or filters) is rather simple. However, there are several scenarios in which server-side rules do not apply, nor would importing the e-mails to services such as Gmail. One could do client-side filtering, for example by creating rules in clients such as Apple.mail. However, what about a smartphone or a tablet? I wanted to have a way to simulate server-side mail filtering with rules so that e-mails on my Macbook Pro, my iPhone, and my iPad are synchronized and already processed when downloaded.

Turns out that somebody has thought about this issue. Imapfilter is indeed a console-based IMAP filtering system. The program connects to one or more IMAP server and processes the e-mails based on some rules. The following diagram conceptualizes the process (yes, I did it for testing the Apple Pencil):

imapfilter

Each two minutes, the imapfilter connects to my work IMAP server, filters the e-mails according to my rules, and exits. Every 15 minutes, the e-mail clients on my devices download the mail. That is it. The system is not perfect, as it might rarely happen that the e-mails are downloaded before imapfilter processes them. However, they will get sorted out anyway at the next run. I am happy with it.

So this post is basically advertisement for imapfilter, as it made me much, much happy but it was not easy to find about it. Perhaps I was looking for wrong terms such as “serverless email filtering” or “cloud based email filter system”. Glad I found it. Imapfilter installation was just a matter of doing

sudo apt install imapfilter

The configuration was easy. I took inspirations from imapfilter man page and the posts in here and in here. The configuration has to be placed in the file config.lua within the directory ~/.imapfilter.

Here is a reduced and anonymized version of my config.lua file:

-- The following four options and contents were taken from
-- https://raymii.org/s/blog/Filtering_IMAP_mail_with_imapfilter.html
-- The time in seconds for the program to wait for a mail server's response (default 60)
options.timeout = 120
-- According to the IMAP specification, when trying to write a message to a 
-- non-existent mailbox, the server must send a hint to the client, 
-- whether it should create the mailbox and try again or not. 
-- However some IMAP servers don't follow the specification and don't 
-- send the correct response code to the client. 
-- By enabling this option the client tries to create the mailbox,
-- despite of the server's response. 
options.create = true
-- By enabling this option new mailboxes that were automatically created,
-- get also subscribed; they are set active in order for IMAP clients to recognize them
options.subscribe = true
-- Normally, messages are marked for deletion and are actually 
-- deleted when the mailbox is closed. When this option is enabled,
-- messages are expunged immediately after being marked deleted.
options.expunge = true

-- server configuration settings
workserver = IMAP {
  server = "new-work-server.com",
  username = "daniel",
  password = "hunter2",
  port = 993,
  ssl = "tls1"
}

-- rules for mails regarding CHASE workshop
-- Either the subject contains "chase", or the
-- to: field contains "chaseresearch.org" or "chase2016"
chase = (
    workserver.INBOX:contain_subject("chase") +
    workserver.INBOX:contain_to("chaseresearch.org") +
    workserver.INBOX:contain_to("chase2016")
)
-- move the messages identified by the rule into the CHASE folder
chase:move_messages(workserver.CHASE)

-- rules for mails regarding JORS journal
-- Either the subject contains "JORS", or the
-- mail text contains "JORS", or the mail comes
-- from a domain associated to JORS
jors = (
    workserver.INBOX:contain_subject("JORS") + 
    workserver.INBOX:contain_body("JORS") +
    workserver.INBOX:contain_from("software.ac.uk") +
    workserver.INBOX:contain_from("ubiquitypress.com")
)
-- move the messages identified by the rule into the EDITORIAL folder
jors:move_messages(workserver.EDITORIAL)

-- move those e-mails related to call for papers to a CFP folder
cfp = (
    workserver.INBOX:contain_subject("AISWorld") +
    workserver.INBOX:contain_subject("SEworld") +
    workserver.INBOX:contain_subject("CFP") +
    workserver.INBOX:contain_subject("Call for papers")
)
cfp:move_messages(workserver.CFP)

-- handle the tons of mails from ACM and IEEE unless they
-- mention a conference I am often associated with (note the -)
societies = (
    workserver.INBOX:contain_to("acm.org") + 
    workserver.INBOX:contain_to("ieee.org") -
    workserver.INBOX:contain_message("PROFES")
)
societies:move_messages(workserver.SOCIETIES)

-- all mails received by the old work's account (redirected to the new one), that
-- are not directed directly to me are moved to OLDWORK
oldworknotdirect = (
    workserver.INBOX:contain_to("somemail@oldwork.it") +
    workserver.INBOX:contain_to("distribution_list@oldwork.it") 
)
oldworknotdirect:move_messages(workserver.OLDWORK)

For running imapfilter each two minutes, I created the file ~/bin/cleanimap.sh, which uses a PID file system to run imapfilter only when it is not running already (for sessions longer than two minutes).

#!/bin/bash
mkdir -p "$HOME/tmp"
PIDFILE="$HOME/tmp/imapfilter.pid"

if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |
                           grep -P "^\s*$(cat ${PIDFILE})$" &> /dev/null); then
  echo "Already running."
  exit 99
fi

/usr/bin/imapfilter -c /home/daniel/.imapfilter/config.lua > $HOME/tmp/imapfilter.log &

echo $! > "${PIDFILE}"
chmod 644 "${PIDFILE}"

Finally, for excuting the script each two minutes, I run the command crontab -e, and I added the following line.

*/2 * * * * /home/daniel/bin/cleanimap.sh

About the author

dgraziotin

Dr. Daniel Graziotin is a senior researcher (Akademischer Rat) at the University of Stuttgart, Germany. His research interests include human, behavioral, and psychological aspects of empirical software engineering, studies of science, and open science. He is associate editor at the Journal of Open Research Software and academic editor at the Research Ideas and Outcomes (RIO) journal. Daniel was awarded an Alexander von Humboldt Fellowship for postdoctoral researchers in 2017, the European Design Award (bronze) in 2016, and the Data Journalism Award in 2015. He received his Ph.D. in computer science at the Free University of Bozen-Bolzano, Italy.

About Author

dgraziotin

Dr. Daniel Graziotin is a senior researcher (Akademischer Rat) at the University of Stuttgart, Germany. His research interests include human, behavioral, and psychological aspects of empirical software engineering, studies of science, and open science. He is associate editor at the Journal of Open Research Software and academic editor at the Research Ideas and Outcomes (RIO) journal. Daniel was awarded an Alexander von Humboldt Fellowship for postdoctoral researchers in 2017, the European Design Award (bronze) in 2016, and the Data Journalism Award in 2015. He received his Ph.D. in computer science at the Free University of Bozen-Bolzano, Italy.