Ramblings of Daniel Graziotin

If you can’t beat ‘em..clean ’em. Using imapfilter for remote rules to an IMAP mailbox

I

Software engineers receive tons of e-mails. Academics do, too. I am both. As several users do, I use inbox rules to flow my incoming messages. For example, e-mails concerning call for papers go straight to an inbox folder called CFP. E-mails related to editorial concerns go to a folder called EDITORIAL. E-mails sent to several distribution lists go to..the trash.

And so on.

When using systems such as Gmail and Exchange, creating inbox rules (or filters) is rather simple. However, there are several scenarios in which server-side rules do not apply, nor would importing the e-mails to services such as Gmail. One could do client-side filtering, for example by creating rules in clients such as Apple.mail. However, what about a smartphone or a tablet? I wanted to have a way to simulate server-side mail filtering with rules so that e-mails on my Macbook Pro, my iPhone, and my iPad are synchronized and already processed when downloaded.

Turns out that somebody has thought about this issue. Imapfilter is indeed a console-based IMAP filtering system. The program connects to one or more IMAP server and processes the e-mails based on some rules. The following diagram conceptualizes the process (yes, I did it for testing the Apple Pencil):

imapfilter

Each two minutes, the imapfilter connects to my work IMAP server, filters the e-mails according to my rules, and exits. Every 15 minutes, the e-mail clients on my devices download the mail. That is it. The system is not perfect, as it might rarely happen that the e-mails are downloaded before imapfilter processes them. However, they will get sorted out anyway at the next run. I am happy with it.

So this post is basically advertisement for imapfilter, as it made me much, much happy but it was not easy to find about it. Perhaps I was looking for wrong terms such as “serverless email filtering” or “cloud based email filter system”. Glad I found it. Imapfilter installation was just a matter of doing

sudo apt install imapfilter

The configuration was easy. I took inspirations from imapfilter man page and the posts in here and in here. The configuration has to be placed in the file config.lua within the directory ~/.imapfilter.

Here is a reduced and anonymized version of my config.lua file:

-- The following four options and contents were taken from
-- https://raymii.org/s/blog/Filtering_IMAP_mail_with_imapfilter.html
-- The time in seconds for the program to wait for a mail server's response (default 60)
options.timeout = 120
-- According to the IMAP specification, when trying to write a message to a 
-- non-existent mailbox, the server must send a hint to the client, 
-- whether it should create the mailbox and try again or not. 
-- However some IMAP servers don't follow the specification and don't 
-- send the correct response code to the client. 
-- By enabling this option the client tries to create the mailbox,
-- despite of the server's response. 
options.create = true
-- By enabling this option new mailboxes that were automatically created,
-- get also subscribed; they are set active in order for IMAP clients to recognize them
options.subscribe = true
-- Normally, messages are marked for deletion and are actually 
-- deleted when the mailbox is closed. When this option is enabled,
-- messages are expunged immediately after being marked deleted.
options.expunge = true

-- server configuration settings
workserver = IMAP {
  server = "new-work-server.com",
  username = "daniel",
  password = "hunter2",
  port = 993,
  ssl = "tls1"
}

-- rules for mails regarding CHASE workshop
-- Either the subject contains "chase", or the
-- to: field contains "chaseresearch.org" or "chase2016"
chase = (
    workserver.INBOX:contain_subject("chase") +
    workserver.INBOX:contain_to("chaseresearch.org") +
    workserver.INBOX:contain_to("chase2016")
)
-- move the messages identified by the rule into the CHASE folder
chase:move_messages(workserver.CHASE)

-- rules for mails regarding JORS journal
-- Either the subject contains "JORS", or the
-- mail text contains "JORS", or the mail comes
-- from a domain associated to JORS
jors = (
    workserver.INBOX:contain_subject("JORS") + 
    workserver.INBOX:contain_body("JORS") +
    workserver.INBOX:contain_from("software.ac.uk") +
    workserver.INBOX:contain_from("ubiquitypress.com")
)
-- move the messages identified by the rule into the EDITORIAL folder
jors:move_messages(workserver.EDITORIAL)

-- move those e-mails related to call for papers to a CFP folder
cfp = (
    workserver.INBOX:contain_subject("AISWorld") +
    workserver.INBOX:contain_subject("SEworld") +
    workserver.INBOX:contain_subject("CFP") +
    workserver.INBOX:contain_subject("Call for papers")
)
cfp:move_messages(workserver.CFP)

-- handle the tons of mails from ACM and IEEE unless they
-- mention a conference I am often associated with (note the -)
societies = (
    workserver.INBOX:contain_to("acm.org") + 
    workserver.INBOX:contain_to("ieee.org") -
    workserver.INBOX:contain_message("PROFES")
)
societies:move_messages(workserver.SOCIETIES)

-- all mails received by the old work's account (redirected to the new one), that
-- are not directed directly to me are moved to OLDWORK
oldworknotdirect = (
    workserver.INBOX:contain_to("somemail@oldwork.it") +
    workserver.INBOX:contain_to("distribution_list@oldwork.it") 
)
oldworknotdirect:move_messages(workserver.OLDWORK)

For running imapfilter each two minutes, I created the file ~/bin/cleanimap.sh, which uses a PID file system to run imapfilter only when it is not running already (for sessions longer than two minutes).

#!/bin/bash
mkdir -p "$HOME/tmp"
PIDFILE="$HOME/tmp/imapfilter.pid"

if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |
                           grep -P "^\s*$(cat ${PIDFILE})$" &> /dev/null); then
  echo "Already running."
  exit 99
fi

/usr/bin/imapfilter -c /home/daniel/.imapfilter/config.lua > $HOME/tmp/imapfilter.log &

echo $! > "${PIDFILE}"
chmod 644 "${PIDFILE}"

Finally, for excuting the script each two minutes, I run the command crontab -e, and I added the following line.

*/2 * * * * /home/daniel/bin/cleanimap.sh

About the author

dgraziotin

Dr. Daniel Graziotin received his PhD in computer science, software engineering at the Free University of Bozen-Bolzano, Italy. His research interests include human aspects in empirical software engineering with psychological measurements, Web engineering, and open science. He researches, publishes, and reviews for venues in software engineering, human-computer interaction, and psychology. Daniel is the founder of the psychoempirical software engineering discipline and guidelines. He is associate editor at the Journal of Open Research Software, academic editor at the Research Ideas and Outcomes (RIO) journal, and academic editor at the Open Communications in Computer Science journal. He is the local coordinator of the Italian Open science local group for the Open Knowledge Foundation. He is a member of ACM, SIGSOFT, and IEEE.

Ramblings of Daniel Graziotin

About Author

dgraziotin

Dr. Daniel Graziotin received his PhD in computer science, software engineering at the Free University of Bozen-Bolzano, Italy. His research interests include human aspects in empirical software engineering with psychological measurements, Web engineering, and open science. He researches, publishes, and reviews for venues in software engineering, human-computer interaction, and psychology. Daniel is the founder of the psychoempirical software engineering discipline and guidelines. He is associate editor at the Journal of Open Research Software, academic editor at the Research Ideas and Outcomes (RIO) journal, and academic editor at the Open Communications in Computer Science journal. He is the local coordinator of the Italian Open science local group for the Open Knowledge Foundation. He is a member of ACM, SIGSOFT, and IEEE.