Website/content/blog/007-mutt-setup.md

title	date	description	tags
My Email Setup using Mutt and Vim	2020-05-04	I describe my email setup. It mainly consists of Mutt and Vim although there are various supporting tools as well	[Mutt Vim]

My email system is based around my email client of choice, Mutt; or more specifically NeoMutt, a fork of Mutt with several of the most common patches applied. I will refer to Mutt throughout this blog although, unless otherwise stated, I am referring to NeoMutt.

In this blog post I will document how Mutt and the many surrounding cogs interlock, to get my emails the way I like them. My goals for setting up my email were as follows:

• Exchange - Like many companies, my work emails are handled by MS Exchange. Obviously, it is important that I can read my work email.
• Offline - It is fairly common for me to work in remote locations, often resulting in the need to check my emails before I am connected to the internet.
• Powerful filtering - I get a lot of emails and keeping them organised can be a lot of work. The more this can be automated, the better.
• Terminal based - I spend a lot of time in the terminal. If email can be done in the terminal, I would like to.
• Vim - I use Vim extensively for almost all writing tasks, whether it is programming or letters. It would be ideal if emails could be included on that list.

With Mutt, and some supporting tools, I fulfil all of the above requirements.

Below are the tools I use. I will detail how I use all of them in this blog post.

• Neomutt - Email client
• NeoVim - Email editor
• Mbsync - Email downloader
• Notmuch - Email indexer (for searching)
• Msmtp - Email sender
• Davmail - Bridge for MS Exchange
• ImapFilter - Email filterer
• Abook - Contact manager

If you want to follow along, and understand how everything works, I suggest you get yourself a beer, this will probably take a couple of hours to set up first time.

Install the above tools, on Arch that would be:

sudo pacman -S neomutt isync notmuch msmtp abook
yay -S davmail imapfilter

I will try to describe the options necessary to get a working system, although this isn't a walk through of all of my configuration choices or a guide to using Mutt. To do so would make an already long blog post ridiculously long. However, feel free to peruse my dotfiles. I try to document options in the configuration file.

Connecting to Exchange - Davmail

Davmail is only required if you need to interface with a MS Exchange server that does not allow IMAP / SMTP connections. If this isn't a requirement for you, skip this step.

To start with, install and create a configuration file in ~/.config/davmail/davmail.properties.

The first option I set is to ensure that Davmail will only accept connections from localhost.

# Don't allow remote connections
davmail.allowRemote=false

For the purposes of this blog post, I will be setting it up so the various clients can connect to Davmail without an encrypted connection. That would mean, if a malicious actor or process was already on your computer, they could potentially steal credentials. If this isn't acceptable to you, it would be worth consulting Davmail's documentation about setting up an SSL.

# Don't require client to use ssl when connecting
davmail.ssl.nosecurecaldav=false
davmail.ssl.nosecureimap=false
davmail.ssl.nosecureldap=false
davmail.ssl.nosecuresmtp=false

By default, Davmail listens on ports over 1000 so that it doesn't need extra permissions to run. If you want to change the ports it listens on, you can do so with the following options:

# Ports to listen on
davmail.caldavPort=1080
davmail.imapPort=1143
davmail.ldapPort=1389
davmail.smtpPort=1025

Although Davmail doesn't store any credentials for your Exchange server, you do need to give it a URL to connect to (OWA or EWS). I also find it useful to provide a default domain.

# Exchange details
davmail.mode=Auto
davmail.defaultDomain=<PUT YOUR EXCHANGE DOMAIN HERE>
davmail.url=<PUT YOUR EXCHANGE URL HERE>

Finally, we have a couple of other preference options that should be relatively self explanatory.

# Don't start GUI
davmail.server=true
# Delete messages immediately on IMAP STORE \Deleted flag
davmail.imapAutoExpunge=true
# When a message is sent, put it in the sent folder
davmail.smtpSaveInSent=false
# Send keepalive character during large folder and messages download
davmail.enableKeepAlive=true

Start the Davmail server with davmail ~/.config/davmail/davmail.properties. You could put this in a start-up script although I quite like running it manually.

For more information on configuring Davmail, see their official documentation.

Downloading Emails - Mbsync

Mbsync is the tool I use to download all my emails to a local directory. It manages a two way sync between a local MailDir directory and in IMAP server. In my case, the IMAP server is on localhost and provided by Davmail.

The default configuration file for Mbsync is ~/.mbsyncrc. I like to keep my home directory clean so I instead keep the file in $XDG_CONFIG_HOME/isync/mbsyncrc. To make things easy for myself, I have the following alias:

alias mbsync="mbsync -c \"$XDG_CONFIG_HOME/isync/mbsyncrc\""

To start with, we create an IMAPAccount, which is, as the name suggests, the details for an IMAP account.

IMAPAccount work
# Address to connect to
Host 127.0.0.1
Port 1143
User USERNAME
PassCmd "pass work/email"
SSLType None
AuthMechs LOGIN

In my case, this is provided by Davmail, and as such, the host and port are both non-standard. If you were to use this with a normal IMAP account, you would want to change them. You would also want to use a form of encryption. Mbsync supports both STARTTLS and IMAPS.

The PassCmd option will run a command in order to retrieve a password, this means it isn't necessary to store your password in plain text in a configuration file. I am using Pass although there are many other tools that would work. Most password managers, including Lastpass, provide a command line interface.

If you are happy with the risk (or lazy), you could replace the PasCmd option with Pass and just give your password.

Next we create a store for our account. A store in Mbsync is simply one of the locations being synced. In this example, I call it work-remote.

IMAPStore work-remote
Account work

Then, we create another store for our local MailDir copy.

MailDir is a standard format for storing emails that Mutt is able to read and interface with.

MaildirStore work-local
# Copy folder hierarchy
Subfolders Verbatim
# The trailing "/" is important
Path ~/.mail/work/
Inbox ~/.mail/work/Inbox

The Subfolders option specifies how a hierarchical folder structure should be represented. I've never had to use anything other than Verbatim.

The Path and Inbox options specify where the MailDir and Inbox folders should be stored.

Lastly, we create a channel. This is what joins the two stores together in a Master-Slave relationship.

Channel work
Master :work-remote:
Slave :work-local:
# Include everything
Patterns *
# Automatically create missing mailboxes, both locally and on the server
Create Both
# Save the synchronization state files in the relevant directory
SyncState *

The Patterns option can be used to include / exclude certain folders. I prefer to keep things simple and sync everything.

That should be enough to clone your mailbox. Simply run

mbsync work

This will download your whole mailbox, including attachments. Be prepared for it to take a while.

Viewing Emails - Mutt

Mutt is an extremely powerful and configurable email manager.

To start with, you will want to create the file ~/.config/mutt/muttrc.

In there, you will want to tell Mutt about the folder that Mbsync has just created; where it is and where some of the important folders are in there.

# Folder with emails
set folder = "~/.mail/work"
# Type of mailbox
set mbox_type = Maildir
# Directory to poll for new mail
set spoolfile = +Inbox
# Directory to save sent messages into
set record = +Sent
# Sets the drafts folder
set postponed = +Drafts
# File that headers will be cached
set header_cache = ~/.cache/mutt

With just the configuration above, you should be able to open Mutt and read your emails. However, you won't have a very nice time of it. There are some settings that you will probably want.

# Sort by threads
set sort = threads
# Sort threads by last date recieved - newest first
set sort_aux = reverse-last-date-received
# Show date in year/month/day hour:minute format
set date_format="%y/%m/%d %I:%M%p"

This obviously isn't a full list of options, but it should give you an email client that is in some way similar to other clients you will have used.

HTML Email

Unfortunately, most people don't use plain text emails. This means that we need to tell Mutt how to handle HTML and multipart messages. This starts with a mailcap file.

# Mailcap file is used to tell mutt how to open different types of file
set mailcap_path = "~/.config/mutt/mailcap"

This tells Mutt what to do when it comes across different types of mime type. The following should put you in good stead for opening HTML emails:

text/html; $BROWSER %s
text/html; w3m -I %{charset} -T text/html -dump; copiousoutput;

The syntax for this file is quite simple. The line is delimited by semi-colons. Field 1 is the mime type, field 2 is the command to run, field 3 and higher are optional flags.

You will see that on the second line, the copiousoutput flag is passed which means the command is non-interactive and, as such, Mutt can simply echo the output in its normal pager. Mutt will prefer options with this flag, if trying to view an email using auto_view.

Note, I am using the text based web browser w3m, you may have to install it or use an alternative such as lynx. Additionally, you will need to have the $BROWSER environment variable set or specify the browser command you want to use in its place.

# Tells Mutt to automatically view files with these mime types
auto_view text/html
# Order to try and show multipart emails
alternative_order text/plain text/enriched text/html

The lines above tell Mutt to try and automatically view HTML emails, although it should prefer plain text.

In my experience, this works for the majority of emails. However, for maybe 5% of emails, this isn't enough. For those emails that are image heavy, or use colour to distinguish different sections, you can open the email in your normal browser. To do that, you will need to push v when viewing the email which will take you to the attachment view:

  I     1 <no description>              [multipa/alternativ, 7bit, 4.5K]
  I     2 ├─><no description>         [text/plain, 7bit, us-ascii, 0.6K]
  I     3 └─><no description>          [text/html, 7bit, us-ascii, 3.7K]

By selecting the text/html option and pushing enter, it will be opened in your browser.

Sending emails

The last essential piece to any email system is sending emails.

Neomutt has built in SMTP support, so for basic email needs all you need is:

# Use an external command to get the password
set my_smtp_pass = `pass show work/email`
# Set the smtp url
set smtp_url="smtp://USERNAME:$my_smtp_pass@127.0.0.1:1025"

If you want to include your password directly in the configuration file, you can omit the my_smtp_pass variable and simply provide your password like so:

set smtp_url="smtp://USERNAME:PASSWORD@127.0.0.1:1025"

This supports both smtp and smtps, simply by changing the protocol in the URL. If you need to use startls, you can do so with the following:

set ssl_starttls = yes

Composing Emails

One of the biggest advantages (for me) of Mutt is the ability to use Vim for composing my emails. Whether you use Vim or not, you probably have a favourite text editor.

If you do any programming, you are likely to have spent a non-trivial amount of time configuring your editor. As a result, it makes a lot of sense to use that editor for as much as you can.

To use Vim, set the following:

# Use nvim but don't force text width (looks terible if read on a phone)
set editor = "nvim +':set textwidth=0'"

Note that I set my textwidth to 0 so Vim doesn't try to hard wrap the lines. I do this because if you hard wrap, the text will look terrible on narrow devices (such as phones).

If you don't like Vim, you can use almost any text editor you like, the only catch is that the editor might need to be set to only return once you close it. This is often done with a wait flag, although you will need to consult you editor's documentation. See a couple of examples below:

# Use Sublime Text to compose email.
# -w stops sublime returning until you close
set editor = "subl -w"
# Use VS Code to compose email.
# -w stops Code returning until you close
set editor = "code -w"
# Emacs
set editor = "emacs"
# Nano
set editor = "nano"

HTML Emails

If you do need to send HTML emails, please spare a thought for your recipient; it is not just terminal email client users that could suffer. According to the National Eye Institute, one in twelve men are colour blind; so please don't use only colour to distinguish items. Additionally, many people suffer from vision-loss blindness, who are likely to use screen readers or braille displays. Complex layouts or heavy use of images are going to give these people a poor experience.

That being said, HTML emails can be used for aesthetic purposes, whilst still providing a plain text version for those who want it. That is the method I suggest adopting.

Firstly, you need to set up msmtp which is an SMTP client used to send emails. This is done with a configuration file in ~/.config/msmtp/config. It consists of 1 or more blocks like this:

# Work
account        work
host           localhost
port           1025
tls            off
tls_starttls   off
auth           on
user           USER
passwordeval   "pass show work/email"
from           jonathan.hodgson@example.com

In the msmtp file, you can replace passwordeval with password if you wish not to use a password manager.

Next I use a wrapper script called send-from-mutt which is used to determine which account to send from and whether or not to convert it to an html multipart email.

#!/bin/sh


# Put the message, send to stdin, in a variable
message="$(cat -)"
# Look at the first argument,
# Use it to determine the account to use
# If not set, assume work
# All remaining arguments should be recipient addresses which should be passed to msmtp
case "$(echo "$1" | tr '[A-Z]' '[a-z]')" in
	"work") account="work"; shift ;;
	"home") account="home"; shift ;;
	*) account="work"; ;;
esac

cleanHeaders(){
	# In the headers, delete any lines starting with markdown
	cat - | sed '0,/^$/{/^markdown/Id;}'
}

echo "$message" | sed '/^$/q' | grep -q -i 'markdown: true' \
	&& echo "$message" | cleanHeaders | convertToHtmlMultipart | msmtp --file="$config" --account="$account" "$@" \
	|| echo "$message" | cleanHeaders | msmtp --file="$config" --account="$account" "$@"

What is important is that I can put a fake header in my email:

markdown: true

Which will cause a conversion from markdown to HTML.

I also use this script to do the Markdown -> HTML conversion. Credit should go to Francisco Lopes from whom I took it. If you wish to use this, you will need to have Pandoc installed.

The final thing to do is to make Mutt use the script and add a way to change that fake header. This can be accomplished quite easily by adding the following to the muttrc file.

# Use my msmtp / markdown wrapper script to send emails using the work account
set sendmail = "send-from-mutt work"
# Puts email headers in Vim
set edit_headers=yes
# Adds a header that is used to determine whether my send script should convert the markdown to html
my_hdr Markdown: false

Now, when you compose an email, you will get something like this in Vim

From: Jonathan Hodgson <jonathan.hodgson@example.com>
To: your-recipient@example.com
Cc:
Bcc:
Subject: Your Subject
Reply-To:
Markdown: false

Your Message Here

Now, all you would need to do is change the Markdown header from false to true in order to make an HTML email.

Make sure that you leave an empty line between the headers and your message.

Contacts - Abook

Contacts are an important part of any email system. I use Abook which was designed for use with Mutt. Simply install it and run abook. You will have a terminal interface in which you can add / remove / view your contacts.

##############
#  Contacts  #
##############

# When looking for contacts, use this command
set query_command= "abook --mutt-query '%s'"
# Add current sender to address book
macro index,pager  a "<pipe-message>abook --add-email-quiet<return>" "Add this sender to Abook"
# Auto-complete email addresses by pushing tab
bind editor <Tab> complete-query

The settings above make it easy to interface with Abook in Mutt. When Mutt prompts you to enter an email address, start typing a name or email address, push tab and you will be able to select the address you want.

If viewing an email, push a in order to add the sender to your address book.

Unfortunately, Abook cannot sync with LDAP, which is what Davmail exposes when trying to get Exchange contacts. However, the address book is a very simple plain text format that looks like this:

# abook addressbook file

[format]
program=abook
version=0.6.1


[0]
name=Bob Bobbington
email=bob.bobbington@example.com
workphone=01234 567890

[1]
name=Ed Eddington
email=ed.eddington@example.com
manager=Bob Bobbington
workphone=01234 456789

So, I wrote a little shell script that would make LDAP queries and put them into the format above.

id=0
for i in {a..z}; do
	 while read line; do
		entry=$(echo "$line" | tr '§' '\n' | egrep '^(cn|mail|title|manager|mobile):' | sed 's/mail/email/' | sed 's/cn/name/' | sed 's/mobile/workphone/' | sed 's/: /=/')
		[ $(echo "$entry" | sed '/^$/d' | wc -l) -gt 0 ] && echo -e "[$id]\n$entry\n" && id=$((id + 1))
	done <<<"$( ldapsearch -H 'ldap://localhost:1389/'  -D 'DOMAIN\USERNAME' -w "$(pass show work/email)" -b "ou=people" "mail=$i*" | awk -v RS="\n\n" -v ORS="\n" '{gsub("\n","§",$0); print $0}' )"
done

I won't go into too much detail about how this works because you will almost certainly need to change some of the substitutions. One point to note is that it makes a separate LDAP query for each letter. I.e. everyone starting with an A, then a B. This is because each query seemed to be limited to 100 results. I don't know if this is to do with Davmail or Exchange but seeing as I only need to run this script when I need to update the address book, I thought it would probably be okay to loop through each letter.

Search

Normal Search

The built in search in Mutt is quite powerful. For Vim / Less style searching, you can push the / key and enter a search term. Pushing enter will take you to the first instance, you can then jump again using the n key. To make it more like Less, you might want to add the following mapping which makes Shift+N jump backwards.

# Search back
bind index N search-opposite

Note that, by default, Shift+N is used for marking a message as unread. If this is something you use, you may wish to map it to something different.

The search terms can be quite complex. For example:

'~s "mutt" ~f ("Bob +Bobbington"|"Ed +Eddington")'

Will match emails from either Bob Bobbington or Ed Eddington that contain the word mutt in the subject.

For more examples of what you can do, consult the official documentation.

Limiting Search

A limiting search is more like a filter in other email clients. Rather than jumping between matches, it will show you a list containing only emails that match the search term. You can use the same powerful searching features that can be used in the normal search. To perform a limit search, use the l key. I find these searches particularly useful when searching for an email by date:

~d 15/1/2020*2w ~f "boss@example.com"

This will show me emails from my boss 2 weeks either side of 15^th^ January

Faster, Full Body Searches - Notmuch

One limitation of both of the previous types of search is that, by default, they won't search through the body of emails. You can force it but it is slow, particularly if you have a large mailbox.

To perform more complex searches, I use Notmuch which indexes emails and provides much faster, full body searches.

The homepage claims that Notmuch is still fast when dealing with the order of "millions of messages". I can't say that I have been able to test that claim, although for all of my mailboxes, it is fast!

Notmuch looks in the environment variable NOTMUCH_CONFIG for its configuration file, defaulting to ~/.neomutt-config. Again, I like my home directory to stay clean so I simply set the environment variable to ~/.config/notmuch/config.

After setting the environment variable, you can just run notmuch setup and it will ask you for various details.

After running the setup command, you can simply run notmuch new in order to index your emails. Depending on the size of your mailbox, this could take a while.

Once it is done, you can test it out on the command line. The search options here are even more powerful, for a full list type neomutt help search-terms. As an example though, the following will show me all messages that contain the word exam in the body that have been sent / received in the past 2 weeks.

notmuch search 'body:exam date:-2weeks..now'

To make this work nicely in Mutt, add the following to your configuration file:

######################
#  NotMuch Settings  #
######################
# All the notmuch settings are documented here: https://neomutt.org/feature/notmuch

# Points to the notmuch directory
set nm_default_url = "notmuch://$HOME/.mail/work"
# Makes notmuch return threads rather than messages
set nm_query_type = "threads"
# Binding for notmuch search
bind index \\ vfolder-from-query

This will allow pushing the \ key in the index view to perform a notmuch search. It will work in a similar way to Mutt's built in limiting search in that you will be presented with a list of emails that match your search.

Another nice thing that Notmuch gives us is the ability to create Virtual mailboxes. These are similar to "Smart" mailboxes in other email clients.

virtual-mailboxes "Today's Email" "notmuch://?query=date:today"

Filtering - ImapFilter

The last, but still important, part of my setup is filtering. Of course, if you are happy to do it manually, you can do so with Mutt, but who has time for that? Again, it doesn't respect the XDG_CONFIG_HOME variable so I set an alias for it:

alias imapfilter="imapfilter -c \"$XDG_CONFIG_HOME/imapfilter/config.lua\""

This should give you a hint about why I like it. It is configured in LUA, which is a Turing complete programming language.

That means your filtering functions can be as complex as you like.

There are a couple of options that I set at the beginning of the file:

-- According to the IMAP specification, when trying to write a message
-- to a non-existent mailbox, the server must send a hint to the client,
-- whether it should create the mailbox and try again or not. However
-- some IMAP servers don't follow the specification and don't send the
-- correct response code to the client. By enabling this option the
-- client tries to create the mailbox, despite of the server's response.
-- This variable takes a boolean as a value.  Default is “false”.
options.create = true
-- By enabling this option new mailboxes that were automatically created,
-- get auto subscribed
options.subscribe = true
-- How long to wait for servers response.
options.timeout = 120

Full details can be found in the man page imapfilter_config(5).

Next, you will want to add details of your accounts. Note that ImapFilter has the ability to interface with multiple accounts and (if you wanted it to) move emails between accounts. As with everything else in this post, I will be only be setting it up with one account.

-- Gets password from pass
status, password = pipe_from('pass show work/email')
-- Setup an imap account called work
work = IMAP {
  server = "localhost",
  port = 1143,
  username = "USERNAME",
  password = password
  -- ssl = auto
}

Again, since I am using Davmail, I will be connecting to localhost and won't be encrypting the connection. If you plan on connecting to an IMAP server that is not on localhost, you will almost certainly want to include the ssl option.

Now to start filtering. With this, you are limited only by your programming ability and your imagination, here is a simple example to get you started:

-- This function takes a table of email addresses
-- and flags messages from them in the inbox.
function flagSenders(senders)
  for _, v in pairs(senders) do
    messages = work["Inbox"]:contain_from(v)
    messages:mark_flagged()
  end
end

flagSenders {
	"bob.bobbington@example.com",
	"ed.eddington@example.com"
}

Conclusion

I am very happy with this setup. This blog explains how the different pieces interlock, although doesn't explain each and every configuration choice I have made.

If you are interested in my full configuration, you can find them all in my dotfiles repository. For specific configuration:

• Mutt Configuration
• Davmail Configuration
• Mbsync Configuration
• Msmtp Configuration
• ImapFilter Configuration
• Abook Configuration
• Vim Configuration
• Email Specific Scripts