Sending Ham

(Image: Nemo @ pixabay)

(Image: Nemo @ pixabay)

Email is now a vital part of our lives, and also a very private one, and the choice of our email system (for both sending and receiving) deserves some thought. We may work for an institution or company that requires us to participate in their email system, but even then we may wish our personal messages to be routed via a different email provider, and more and more I see in academia that faculty and staff are choosing to use a single @gmail.com or @yahoo.com address for professional communication, rather than their institutional email. Gmail is now the largest email service provider and its ubiquity makes it easy to forget that there is a cost in privacy of using a free email service (note: this post is not about email security; why we all still write ‘e-postcards’ rather than ‘letters in e-envelopes’ is a mystery, but good news is on the way). There are some excellent paid email service providers with more acceptable privacy policies, but we implicitly end up aligning ourselves with and advertising a commercial service if we use their domain name as part of our email address; it’s like all of us having mailing addresses that are offices within a large corporation we don’t work for.

An obvious alternative for those who have purchased their own domain name is to use an email address associated with that domain. However, while receiving mail at one’s own address is generally simple, getting one’s email past ravenous spam filters can be a major problem (currently the spam filters at gmail are particularly tough). I’m sure I’m not alone in these concerns, and so this post lays out the steps I have taken to increase the chances that my own emails sent with an address at a domain I manage get to an Inbox. There was a period several weeks ago when many gmail recipients were unexpectedly non-responsive, and I started checking my outgoing @camwebb.info mail by sending to my own gmail account and found all my messages going into the gmail Junk mail folder. I suspect most people do not frequently check their junk mail folder, and so at the time perhaps 10-20% of all my emails were not being read at all, which is frustrating (perhaps I let it be more frustrating for me than it could be, but that’s another story!).

Learning about SPAM and HAM

Getting legitimate email (“ham”) round spam filters is always a game of guessing why a particular email was (or might be) classed as spam, and the gmail Spam filter is no doubt a proprietary set of software which means we’ll never know their rules. However, there is a popular and effective open-source spam-filter, SpamAssassin, which gives an insight into the challenges that a legitimate email needs to pass (see its list of tests). There are also some free testing methods for checking the ‘spamminess’ of your ham! For example, see this mail tester, or send mail to Port25’s tester. Below are the steps that I was able to take at a reasonable cost. I’m still looking into whitelisting (e.g., at dnswl.org) as another strategy.

1. Blacklists: the problem with Dreamhost

I’ve been hosting domains (including this one) with Dreamhost for many years and continue to be very impressed with their service and their ‘ethos’ - it’s a fun company to work with. However, because it has a large user base, some of the users will be spammers, sending via the default Dreamhost outgoing mail servers, which has led to the IPs of these servers appearing in some Spam Blacklists, which in turn are used by most spam filters to determine the likelihood that an email is spam. So the first thing to do in getting mail delivered was to use another SMTP relay. There may be open, free, reliable SMTP relays that I haven’t found, but it is likely that such services would also be found by spammers and the IPs would soon be contaminated. I picked a commercial service, AuthSMTP, and have been very pleased with what they offer technically, and with their support. Since their business depends on mail being delivered (they primarily support companies sending out large, legitimate mailings of newsletters), they are constantly monitoring their server IPs for blacklistings. Setting up sending to AuthSMTP with postfix (my laptop’s mail transfer agent) was easy (e.g., see this blog).

2. SPF

Using a different SMTP server is probably the best thing I can do to increase mail delivery, but having control of the DNS record for camwebb.info at Dreamhost, I can also add some DNS information to indicate that AuthSMTP is an allowed sender, i.e., I am not a spammer simply using a @camwebb.info address in the From: field. This is a protocol known as the Sender Policy Framework (SPF), and both gmail and yahoo test for this in their decision to accept a message. The DNS record is:

  TXT v=spf1 a mx include:authsmtp.com ~all

3. DKIM

Another important means of verifying that an email is authorized to come from a particular domain is ‘DomainKeys Identified Mail’ (DKIM). In the DNS record one publishes a public key, and an outgoing mail filter signs the sent message with the corresponding private key. Setting this up turned out to be very easy for postfix under Archlinux, using the OpenDKIM mail filter (see here). The DNS record is:

  TXT v=DKIM1; k=rsa; s=email; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBi
  QKBgQDepglTXTVr7ExTL xSOQSwfn08fX1KTSdw6UGN/g83IlJcJP8KYq4Vk6mjr
  IGpf/pCrfLwhcb9Kv35tnScP0m7evqMCQzzIY /Dewt/V/1wGlDDIj4BxOCCxOd9
  vz0Kpfort0TClqNQ0MzGgF5uYIEAMr7a89dsw/LUkOsDNKbsk8wIDA QAB

Now, in emails arriving at gmail there is a line indicating the successful passing of SPF and DKIM tests:

  Authentication-Results: mx.google.com; spf=pass (google.com: 
  domain of xxx@camwebb.info designates 62.13.136.55 as permitted
  sender) smtp.mail=xxx@camwebb.info; dkim=pass header.i=@camwebb.info

The above mentioned mail-tester.com also gives a very nicely laid out analysis of SPF and DKIM authentication, plus any SpamAssassin ham or spam rules triggered.

4. Hashcash

As I was looking through the SpamAssassin rules, I saw that a large ham score can be added via the use of Hashcash tokens. This is a cool idea: generate a token that requires a demonstrable and repeatable amount of CPU cycles (a ‘proof of work’) and add it as a ‘stamp’ to an email envelope. It is assumed (and likely) that spammers, sending out millions of emails, will not be able to go to the trouble to generate stamps for each email, and so the presence of a hashcash stamp is a strong indication that the email is ham, not spam. As an example, a 25-bit hashcash token takes 4.4 seconds to generate on my Lenovo X220, and adds a 4.0 ham score to the sum of all the other tests in SpamAssassin (as a comparison, a valid DKIM only adds a ham-score of 0.2). Unfortunately, it seems very few SpamAssassin users actually activate this test, so adding a hashcash stamp probably makes no difference in reality. But... it is so geeky and cool I could not resist. And I found it’s very easy to add these tokens given my current setup (gnus as an email client): just install the hashcash executable, and add:

  (setq message-generate-hashcash t)
  (setq hashcash-default-payment 25)

to your .gnus.el config file. Done! You can also add a hashcash mail filter (‘milter’) and configure postfix to pass all outgoing mail through it.

Well...

...What a lot of work! I’m well aware that I should just suck it up and use my gmail or yahoo address for all my email, like everyone else. But it troubles me that, without additional work, my email messages should either be open to analysis and profit-mining by large corporations or should fall foul of spam filters designed to deter the many antisocial users of the internet. And I can’t deny that trying to solve these little annoyances has been a lot of fun! :-)

So hopefully now my messages will be getting through to your Inbox!