Evaluating DNSBL Effectiveness with Postfix Logs
If you run a mail server, whether it's for your own personal email or for your company and its employees, one of the biggest challenges you'll face is keeping spam at bay. Fortunately, several organizations maintain DNSBLs available for public use. These blacklists of known spammer servers and networks are the biggest hammer in a mail admin's toolbox, and are easy to configure in most common MTAs.
Used wisely, a good selection of DNSBLs can reject most of your incoming spam early in the SMTP conversation. This prevents your server from having to process those messages through other, more resource-intensive filtering strategies. On the other hand, each blacklist you enable adds another potential DNS lookup for every incoming message, and another opportunity for false positives that discard legitimate mail. To that end, it's a good idea to keep an eye on how the DNSBLs you choose are working on your system.
In this post I'll share some methods for gauging how DNSBLs are performing on a Postfix install. If you're following along with a production system, you might want to copy its
maillog to another server for processing, especially if its size exceeds your free RAM.
In judging a blacklist, there are two important metrics to consider:
How many junk transactions is it helping you block? A DNSBL that only identifies a handful of abusive senders per day may not be worth keeping around.
- How many false positives are being generated? Even the most permissive of blacklists will flag some legitimate senders as spammy from time to time.
False positives can be a tough nut to crack, because no amount of monitoring can guess when someone's expected messages didn't arrive. Only your users can provide that insight. You should make yourself (or your ticket system) approachable to your users so they can let you know when the system doesn't meet their expectations. Follow up on each report, and if a DNSBL is responsible for missing mail, weigh the benefits of its continued use against its rate of false positives.
A DNSBL's rejection rate is much easier to figure out. To discover the numbers, we'll consult the Postfix log file(s). In its default configuration, Postfix records a DNSBL rejection with a
maillog entry that looks like this:
Feb 6 20:11:39 mailman postfix/smtpd: NOQUEUE: reject: RCPT from unknown[220.127.116.11]: 554 5.7.1 Service unavailable; Client host [18.104.22.168] blocked using sbl-xbl.spamhaus.org; https://www.spamhaus.org/query/ip/22.214.171.124; from=<user@helo-host> to=<user@rcpt-host> proto=ESMTP helo=<example.com>
Since the entries are all formatted the same way, we can use some standard Linux text processing utilities to crunch the log and gather the target numbers. Here's a bash one-liner to tally up the number of messages rejected using each DNSBL. I've broken it into three lines for illustrative purposes:
grep "$(date +%b\ %e)" /var/log/maillog* | grep 'Service unavailable' | \ cut -f $((20 + $(date +%e | tr -cd ' ' | wc -c))) -d' ' | \ sort -n | uniq -c | sort -nr
The first line filters the
maillog files for records from today that indicate a DNSBL rejection. (If your
maillog rotates into zipped files, you may need to hack on this a bit.)
That funky second line determines where to find the DNSBL hostnames in the log records. If you imagine each
maillog entry as a list of columns separated by spaces, the DNSBL's hostname appears in field #20... Unless the current day of the month is only 1 digit long. In that case, Postfix pads the date with a space, so we have to shift one more column to the right.
The last line performs the accounting and sorts everything into a list.
Running that command, I get the following sample output:
5490 sbl-xbl.spamhaus.org; 66 dyna.spamrats.com; 60 psbl.surriel.com; 52 truncate.gbudb.net; 25 spam.dnsbl.anonmails.de; 11 noptr.spamrats.com; 8 dnsbl.dronebl.org; 6 all.s5h.net; 1 cbl.abuseat.org;
Here we can see each of the DNSBLs that have caused mail rejections, along with their respective counts of blocked messages. It's worth noting that there are other DNSBLs enabled, but they don't appear here because they didn't reject any messages recently.
These numbers provide visibility into which DNSBLs are working well today, but the data points will fluctuate. Some blacklists are better at catching different types of abuse than others, and may only demonstrate their value under certain circumstances. For example, the DroneBL is most effective on days when large spam runs are being sent through open SOCKS proxies. On this day, no such campaigns were ongoing, so DroneBL doesn't look like it's doing much.
Tracking data for better analysis
To get a better handle on overall performance, it's useful to track these daily statistics and periodically perform some big-picture analysis. To achieve this, I created a daily cron job that parses the
maillog as above and stores the DNSBL rejection counts to a MySQL database. You can find the script and the table schema on Github:
After collecting these statistics for awhile, you can query the database for a variety of interesting figures. Let's take a look at my DNSBL rejection counts for the entire 2017 year:
SELECT DISTINCT(dnsbl), SUM(rejections) rejections FROM dnsbl_reject_stats WHERE date BETWEEN '2017-01-01' AND '2017-12-31' GROUP BY(dnsbl) ORDER BY rejections DESC;
These numbers show that DroneBL is indeed helping to block a lot of spam, in fact it's my second best spam-killer, even though it wasn't very performant on the single day I examined before:
+-------------------------+------------+ | dnsbl | rejections | +-------------------------+------------+ | sbl-xbl.spamhaus.org | 716327 | | dnsbl.dronebl.org | 53063 | | dyna.spamrats.com | 10099 | | bad.psky.me | 8022 | /* See footnote  */ | spam.dnsbl.anonmails.de | 4735 | | psbl.surriel.com | 4545 | | noptr.spamrats.com | 4015 | | truncate.gbudb.net | 2342 | | all.s5h.net | 812 | | cbl.abuseat.org | 787 | | bb.barracudacentral.org | 730 | | bl.spameatingmonkey.net | 67 | | auth.spamrats.com | 45 | | dnsbl.anticaptcha.net | 3 | +-------------------------+------------+ 14 rows in set (0.01 sec)
Another handy data point is the mean average number of daily rejections each DNSBL generated, again looking at the entirety of 2017:
SELECT DISTINCT(dnsbl), ROUND(AVG(rejections)) rejections FROM dnsbl_reject_stats WHERE date BETWEEN '2017-01-01' AND '2017-12-31' GROUP BY(dnsbl) ORDER BY rejections DESC;
The ranking here generally aligns with the total rejection numbers:
+-------------------------+------------+ | dnsbl | rejections | +-------------------------+------------+ | sbl-xbl.spamhaus.org | 1603 | | dnsbl.dronebl.org | 126 | | bad.psky.me* | 61 | /* See footnote  */ | dyna.spamrats.com | 24 | | bb.barracudacentral.org | 17 | | spam.dnsbl.anonmails.de | 17 | | noptr.spamrats.com | 11 | | psbl.surriel.com | 10 | | truncate.gbudb.net | 8 | | cbl.abuseat.org | 6 | | all.s5h.net | 3 | | dnsbl.anticaptcha.net | 2 | | auth.spamrats.com | 1 | | bl.spameatingmonkey.net | 1 | +-------------------------+------------+ 14 rows in set (0.01 sec)
With these broader statistics in mind, you can make a more informed decision about which DNSBLs to continue using. Of course, you'll need to take several factors from your environment into account as well. Are your users complaining about too much spam, or too many false positives? Is your CTO adamant that you block every last possible junk email?
In the case of my personal mail server, I've determined that the bottom three performers don't provide enough value to leave enabled. They aren't bad blacklists, but the low numbers indicate they aren't bringing many unique hosts to the table. The vast majority of my inbound spam is being rejected by other DNSBLs first, so to me, it's not worth the processing time and DNS traffic to keep these three turned on.
bad.psky.me was a fraudulent DNSBL that was pirating its data from Spamhaus lists. I had it enabled until the ruse was discovered mid-2017, so it shows up here, but you shouldn't use it.