AVSx: Hardening the SPAM Perimeter

In the course of a recent AVSx rollout, we had the opportunity to mitigate the serious SPAM problem of a customer. This included analyzing the specific situation, considering different approaches to eliminate or at least minimize known issues and also involved a detailed measurement of the spam statistics over time, ultimately leading to a short whitepaper which is yet to be published. One important element of our first approach is to not depend too much on external (or internal) servers at runtime, so that the solution is pretty much self-contained and thus working autonomously. Pre-Rollout Situation The customer receives thousands of spammails per day, many of which get forwarded to the internal mailserver. Since at that internal mailserver, no catchall account is defined (the sub-contractor says that this is not possible), it occurs that non-delivery-reports (NDR) are sent out. And lots of them. This has also led to the problem that the customer itself even got blacklisted in recent months by this well-known backscatter problem. Also, there was heavy usage of black- and whitelists, which may have influenced the overall situation in a negative way as well, since that badly interferes w/ bayes filtering and the overall learning process if not handled carefully. Improvement v1: Static LDAP There are multiple ways to reduce the amount of bad e-mail, but the most commonly used  overall practice is to make use of spamassassin and clamav. Having these tools at hand together w/ a good mailserver S/W like postfix, things are to be tuned in a very efficient way. Postfix hardening already brings quite some mechanisms. Using block- and blacklists is another approach which should not be used from the start on, because the spamfilter might not get trained if nearly no mail reaches the server itself. Imagine a non-trained spamfilter if the blacklist is unreachable. In my eyes, it is crucial to closely survey the situation of spam occurence and scores as a basis to create statistics. This again serves as a base for the definitition of the SPAM tag- and kill-levels, which should normally be set to 4.5 and 16 respectively. When analyzing the score distribution, we can identify a peak which should ideally be not too far left of the kill-level, or in contrast, on a well trained system, even right of the kill level, which means that already most mails get discarded. But what about all the NDRs? We can query the internal LDAP server to get the information we need. So, at first, installing ldapvi makes sense. Then, if we have valid credentials, we can simply do

ldapvi -b "ou=xxx, dc=xxx, dc=xx" -h v.w.x.y -D you@yourdomain > ALLDATA.ldap.txt

and then get a list of valid e-mail adresses out of this by doing

grep "yourdomain" ALLDATA.ldap.txt | grep mail | cut -d ":" -f 2 | cut -d " " -f 2 | tee -a /etc/postfix/relay_recipients

This list of emails is the base for the definition of a relay_recipients file, which will accept mails only if the corresponding adresses are also found in the LDAP directory. This eliminates the complete problematic NDR situation previously found, b/c the internal mailserver normally never has to state that mail is sent to a user w/o a corresponding mail: entry in the LDAP directory dump. In a static LDAP scenario, if an account gets removed or the internal mailserver itself has problems, would produce a NDR. One could query the LDAP DB once a month and recreate the relay_recipients file from that, but in general I guess manually changing/adding/removing users in the file makes more sense, and again, we do not want to depend too heavy on other servers.  To use the relay_recipients file, the following parameter should be set in the postfix config

d1g@isp:~$ sudo postconf relay_recipient_maps
relay_recipient_maps = hash:/etc/postfix/relay_recipients

The maintenance of the valid recpients - if needed at all - could optionally be done by the local admin if we have a (cron-)skript in place that rebuilds the DB regularly by doing e.g.

postmap -v /etc/postfix/relay_recipients
postmap: name_mask: all
postmap: inet_addr_local: configured 2 IPv4 addresses
postmap: inet_addr_local: configured 2 IPv6 addresses
postmap: set_eugid: euid 1000 egid 1000
postmap: open hash relay_recipients
postmap: Compiled against Berkeley DB: 5.3.28?
postmap: Run-time linked against Berkeley DB: 5.3.28?

In this static LDAP setup, the mailserver accepts mail only for previously defined valid users, and simply by this already rejects a very high percentage of all spam mails, especially all that produced the problematic NDRs. Improvement v2: Dynamic Lookups While solely relying on dynamic lookups might get you into trouble as described above, having dynamic lookups only in cases where the recipient is not found statically makes sense, and also brings any change made in the LDAP directory immediately to the outside world. In order to enable the dynamic lookup feature, we have to first install postfix-ldap, and then define

relay_recipient_maps = hash:/etc/postfix/relay_recipients, ldap:/etc/postfix/ldap-aliases.cf

The tricky part is the ldap-aliases.cf file itself, as in our case it had to contain

server_host = x.x.x.x
search_base = ou=xxx, dc=xxx, dc=xx
version = 3
timeout = 10
leaf_result_attribute = mail
bind_dn = user@domain
bind_pw = userpassword
query_filter = (mail=%s) 
result_attribute = mail, addressToForward

Afterwards, restart postfix, and/or optionally test the setup by doing

postmap -vq user@domain ldap:/etc/postfix/ldap-aliases.cf

So, we now have both a static and dynamic mechanism in place, which makes the system rather failsafe and ready for immediate LDAP directory change propagation. Last but not least: Keep in mind - if a valid user is listed in LDAP, but the corresponding mailbox is not available for whatever reason on the local mailserver, non-delivery receipts (NDR) will be sent out! Improvement v3: Query Proxy Addresses In some cases, the ldap query has to be adjusted to the given scenario:

server_host = x.x.x.x
search_base = ou=xxx, dc=xxx, dc=xx
version = 3
timeout = 10
leaf_result_attribute = mail
bind_dn = user@domain
bind_pw = userpassword
query_filter = (proxyAddresses=smtp:%s) 
result_attribute = mail, addressToForward

After a restart of postfix, the mechanism works as intended.