User Tools

Site Tools


internet:mail:smeserver_spam

This is an old revision of the document!


Spam Filter Configuration for SME 8

This is a quick configuration howto, not an in-depth look at SpamAssassin. Much more can be done beyond this document, but this will take a big dent out of your spam and free up CPU cycles on your server.

:!: If you upgraded your SME server to version 7.2 (or later) from 7.1.3 (or earlier), follow the instructions here first.

More Information

Informative URLs

Spamassassin Documentation

Enter this command at a console:

perldoc Mail::SpamAssassin::Conf

Server-Manager

Basic Settings

Using the Server-Manager Configuration/E-Mail panel, adjust the settings to these reasonable defaults.

Option Suggested Setting
Virus scanning Enabled
Spam filtering Enabled
Spam sensitivity Custom
Custom spam tagging level 4
Custom spam rejection level 12
Sort spam into junkmail folder Enabled
Modify subject of spam messagesEnabled

I would also recommend blocking all executable content. To do so, select (highlight) all of the attachment types other than zip files (control-click all but the last two).

Click Save.

:!: These Server-Manager settings alone will do a reasonably adequate job of filtering junkmail. You can stop here if you just want 'Easy and Mostly Effective'.

Adjustments

If you think you are losing misclassified mail, adjust the 'Custom spam rejection level' higher.

If too much spam is making through to your Inbox, carefully adjust the 'Custom spam tagging level' down. Many people use the level 4. Anything below that may result in false-positives. Be conservative when testing. If you want to try a more strict setting, try '3.5' first.

If too much spam is building up in your (IMAP) junkmail folder, adjust the 'Custom spam rejection level' down or change the number of days spam is kept in the junkmail folder before being automatically deleted by the server.

How It Works

When receiving an incoming message, the server first tests for RBL and DNSBL listings, if enabled. If the sender is blacklisted, the messages are blocked outright and Spamassassin never sees it.

With the above configuration:

  • The spammiest messages, those marked as 12 or above, will be rejected at the SMTP level
  • Those spam messages marked between 4 and 12, will be routed to the users' (IMAP) 'junkmail' folder
  • Users may check their junkmail folders for false-positives (valid messages that were classified as spam by SpamAssassin) via webmail, or, if they are using an IMAP mail client, by simply checking the junkmail folder exposed by their mail client.

https://servername/webmail

Advanced Filter Configuration

Clam Antivirus

Update and check your Clam Antivirus with this command. This is normally done automatically every hour via cron.

freshclam -v

or

freshclam --debug

Verify hourly update checking by viewing the “freshclam/current log” file via the Server-Manager View Log Files panel.

Realtime Blackhole Lists and DNS Blacklists

:!: The DNSBL and RBL lists are not enabled by default but are an important addition to the spam filter configuration.

:!: Enabling both of these checks should probably be the first steps you take beyond the Basic Settings listed above.

:!: Note: here in the US, we have had problems with the (European) Level 2 list at uceprotect.net. If you receive any complaints from your clients about mail not going through because it was listed on this list, I recommend you remove it. We have also had issues with SORBS, so it is also removed here.

To specify multiple RBLs, use commands like these:

config setprop qpsmtpd RBLList zen.spamhaus.org:whois.rfc-ignorant.org:dnsbl.njabl.org
config setprop qpsmtpd SBLList bogusmx.rfc-ignorant.org:multi.surbl.org:black.uribl.com:bulk.rhs.mailpolice.com:fraud.rhs.mailpolice.com:porn.rhs.mailpolice.com:adult.rhs.mailpolice.com:ex.dnsbl.org
config setprop qpsmtpd DNSBL enabled RHSBL enabled

config show qpsmtpd                    #show settings

signal-event email-update
svc -t /service/qpsmtpd

Enable/Disable Filtering Per-User

This procedure doesn't really disable the spam filtering, it just stops the spam from being routed to the 'junkmail' folder.

Per-user filtering is enabled by default. Disable filtering with the following command, as root:

db accounts setprop USERNAME SortSpam disabled
signal-event user-modify USERNAME

db accounts show USERNAME                                   # only displays settings

SPAM Retention Period

The server will automatically delete old spam in the junkmail folders after 90 days. You can control the number of days old spam is kept with the following commands.

Where 15 is the number of days you want to keep messages, do:

db configuration setprop spamassassin MessageRetentionTime 15
signal-event email-update
svc -t /service/qpsmtpd

config show spamassassin

Bayesian Autolearning

The default SME settings do not include bayesian filtering in spamassassin to allow spamassassin to learn from received email and improve over time.

The following command will enable the bayesian learning filter and set thresholds for the bayesian filter.

config setprop spamassassin UseBayes 1
config setprop spamassassin BayesAutoLearnThresholdSpam 4.00
config setprop spamassassin BayesAutoLearnThresholdNonspam 0.10
expand-template /etc/mail/spamassassin/local.cf
sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes.mutex
chmod 640 /var/spool/spamd/.spamassassin/bayes_* 
config setprop spamassassin status enabled
config setprop spamassassin RejectLevel 12
config setprop spamassassin TagLevel 4
config setprop spamassassin Sensitivity custom
signal-event email-update

These commands will:

  • Enable spamassassin
  • Configure spamassassin to reject any email with a score above 12
  • Tag spam scored between 4 and 12 in the email header
  • Enable bayesian filter
  • 'autolearn' as SPAM any email with a score above 4.00
  • 'autolearn' as HAM any email with a score below 0.10

Bayesian Filter Training and Statistics

This section has not been updated for SME8 - do not use blindly

Install the LearnAsSpam.pl and mailstats scripts, and configure nightly cron jobs like this:

cd /usr/bin
wget http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/bread/mailstats/LearnAsSpam.pl
cd /etc/cron.d
wget http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/bread/mailstats/LearnAsSpam.cron
/etc/rc.d/init.d/crond restart

Using an IMAP mail client, create a new folder called 'LearnAsSpam' (case sensitive). It can be created at the top level (like 'Inbox') or as a sub-folder. Create the folder for each user that will help train the Bayesian filter. Webmail will work fine for creating this folder, as well as for checking the junkmail (filtered mail or quarantine) folder.

If any spam messages make it past the filter and into your inbox, just move them into the LearnAsSpam folder. A nightly cron job will process them and delete them for you. This is how you train the Bayesian filter. Manual Training

You can manually train the bayesian filter with commands like these:

sa-learn --ham /home/e-smith/files/users/[your-user-name]/Maildir/cur/*
sa-learn --spam /home/e-smith/files/users/[your-user-name]/Maildir/.junkmail/cur/*

If you save real mail in other folders, you could also teach SA about those emails. If you make a mistake with a folder or a few mails just rescan them with the right rule and they will be recategorized.

Mail Statistics

This section has not been updated for SME8 - do not use blindly

See here for details on the mailstats package.

yum install --enablerepo=smecontribs smeserver-mailstats

Testing and Troubleshooting

Check the log of SMTP events using a command like:

tail /var/log/qpsmtpd/current | tai64nlocal

You can check the auto-learning statistics with this command. You will be able to note the accumulation of the spam tokens (or not). Note that the Bayesian filtering must receive 200 spam messages before it starts to function, so don't expect instantaneous results.

sa-learn --dump magic

You can check the spam filter log with this command:

tail -50 /var/log/spamd/current | tai64nlocal

Check spamassassin configuration like this:

spamassassin -D --lint

If you ever see an error such as:

warn: bayes: cannot open bayes databases /etc/mail/spamassassin/bayes_* R/W: tie failed: Permission denied

Try adjusting some permissions with these commands:

chown :spamd /var/spool/spamd/.spamassassin/*
chmod g+rw /var/spool/spamd/.spamassassin/*

Whitelist and Blacklist

If mail comes in and it is misclassified as spam, you can add the sender to the whitelist so that future messages coming in from that sender are not filtered.

Conversely, you can add a spammer to the blacklist so you never see their spam again.

Add senders (or their entire domains) to the global whitelist (or blacklist) with commands similar to these (as root):

db spamassassin setprop wbl.global *@vonage.com White
db spamassassin setprop wbl.global *domain2.com White
db spamassassin setprop wbl.global user@domain3.com White
db spamassassin setprop wbl.global spammer@spamdomain.com Black
expand-template /etc/mail/spamassassin/local.cf
svc -t /service/spamd

You can view the white/black lists with this command:

db spamassassin show

If you have a text file of e-mail addresses and/or domains (one per line and with entries formatted as above) that you want to whitelist, here is a command line to be run as root that will parse the list and add them to the database all at one time.

dos2unix /tmp/whitelist.txt    # make sure the list of addresses and domains is in unix format
for id in `cat /tmp/whitelist.txt`; do db spamassassin setprop wbl.global $id White; done

The Horde webmail whitelisting feature does not have the desired effect. However, if you have had users try to use the Horde whitelisting feature, you can extract those whitelist entries into a text file to be used as above.

Greylisting

http://wiki.contribs.org/Greylisting

Greylisting is an incredibly efficient way to cut back on spam. It does so by delaying messages from new senders. Valid senders will always retry later and spammers most often will not.

:!: Greylisting performs its miracles with an absolute minimum of server resources, so if your server is old, slow or light on resources, this is a very useful technique.

mkdir -p /var/lib/qpsmtpd/greylisting
chown qpsmtpd:qpsmtpd /var/lib/qpsmtpd/greylisting
mkdir -p /etc/e-smith/templates-custom/var/service/qpsmtpd/config/whitelisthosts
cat >> /etc/e-smith/templates-custom/var/service/qpsmtpd/config/whitelisthosts/10whitelisthosts << EOF
127.0.0.1 # Of course we don't want to delay ourselves or local users
192.168 # Don't delay our private networks either
10 # Private net (class A)
172.16 # Another private net (inidividual entries, since can't
172.17 # do a /12 netmask easily
172.18
172.19
172.20
172.21
172.22
172.23
172.24
172.25
172.26
172.27
172.28
172.29
172.30
172.31

# Public Servers

12.5.136.141 # Southwest Airlines (unique sender, no retry)
12.5.136.142 # Southwest Airlines (unique sender, no retry)
12.5.136.143 # Southwest Airlines (unique sender, no retry)
12.5.136.144 # Southwest Airlines (unique sender, no retry)
12.107.209.244 # kernel.org mailing lists (high traffic, unique sender per mail)
63.82.37.110 # SLmail
63.169.44.143 # Southwest Airlines (unique sender, no retry)
63.169.44.144 # Southwest Airlines (unique sender, no retry)
64.7.153.18 # sentex.ca (common pool)
64.12.137 # AOL (common pool) - http://postmaster.aol.com/servers/imo.html
64.12.138 # AOL (common pool)
64.124.204.39 # moveon.org (unique sender per attempt)
64.125.132.254 # collab.net (unique sender per attempt)
#64.233.162 # zproxy.gmail.com (common server pool, bad 451 handling?)
#64.233.170 # rproxy.gmail.com (common server pool, bad 451 handling?)
#64.233.182 # nproxy.gmail.com (common server pool, bad 451 handling?)
#64.233.184 # wproxy.gmail.com (common server pool, bad 451 handling?)
#65.82.241.160 # Groupwise?
66.94.237 # Yahoo Groups servers (common pool, no retry)
66.100.210.82 # Groupwise?
66.135.209 # Ebay (for time critical alerts)
66.135.197 # Ebay (common pool)
66.162.216.166 # Groupwise?
66.206.22.82 # PLEXOR
66.206.22.83 # PLEXOR
66.206.22.84 # PLEXOR
66.206.22.85 # PLEXOR
66.218.66 # Yahoo Groups servers (common pool, no retry)
66.218.67 # Yahoo Groups servers (common pool, no retry)
66.218.69 # Yahoo Groups servers (common pool, no retry)
#66.249.82 # gmail (common server pool, bad 451 handling)
66.27.51.218 # ljbtc.com (Groupwise)
#66.89.73.101 # Groupwise?
#68.15.115.88 # Groupwise?
#72.14.204 # qproxy.gmail.com (common server pool, bad 451 handling?)
152.163.225 # AOL (common pool)
194.245.101.88 # Joker.com (email forwarding server)
195.235.39.19 # Tid InfoMail Exchanger v2.20
195.238.2 # skynet.be (wierd retry pattern, common pool)
195.238.3 # skynet.be (wierd retry pattern, common pool)
#204.60.8.162 # Groupwise?
204.107.120.10 # Ameritrade (no retry)
205.188.139.136 # AOL (common pool)
205.188.139.137 # AOL (common pool)
205.188.144.207 # AOL (common pool)
205.188.144.208 # AOL (common pool)
205.188.156.66 # AOL (common pool)
205.188.157 # AOL (common pool)
205.188.159.7 # AOL (common pool)
205.206.231 # SecurityFocus.com (unique sender per attempt)
205.211.164.50 # sentex.ca (common pool)
207.115.63 # Prodigy (broken software that retries continually with no delay)
207.171.168 # Amazon.com (common pool)
207.171.180 # Amazon.com (common pool)
207.171.187 # Amazon.com (common pool)
207.171.188 # Amazon.com (common pool)
207.171.190 # Amazon.com (common pool)
#209.104.63 # Ticketmaster (poor retry config)
209.132.176.174 # sourceware.org mailing lists (high traffic, unique sender per mail)
211.29.132 # optusnet.com.au (wierd retry pattern and more than 48hrs)
213.136.52.31 # Mysql.com (unique sender)
#216.136.226.0 # Yahoo Mail?
#216.157.204.5 # Groupwise?
#216.239.56 # proxy.gmail.com (common server pool, bad 451 handling?)
217.158.50.178 # AXKit mailing list (unique sender per attempt)
EOF
expand-template /var/service/qpsmtpd/config/whitelisthosts
mkdir -p /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0
echo whitelist_soft > /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/05whitelist_soft
echo greylisting black_timeout 60 db_dir /var/lib/qpsmtpd/greylisting > /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/10greylisting
expand-template /var/service/qpsmtpd/config/peers/0
signal-event email-update
internet/mail/smeserver_spam.1364146033.txt.gz · Last modified: 2014/10/23 15:36 (external edit)