At the end of my rope with server loads caused by weblog and email spammers. SpamAssassin and Akismet etc. may keep spam away from users, but all that stuff still needs to be processed (and we’re talking about a huge percentage of all traffic).
Recently switched from the APF firewall to ConfigServer’s excellent CSF, which is integrated into WebHost Manager (the admin back-end for cPanel systems), and got thinking — the most heavily trafficked blogs here are already using spam rating systems that track IPs. The right script could harvest and rank those IPs and load them into the firewall in near real-time. Spent the past few evenings building a shell script to do just that.
reducer: Harvests bad IP addresses from multiple sources and adds them to the CSF firewall for cPanel systems. This version works with WordPress and Movable Type weblogs, and optionally the exim ACL deny system. Future versions will scan other sources for bad IPs as well.
Update, April 2008: Birdhouse Hosting has been running reducer system-wide for almost two years now, with great success. At this point, we wouldn’t even consider running a hosting business without it.
As I recall, SpamAssassin’s setup is to hand inbound messages off to the milters for (cpu-intensive) filtering, then hand off the results to MTA for local delivery. Even if you’re doing host-based blacklisting at the border, you’re still processing all the “is this a valid address” fishing messages before the local MTA gets to determine if a local recipient can handle the message.
We addressesed most of our CPU load issues on our mail gateway by – before milters – rejecting any inbound message that didn’t have a valid recipient address. In our case, the table of valid addresses (a few thousand) is generated by a cron job that runs an LDAP query against our AD server every 30 min.
Rejecting messages with invalid recipients and from blacklisted hosts BEFORE spam analysis kills about 80% of our inbound email, and greatly reduced the load from the CPU-intensive spam analysis process…
Jim, thankfully the order of operations in this case is already sensible on cPanel systems. As long as customers have set :fail: for non-existent addresses, as they should (and yes, I do check to make sure they have), the messages are dropped before they’re ever received into the server.
Can’t even imagine the loads that would be generated if we were doing spam processing on messages to fake users on legit domains. Yikes.
Ahhh, so that’s why things have felt a bit slow-ish lately (?) Good luck, and thanks!
I know my work (a university) uses a system that immediately bounces a mail from a (previously) unknown user/address. Apparently any mail server worth a grain of salt will automatically retry to deliver the message. If the message comes back because of the retry, the address is then placed on the blessed list and the message (and future messages) is delivered. Seems to get rid of most spam as undoubtedly, the real origin of spam mail is always disguised so no retry will occur. I really know very little about spam or spam filters when it comes down to it. I used to be completely immune to spam but I get hammered on my mac.com address which I now have trained Entourage’s filter to catch pretty well (although I must still periodically sift through). mac.com sure needs a better filter on it. Stupid Apple.
Interesting. My mac.com address seems blessedly spam-free, but then I’ve never advertised it. My birdhouse accts are virtually spam free as well, but I use SA pretty agressively.
Very interesting idea about bouncing messages for which there is no return attempt. I should look into that. Though it does put a small extra burden on first-time legit sending servers.