11 ms·

Your own Debian Mail Server (part II): how to prove you are not a spammer

276 points by tastalian 11y ago

efesak 11y ago
If you like Docker you can try Poste.io https://hub.docker.com/r/analogic/poste.io/ https://hub.docker.com/r/analogic/poste.io/
fensipens 11y ago
Again: SPF, DKIM and DMARC are no indicators of spamminess of a source. These systems have a completely different purpose.
- tastalian 11y ago
  Well, I understand their purpose may be different, but it is nonetheless having my e-mails ending up in "Spam" GMail boxes which triggered me to write this post ;)
  - fooyc 11y ago
    The title is confusing: even if it helps your messages to be delivered properly, it's not a mean to prove that you are not a spammer.
- dchest 11y ago
  You have to configure them in order for your mail to be successfully delivered and not put into Spam folder on major email providers.
  - dsr_ 11y ago
    This is demonstrably not true. SPF and DKIM are neither necessary for mail delivery nor sufficient to assure delivery. There is, in fact, nothing you can do to guarantee delivery of your mail once you offer it to another mail server. If the recipient doesn't like it, it will be dropped. You can do lots of things to help. The most important is to not be a spammer. Don't send substantially the same mail to lots of people who haven't asked for it.
    - merb 11y ago
      It's somewhat true. DKIM checks the authenticity of a email domain. So it definitly helps. SPF does barely the same. that's why you got into spam if you not set RDNS OR DKIM OR SPF. Since the other server can't be sure if the server is allowed to send mails with the provided domain. Mailservers are simple, basically you can send with every domain available, however that won't work since other servers will handle that via SPF, RDNS or DKIM.
    - FooBarWidget 11y ago
      > SPF and DKIM are neither necessary for mail delivery nor sufficient to assure delivery. This statement, while true, is completely useless. Without SPF and DKIM email providers will view your emails with more suspicion, so that a larger percentage of your email ends up in spam even when it really isn't. SPF and DKIM do not guarantee delivery but they reduce the chance of your email being inappropriately recognized as spam.
    - slacka 11y ago
      If it's demonstrably not true, why is it then that multiple reports of mail going into SPAM folder stop coming in, once I setup SPF and DKIM? While it's true that reverse DNS is probably a bigger factor. Not having these 2 setup is going to increase your odds of ending up in the SPAM mailbox.
      - thrownaway2424 11y ago
        Existence of SPF and DKIM will not necessarily keep your message out of the "spam" folders, but absence of them will significantly increase the chances of being delivered there.
      - spc476 11y ago
        I have reverse DNS and SPF, but not DKIM and I don't have that many issues getting my email delivered. I did have issues with AOL, but once I registered as the contact for my IP address with them, those issues went away. I've also had one or two issues with GMail over the years, but last I knew, it was okay (I normally don't deal with GMail addresses that much). Then again, I've had my domain for 17 years, self hosting everything for 16 years (with the occasional IP change, but I think I've had the same IP now for almost ten years so go figure).
  - wampus 11y ago
    I've run mail servers for decades without configuring them and have never had issues. Reputation is probably the most important (note that my domains and even some of my servers were online before these technologies existed) and it's extremely important to get your DNS right, especially Forward-confirmed reverse DNS (FCrDNS). Strictly enforce authentication on submission port 587 and segregate user submissions from application generated submissions so you can tweak each configuration appropriately. Keep in mind that marking messages as spam involves a complex chain of weighting, so if a minor adjustment gets your messages accepted, you could still be straddling a line and would benefit from fixing the basics. And never launch a server on an IP without first checking it against blacklists (demand a new one if it's listed anywhere).
    - dchest 11y ago
      Reputation is everything, but when you need to setup a new server on a new blacklist-checked IP for (non-spammy) mass mailing, without SPF and DKIM your emails will most likely go to the Spam folder, in 2015. Of course, those things are not guaranteeing delivery, but they play an important role.
    - spotman 11y ago
      Yes but you can't start from scratch without them and do a moderate amount of traffic. If you have the same clean ips from pre dkim/domainkeys days then don't lose them, or it may be an uphill battle which I would be surprised if you didn't engage dkim to aid in fighting at that point.
    - e12e 11y ago
      Google is particularly insidious: gmail will happily throw away email (not just mark as spam) to "new" recipients, while your own account, which will usually already have a "relationship" with your domain, might receive email just fine. I just recently had an issue where I tried to send an email to a someone I'd just met. The cc-part that went to my gmail-account got through fine. He didn't even receive spam. After I set up spf, I successfully sent an email to the exact same gmail address. If gmail had rejected the mail, there'd be no problem -- then I'd know that I'd have to take action. Quietly eating the mail... not cool. I wonder how long until the only way to send email into gmail/outlook is to set up routing rules that send email to gmail/outlook addresses by logging in to those respective services, and sending directly, bypassing traditional unauthenticated smtp... presumably setting up one "major" delivery would be enough, as gmail can't ignore outlook.com and vice-versa...
- e12e 11y ago
  I've been running my own (personal) email for a while now. And for a while, both hotmail/outlook.com and gmail have been eating my mail. Google is enough of an asshole to not report anything to the sending smtp, while outlook.com/hotmail at least gives you an error, so you know they got the mail all right, just didn't like your sending ip. My ip/domain name was in no (public) black lists, however - when I finally set up SPF google stopped black-holing my email. After I managed to get hold of admins of outlook.com via (I think, there were a few redundant hoops I jumped through): https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&locale=en-us&ccsid=635707736344041971 https://support.microsoft.com/en-us/getsupport?oaspworkflow=... Outlook.com/hotmail.com provisionally started accepting my email again. All this without my domain sending any spam the past few years. Personally I think SPF is rather silly, but apparently it's considered an important filter-knob by certain services. I'd much rather gmail/outlook require valid certificates for smtp, and turn of plain-text, than all these add-on protocols that are supposed to avoid "forged sender"-type stuff. Then I'd have to move over from cacert to a "real" cert, but hopefully that bar will be easier to clear once letsencrypt is up and running.
INTPenis 11y ago
Is anyone else peeved at the fact that SpamAssassin is still the de-facto standard for spam filtering? Works pretty well for post-queue but as soon as you try to pre-queue filter anything you're in deep trouble. Has anyone tried compiling SpamAssassin or writing a faster version of it in another language? It was a long time since I played with perllibs but I seem to remember being able to load perl code into c programs.
- hendry 11y ago
  Greylisting is the #1 standard for spam filtering.
  - spc476 11y ago
    Um ... maybe. I run a greylist daemon and I wrote about its effectiveness a few months ago (http://boston.conman.org/2015/04/12.1 http://boston.conman.org/2015/04/12.1), which also goes into how effective SPF would be had I actually used it in accepting incoming email. I also wrote about the effectiveness of the various blacklists out there as well (http://boston.conman.org/2015/05/11.1 http://boston.conman.org/2015/05/11.1).
- pmlnr 11y ago
  http://dspam.nuclearelephant.com/ http://dspam.nuclearelephant.com/ It's a bit picky and horribly documented, but extremely fast and low on resources.
  - spotman 11y ago
    Except database size. Will grow to gigs and gigs and gigs and gigs. Dspam can be very difficult to manage with a 400gb token database when you have a large system. ( last version I used was 3.9 maybe they have improved this?)
- pflanze 11y ago
  (The TL,DR of the following: SpamAssassin still works very well, and can be used on the incoming SMTP connection just fine ("pre-queue" as you prefer to call it).) I used SpamAssassin ca. 2000-2008, then moved to Gmail, and recently went back to maintaining my non-@gmail.com addresses with my own mail setup, again with SA. In the last 30 days I got 43 spams delivered to my spam folders, 16 spams delivered to non-existing addresses at my domains (captured to prevent backscatter), and 101 spams rejected right away on the SMTP level due to high enough score, and IIRC zero delivered to my inbox (IIRC even across the last 2-3 months). I'm not sure why so few spams are even being sent to my handful of non-@gmail.com addresses, I've been using some of them on mailing lists for about 8 years now, too. (My single @gmail.com address gets about 40 spams in the Gmail spam folder per day.) I got 2 false "half-positives" in the last 30 days from the same company before training them (and 0 fp to the spam folder); I say half-positives sine I'm filtering mails with a low enough score to a "possible spam" folder, with the idea of reducing the amount of work for checking. It took some effort to get everything working well (not the fault of SpamAssassin, except for pulling some hair about its relatively messy setup (quite complex and not very clean, so needs a calm mind when doing a non-standard setup)). I wrote some software[0] for this, though some people will shake their heads that I'm using djbdns and Qmail (my motivation for the latter is that I know its workings and that it's the original backend for qpsmtpd). [0] https://github.com/pflanze/better-qmail-remote https://github.com/pflanze/better-qmail-remote, https://github.com/pflanze/mailmover https://github.com/pflanze/mailmover, (and https://github.com/pflanze/tinydns-scm https://github.com/pflanze/tinydns-scm for generating tinydns configurations programmatically using Scheme, including SPF records, once I get around cleaning up the code and pushing it here) > as soon as you try to pre-queue filter anything you're in deep trouble. I'm using qpsmtpd as the incoming SMTP server, which is also written in Perl, but it doesn't matter, as SpamAssassin offers a daemon approach (spamd) which even qpsmtpd uses, thus you just execute the small "spamc" program and pass the mail on stdin (or use a library that reimplements the protocol in the language of choice). Given this I see no reason to be in deep trouble, and rejecting spams during the SMTP stage works just fine for me. PS. yes, you can also embed Perl in a C program, but why deal with the complexities of mixing languages in the same process when scanning a mail takes several seconds of real time and a sizable fraction of a second of CPU time, thus the IPC overhead is completely negligible.
jfaucett 11y ago
My number one product I wish existed is this: A complete email server package that is easy to configure, setup, manage, is secure, and is accepted by other email service providers (gmail, yahoo, etc) out of the box. And with easy I mean as easy as apt-get install or just downloading a binary. For anyone that has configured email servers, you know it is a headache, this tutorial makes it look easy but its only adressing a a tiny portion of the problem (albiet a important one) - email spoofing. (EDIT: it does mention spam assassin at the end so there's a little bit of info about spam filtering)
- junto 11y ago
  I use virtualmin/webmin for this purpose.
  - SwellJoe 11y ago
    I work on Virtualmin. I'm glad it works well for you in this role. The mail stack is probably he single most complicated portion of the stuff Virtualmin manages (it certainly has a long dependency list). As the person that maintains some of it, I also wish there were a simpler way! The number of components we have to keep up in order to make it easy is mind blowing... That said, I'm really surprised at how many mail sending services there are. Sending mail really shouldn't be hard (and it isn't if you understand all the components, but it's still time-consuming enough to be a challenge for many). I am not one of those folks who believes email should be replaced by a whole new thing, but I do think a simplification of the stack would be lovely. How we do that without introducing even more new mail related standards is the conundrum.
- maaarghk 11y ago
  There are a few docker boxes which look quite promising, and once you have docker on your system it is actually pretty much as easy as apt-get install. (docker pull / docker run)
- maxmouchet 11y ago
  Mail-in-a-Box [1] is very well-done, with a clean and modern mail server configuration. However it is not very customisable and requires a dedicated vm. [1] https://mailinabox.email https://mailinabox.email
exratione 11y ago
A different and less comprehensive recipe to compare with for Ubuntu: https://www.exratione.com/2014/07/setting-up-spf-and-dkim-for-an-ubuntu-1404-mail-server/ https://www.exratione.com/2014/07/setting-up-spf-and-dkim-fo...
hit8run 11y ago
Been there done that. But in the end I went with http://mandrill.com/ http://mandrill.com/
jvehent 11y ago
Been doing it for 10 years, and it's still fun :) https://jve.linuxwall.info/blog/index.php?post/2015/03/11/10-years-of-self-hosting-Linuxwall.info https://jve.linuxwall.info/blog/index.php?post/2015/03/11/10...
aroch 11y ago
This is more or less a problem that you're never going to solve as a normal person running their own mailserver. Hell, apparently Google has problems with it. They mark emails I send from gmail.com to others within my own GApps org as spam because the headers don't match. What??
- deleted 11y ago
  [deleted]
plg 11y ago
what about os x server?
- josho 11y ago
  OS X server mail setup is easy. But, if you need to customize anything it starts to get scary, i.e. If you don't change the settings the OS X way then you risk an update blowing away your customizations. The OS X way is a minimally documented serveradmin tool from terminal. Oh and OS X server doesn't support anything newer than tls1.0. So for these reasons my next mail server will be something more mainstream.
  - plg 11y ago
    Yes---you're absolutely right. Thinking back, I have run into similar problems with the os x server apache implementation, where (a) anything slightly non-standard is not possible from Apple's GUI interface, but (b) editing config files by hand works UNTIL an update wipes it all away, etc.
illuminated 11y ago
I'm using kolab.org for few years, great product overall.
dfvgskdfjghs 11y ago
To me, the main obstacle is not the software (I can install a server in VM), it's the implication that I have to rent a VPS, or buy a DNS record, or subscribe to a different kind of ("business") Internet plan, when I already have packets flowing, just so that the email giants and everyone else believe I'm a legitimate participant in humanity. I feel stymied by email giants and ISPs that seem to collaborate to prevent me from doing something that even I agree is simple, to the point that I don't bother. That's 100% a social problem. I think the desire to run one's own email server today mostly reflects a longing to re-discover the highly-accessible anarchy of the early Internet. Unfortunately, if that is ever to be found, it likely won't be in the form of complying with the highly-burdensome mostly-social requirements of our modern, well-centralized, email system. More likely, it will come from painting over it.
- spotman 11y ago
  It's not just corporations that would not think your legitimate but everyone really. These sort of unspoken rules you mention are very largely due to combat spam. If anyone with port 25 open was trusted the amount of spam would be intolerable. Seems sort of silly to want to use email without taking these steps to make it official as possible. This is not some manipulative plot by google to get you to spend an extra 60 dollars a year, sorry.
  - mjn 11y ago
    Yes, treating dialup sources as likely spam sources has been done since long before Google became a significant player in email. In earlier years the main approach was to try to convince ISPs to filter outgoing port 25 by default on their dialup IP ranges. Later, people started compiling lists of dialup IP ranges (later expanded to DSL/cable/etc.) to block them at the recipient side, since there were too many ISPs who weren't filtering. Recipients disliked email from dialup IPs because ISPs seemed unwilling or unable to police their customers and respond to individual abuse reports, and so little legitimate email originated there anyway that it was easier to just cut them off. I don't think the bigger end-user providers have been very involved in developing those kinds of policies. The NANAE crowd was/is mostly administrators of smaller and university servers, not Yahoo/AOL/Hotmail/Gmail administrators.
devereaux 11y ago
I have a mail server hosted at Linode on: - a clean IP, not on RBL, not blacklisted elsewhere to the best of my knowledge (outlook.com tells you when you IP is bad) - also accessible on IPv6 - both IPs having a proper rDNS on my domain - supporting SSL on port 487 and 565, with a certificate from a known authority - with DKIM and SPF both passing according to gmail Yet it ends up in gmail spam folder. And I'm only sending email to myself and 2 other persons, so it's not even mass mailing. I think there are other factors at play.
- spotman 11y ago
  Check PBL too
- deftnerd 11y ago
  That must be heart-breaking and very frustrating. You've done everything you can do to be a good internet citizen and support all the proper technology, but you're still treated badly. If I am correct, gmail now defaults to a "don't trust an IP by default" procedure. If enough people get your emails in a spam folder and mark it as not being spam, the system will start trusting the IP address. It also has to do with volume. This might be a good service offering. If someone could operate a few hundred gmail accounts, they could let people send them messages and mark those messages as not being spam in order to train the filters.
  - thrownaway2424 11y ago
    Such accounts would be permanently disabled. Spammers have been trying to use sock-puppet networks for years.
- deleted 11y ago
  [deleted]
- sn 11y ago
  Google wants you to do special things for them. Have you gone to https://postmaster.google.com https://postmaster.google.com and set up your domain there?
soneil 11y ago
I'm not sure I'd put much stock in the linked mail-tester. It has a very incomplete SPF implementation that'll heavily penalise perfectly valid records. (eg, my record is simply "mx -all". So if a host is listed as an MX for my domain, it's also a valid sender. mail-tester doesn't appear to understand 'a' or 'mx' entries, so fails this entirely.)
kierkegaard9 11y ago
Avoiding spam filters is unfortunately not as easy as setting up rDNS, SPF, DKIM and DMARC. Reputation is also becoming a big issue: http://liminality.xyz/the-hostile-email-landscape/ http://liminality.xyz/the-hostile-email-landscape/
alexkavon 11y ago
Lots of people think it's a great idea to make packaged all-in-one mail systems, but what you're really doing is making it easier for spammers to get up and running.
xbeta 11y ago
Kinda off-topic, but I was wondering if anyone can recommend some setup for a CLI based mail client that works with IMAP and Gmail? I'm on a Mac now.
- jlgaddis 11y ago
  I used mutt on a Mac and was quite satisfied with it, although I'm kinda "GUI-averse" anyways. mutt itself is kinda slow if you actually use its internal IMAP and SMTP features over the Internet (such as with Gmail). Instead, I used offlineimap to pull down all of my mail to the Mac and pointed mutt at the (Maildir) directory where it was stored (which was incredibly fast, as you might guess). In addition, instead of having mutt send outbound mail directly to Gmail itself, I fed messages to msmtp (locally) and let that take care of sending them off to Gmail in the background. That setup isn't for everybody, of course, but I was very happy with it. In particular, {processing/catching up on} all of the mailing lists I subscribe to was much, much quicker.
joering2 11y ago
Slight OT/marketing, but I'm constantly having deliverability issues with Yahoo and spent weeks figuring out whats wrong, when spam-tester shows score 10/10, SenderScore 97, and yahoo keeps automatically marking my messages as spam, even when many times I contact client via phone and they swear they never clicked "mark as spam", which I have no reason no to believe. The spam complaint rate keeps being broken (around 0.3%) because of yahoo, and utilizing Sendgrid as an EPS, I'm afraid of losing ability to send. At this point I would love to hear an opinion of a e-mail deliveribility expert/veteran, or someone who can help me get off the cloud and host own email server that will be well-configured and maintained. I'm aware this service might come with a hefty $ bill. Please contact me via my email in my profile. </shameless plug for help>
mpnordland 11y ago
All this is moot if your server is in a blacklisted ip range. I found it out the hard way. Not just any VPS provider will do, make sure you get one from a company with a clean ip range.

276 points by tastalian 11y ago

If you like Docker you can try Poste.io https://hub.docker.com/r/analogic/poste.io/ https://hub.docker.com/r/analogic/poste.io/

fensipens 11y ago

Again: SPF, DKIM and DMARC are no indicators of spamminess of a source. These systems have a completely different purpose.

tastalian 11y ago
Well, I understand their purpose may be different, but it is nonetheless having my e-mails ending up in "Spam" GMail boxes which triggered me to write this post ;)
- fooyc 11y ago
  The title is confusing: even if it helps your messages to be delivered properly, it's not a mean to prove that you are not a spammer.
dchest 11y ago
You have to configure them in order for your mail to be successfully delivered and not put into Spam folder on major email providers.
- dsr_ 11y ago
  This is demonstrably not true. SPF and DKIM are neither necessary for mail delivery nor sufficient to assure delivery. There is, in fact, nothing you can do to guarantee delivery of your mail once you offer it to another mail server. If the recipient doesn't like it, it will be dropped. You can do lots of things to help. The most important is to not be a spammer. Don't send substantially the same mail to lots of people who haven't asked for it.
  - merb 11y ago
    It's somewhat true. DKIM checks the authenticity of a email domain. So it definitly helps. SPF does barely the same. that's why you got into spam if you not set RDNS OR DKIM OR SPF. Since the other server can't be sure if the server is allowed to send mails with the provided domain. Mailservers are simple, basically you can send with every domain available, however that won't work since other servers will handle that via SPF, RDNS or DKIM.
  - FooBarWidget 11y ago
    > SPF and DKIM are neither necessary for mail delivery nor sufficient to assure delivery. This statement, while true, is completely useless. Without SPF and DKIM email providers will view your emails with more suspicion, so that a larger percentage of your email ends up in spam even when it really isn't. SPF and DKIM do not guarantee delivery but they reduce the chance of your email being inappropriately recognized as spam.
  - slacka 11y ago
    If it's demonstrably not true, why is it then that multiple reports of mail going into SPAM folder stop coming in, once I setup SPF and DKIM? While it's true that reverse DNS is probably a bigger factor. Not having these 2 setup is going to increase your odds of ending up in the SPAM mailbox.
    - thrownaway2424 11y ago
      Existence of SPF and DKIM will not necessarily keep your message out of the "spam" folders, but absence of them will significantly increase the chances of being delivered there.
    - spc476 11y ago
      I have reverse DNS and SPF, but not DKIM and I don't have that many issues getting my email delivered. I did have issues with AOL, but once I registered as the contact for my IP address with them, those issues went away. I've also had one or two issues with GMail over the years, but last I knew, it was okay (I normally don't deal with GMail addresses that much). Then again, I've had my domain for 17 years, self hosting everything for 16 years (with the occasional IP change, but I think I've had the same IP now for almost ten years so go figure).
- wampus 11y ago
  I've run mail servers for decades without configuring them and have never had issues. Reputation is probably the most important (note that my domains and even some of my servers were online before these technologies existed) and it's extremely important to get your DNS right, especially Forward-confirmed reverse DNS (FCrDNS). Strictly enforce authentication on submission port 587 and segregate user submissions from application generated submissions so you can tweak each configuration appropriately. Keep in mind that marking messages as spam involves a complex chain of weighting, so if a minor adjustment gets your messages accepted, you could still be straddling a line and would benefit from fixing the basics. And never launch a server on an IP without first checking it against blacklists (demand a new one if it's listed anywhere).
  - dchest 11y ago
    Reputation is everything, but when you need to setup a new server on a new blacklist-checked IP for (non-spammy) mass mailing, without SPF and DKIM your emails will most likely go to the Spam folder, in 2015. Of course, those things are not guaranteeing delivery, but they play an important role.
  - spotman 11y ago
    Yes but you can't start from scratch without them and do a moderate amount of traffic. If you have the same clean ips from pre dkim/domainkeys days then don't lose them, or it may be an uphill battle which I would be surprised if you didn't engage dkim to aid in fighting at that point.
  - e12e 11y ago
    Google is particularly insidious: gmail will happily throw away email (not just mark as spam) to "new" recipients, while your own account, which will usually already have a "relationship" with your domain, might receive email just fine. I just recently had an issue where I tried to send an email to a someone I'd just met. The cc-part that went to my gmail-account got through fine. He didn't even receive spam. After I set up spf, I successfully sent an email to the exact same gmail address. If gmail had rejected the mail, there'd be no problem -- then I'd know that I'd have to take action. Quietly eating the mail... not cool. I wonder how long until the only way to send email into gmail/outlook is to set up routing rules that send email to gmail/outlook addresses by logging in to those respective services, and sending directly, bypassing traditional unauthenticated smtp... presumably setting up one "major" delivery would be enough, as gmail can't ignore outlook.com and vice-versa...
e12e 11y ago
I've been running my own (personal) email for a while now. And for a while, both hotmail/outlook.com and gmail have been eating my mail. Google is enough of an asshole to not report anything to the sending smtp, while outlook.com/hotmail at least gives you an error, so you know they got the mail all right, just didn't like your sending ip. My ip/domain name was in no (public) black lists, however - when I finally set up SPF google stopped black-holing my email. After I managed to get hold of admins of outlook.com via (I think, there were a few redundant hoops I jumped through): https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&locale=en-us&ccsid=635707736344041971 https://support.microsoft.com/en-us/getsupport?oaspworkflow=... Outlook.com/hotmail.com provisionally started accepting my email again. All this without my domain sending any spam the past few years. Personally I think SPF is rather silly, but apparently it's considered an important filter-knob by certain services. I'd much rather gmail/outlook require valid certificates for smtp, and turn of plain-text, than all these add-on protocols that are supposed to avoid "forged sender"-type stuff. Then I'd have to move over from cacert to a "real" cert, but hopefully that bar will be easier to clear once letsencrypt is up and running.

INTPenis 11y ago

Is anyone else peeved at the fact that SpamAssassin is still the de-facto standard for spam filtering? Works pretty well for post-queue but as soon as you try to pre-queue filter anything you're in deep trouble. Has anyone tried compiling SpamAssassin or writing a faster version of it in another language? It was a long time since I played with perllibs but I seem to remember being able to load perl code into c programs.

hendry 11y ago
Greylisting is the #1 standard for spam filtering.
- spc476 11y ago
  Um ... maybe. I run a greylist daemon and I wrote about its effectiveness a few months ago (http://boston.conman.org/2015/04/12.1 http://boston.conman.org/2015/04/12.1), which also goes into how effective SPF would be had I actually used it in accepting incoming email. I also wrote about the effectiveness of the various blacklists out there as well (http://boston.conman.org/2015/05/11.1 http://boston.conman.org/2015/05/11.1).
pmlnr 11y ago
http://dspam.nuclearelephant.com/ http://dspam.nuclearelephant.com/ It's a bit picky and horribly documented, but extremely fast and low on resources.
- spotman 11y ago
  Except database size. Will grow to gigs and gigs and gigs and gigs. Dspam can be very difficult to manage with a 400gb token database when you have a large system. ( last version I used was 3.9 maybe they have improved this?)
pflanze 11y ago
(The TL,DR of the following: SpamAssassin still works very well, and can be used on the incoming SMTP connection just fine ("pre-queue" as you prefer to call it).) I used SpamAssassin ca. 2000-2008, then moved to Gmail, and recently went back to maintaining my non-@gmail.com addresses with my own mail setup, again with SA. In the last 30 days I got 43 spams delivered to my spam folders, 16 spams delivered to non-existing addresses at my domains (captured to prevent backscatter), and 101 spams rejected right away on the SMTP level due to high enough score, and IIRC zero delivered to my inbox (IIRC even across the last 2-3 months). I'm not sure why so few spams are even being sent to my handful of non-@gmail.com addresses, I've been using some of them on mailing lists for about 8 years now, too. (My single @gmail.com address gets about 40 spams in the Gmail spam folder per day.) I got 2 false "half-positives" in the last 30 days from the same company before training them (and 0 fp to the spam folder); I say half-positives sine I'm filtering mails with a low enough score to a "possible spam" folder, with the idea of reducing the amount of work for checking. It took some effort to get everything working well (not the fault of SpamAssassin, except for pulling some hair about its relatively messy setup (quite complex and not very clean, so needs a calm mind when doing a non-standard setup)). I wrote some software[0] for this, though some people will shake their heads that I'm using djbdns and Qmail (my motivation for the latter is that I know its workings and that it's the original backend for qpsmtpd). [0] https://github.com/pflanze/better-qmail-remote https://github.com/pflanze/better-qmail-remote, https://github.com/pflanze/mailmover https://github.com/pflanze/mailmover, (and https://github.com/pflanze/tinydns-scm https://github.com/pflanze/tinydns-scm for generating tinydns configurations programmatically using Scheme, including SPF records, once I get around cleaning up the code and pushing it here) > as soon as you try to pre-queue filter anything you're in deep trouble. I'm using qpsmtpd as the incoming SMTP server, which is also written in Perl, but it doesn't matter, as SpamAssassin offers a daemon approach (spamd) which even qpsmtpd uses, thus you just execute the small "spamc" program and pass the mail on stdin (or use a library that reimplements the protocol in the language of choice). Given this I see no reason to be in deep trouble, and rejecting spams during the SMTP stage works just fine for me. PS. yes, you can also embed Perl in a C program, but why deal with the complexities of mixing languages in the same process when scanning a mail takes several seconds of real time and a sizable fraction of a second of CPU time, thus the IPC overhead is completely negligible.

jfaucett 11y ago

My number one product I wish existed is this: A complete email server package that is easy to configure, setup, manage, is secure, and is accepted by other email service providers (gmail, yahoo, etc) out of the box. And with easy I mean as easy as apt-get install or just downloading a binary. For anyone that has configured email servers, you know it is a headache, this tutorial makes it look easy but its only adressing a a tiny portion of the problem (albiet a important one) - email spoofing. (EDIT: it does mention spam assassin at the end so there's a little bit of info about spam filtering)

junto 11y ago
I use virtualmin/webmin for this purpose.
- SwellJoe 11y ago
  I work on Virtualmin. I'm glad it works well for you in this role. The mail stack is probably he single most complicated portion of the stuff Virtualmin manages (it certainly has a long dependency list). As the person that maintains some of it, I also wish there were a simpler way! The number of components we have to keep up in order to make it easy is mind blowing... That said, I'm really surprised at how many mail sending services there are. Sending mail really shouldn't be hard (and it isn't if you understand all the components, but it's still time-consuming enough to be a challenge for many). I am not one of those folks who believes email should be replaced by a whole new thing, but I do think a simplification of the stack would be lovely. How we do that without introducing even more new mail related standards is the conundrum.
maaarghk 11y ago
There are a few docker boxes which look quite promising, and once you have docker on your system it is actually pretty much as easy as apt-get install. (docker pull / docker run)
maxmouchet 11y ago
Mail-in-a-Box [1] is very well-done, with a clean and modern mail server configuration. However it is not very customisable and requires a dedicated vm. [1] https://mailinabox.email https://mailinabox.email

exratione 11y ago

A different and less comprehensive recipe to compare with for Ubuntu: https://www.exratione.com/2014/07/setting-up-spf-and-dkim-for-an-ubuntu-1404-mail-server/ https://www.exratione.com/2014/07/setting-up-spf-and-dkim-fo...

hit8run 11y ago

Been there done that. But in the end I went with http://mandrill.com/ http://mandrill.com/

jvehent 11y ago

Been doing it for 10 years, and it's still fun :) https://jve.linuxwall.info/blog/index.php?post/2015/03/11/10-years-of-self-hosting-Linuxwall.info https://jve.linuxwall.info/blog/index.php?post/2015/03/11/10...

aroch 11y ago

This is more or less a problem that you're never going to solve as a normal person running their own mailserver. Hell, apparently Google has problems with it. They mark emails I send from gmail.com to others within my own GApps org as spam because the headers don't match. What??

deleted 11y ago
[deleted]

plg 11y ago

what about os x server?

josho 11y ago
OS X server mail setup is easy. But, if you need to customize anything it starts to get scary, i.e. If you don't change the settings the OS X way then you risk an update blowing away your customizations. The OS X way is a minimally documented serveradmin tool from terminal. Oh and OS X server doesn't support anything newer than tls1.0. So for these reasons my next mail server will be something more mainstream.
- plg 11y ago
  Yes---you're absolutely right. Thinking back, I have run into similar problems with the os x server apache implementation, where (a) anything slightly non-standard is not possible from Apple's GUI interface, but (b) editing config files by hand works UNTIL an update wipes it all away, etc.

illuminated 11y ago

I'm using kolab.org for few years, great product overall.

dfvgskdfjghs 11y ago

To me, the main obstacle is not the software (I can install a server in VM), it's the implication that I have to rent a VPS, or buy a DNS record, or subscribe to a different kind of ("business") Internet plan, when I already have packets flowing, just so that the email giants and everyone else believe I'm a legitimate participant in humanity. I feel stymied by email giants and ISPs that seem to collaborate to prevent me from doing something that even I agree is simple, to the point that I don't bother. That's 100% a social problem. I think the desire to run one's own email server today mostly reflects a longing to re-discover the highly-accessible anarchy of the early Internet. Unfortunately, if that is ever to be found, it likely won't be in the form of complying with the highly-burdensome mostly-social requirements of our modern, well-centralized, email system. More likely, it will come from painting over it.

spotman 11y ago
It's not just corporations that would not think your legitimate but everyone really. These sort of unspoken rules you mention are very largely due to combat spam. If anyone with port 25 open was trusted the amount of spam would be intolerable. Seems sort of silly to want to use email without taking these steps to make it official as possible. This is not some manipulative plot by google to get you to spend an extra 60 dollars a year, sorry.
- mjn 11y ago
  Yes, treating dialup sources as likely spam sources has been done since long before Google became a significant player in email. In earlier years the main approach was to try to convince ISPs to filter outgoing port 25 by default on their dialup IP ranges. Later, people started compiling lists of dialup IP ranges (later expanded to DSL/cable/etc.) to block them at the recipient side, since there were too many ISPs who weren't filtering. Recipients disliked email from dialup IPs because ISPs seemed unwilling or unable to police their customers and respond to individual abuse reports, and so little legitimate email originated there anyway that it was easier to just cut them off. I don't think the bigger end-user providers have been very involved in developing those kinds of policies. The NANAE crowd was/is mostly administrators of smaller and university servers, not Yahoo/AOL/Hotmail/Gmail administrators.

devereaux 11y ago

I have a mail server hosted at Linode on: - a clean IP, not on RBL, not blacklisted elsewhere to the best of my knowledge (outlook.com tells you when you IP is bad) - also accessible on IPv6 - both IPs having a proper rDNS on my domain - supporting SSL on port 487 and 565, with a certificate from a known authority - with DKIM and SPF both passing according to gmail Yet it ends up in gmail spam folder. And I'm only sending email to myself and 2 other persons, so it's not even mass mailing. I think there are other factors at play.

spotman 11y ago
Check PBL too
deftnerd 11y ago
That must be heart-breaking and very frustrating. You've done everything you can do to be a good internet citizen and support all the proper technology, but you're still treated badly. If I am correct, gmail now defaults to a "don't trust an IP by default" procedure. If enough people get your emails in a spam folder and mark it as not being spam, the system will start trusting the IP address. It also has to do with volume. This might be a good service offering. If someone could operate a few hundred gmail accounts, they could let people send them messages and mark those messages as not being spam in order to train the filters.
- thrownaway2424 11y ago
  Such accounts would be permanently disabled. Spammers have been trying to use sock-puppet networks for years.
deleted 11y ago
[deleted]
sn 11y ago
Google wants you to do special things for them. Have you gone to https://postmaster.google.com https://postmaster.google.com and set up your domain there?

soneil 11y ago

I'm not sure I'd put much stock in the linked mail-tester. It has a very incomplete SPF implementation that'll heavily penalise perfectly valid records. (eg, my record is simply "mx -all". So if a host is listed as an MX for my domain, it's also a valid sender. mail-tester doesn't appear to understand 'a' or 'mx' entries, so fails this entirely.)

kierkegaard9 11y ago

Avoiding spam filters is unfortunately not as easy as setting up rDNS, SPF, DKIM and DMARC. Reputation is also becoming a big issue: http://liminality.xyz/the-hostile-email-landscape/ http://liminality.xyz/the-hostile-email-landscape/

alexkavon 11y ago

Lots of people think it's a great idea to make packaged all-in-one mail systems, but what you're really doing is making it easier for spammers to get up and running.

xbeta 11y ago

Kinda off-topic, but I was wondering if anyone can recommend some setup for a CLI based mail client that works with IMAP and Gmail? I'm on a Mac now.

jlgaddis 11y ago
I used mutt on a Mac and was quite satisfied with it, although I'm kinda "GUI-averse" anyways. mutt itself is kinda slow if you actually use its internal IMAP and SMTP features over the Internet (such as with Gmail). Instead, I used offlineimap to pull down all of my mail to the Mac and pointed mutt at the (Maildir) directory where it was stored (which was incredibly fast, as you might guess). In addition, instead of having mutt send outbound mail directly to Gmail itself, I fed messages to msmtp (locally) and let that take care of sending them off to Gmail in the background. That setup isn't for everybody, of course, but I was very happy with it. In particular, {processing/catching up on} all of the mailing lists I subscribe to was much, much quicker.

joering2 11y ago

Slight OT/marketing, but I'm constantly having deliverability issues with Yahoo and spent weeks figuring out whats wrong, when spam-tester shows score 10/10, SenderScore 97, and yahoo keeps automatically marking my messages as spam, even when many times I contact client via phone and they swear they never clicked "mark as spam", which I have no reason no to believe. The spam complaint rate keeps being broken (around 0.3%) because of yahoo, and utilizing Sendgrid as an EPS, I'm afraid of losing ability to send. At this point I would love to hear an opinion of a e-mail deliveribility expert/veteran, or someone who can help me get off the cloud and host own email server that will be well-configured and maintained. I'm aware this service might come with a hefty $ bill. Please contact me via my email in my profile. </shameless plug for help>

mpnordland 11y ago

All this is moot if your server is in a blacklisted ip range. I found it out the hard way. Not just any VPS provider will do, make sure you get one from a company with a clean ip range.