9 ms·

I think the biggest pain in hosting your own mail server is getting your outbound mail delivered into the mailboxes of the large providers without being marked

I think the biggest pain in hosting your own mail server is getting your outbound mail delivered into the mailboxes of the large providers without being marked as spam. Especially if you don't actually send a lot of mail, so you can never really build up a good IP reputation. That's why I generally recommend a hybrid setup: Host inbound mail completely by yourself so that you have full control, but ship off outbound mail to a trusted relay of a privacy oriented provider. For example https://posteo.de/en https://posteo.de/en doesn't filter any outgoing messages by sender, so you can send mail from your own domains. Your local Postfix can DKIM sign your mail before sending them to Posteo and if you add include:posteo.de to your SPF record all your mail will be DKIM signed, SPF authenticated and coming from a reputable IP, so all your deliverability issues will be gone.

SahAssar 7y ago
This is also talked about in the article, is your experience to the contrary?
- akerl_ 7y ago
  The article handwaves this by pointing out that it’s “proof-of-work”. Which is a fancy way of saying “it requires a bunch of work, continuously as the rules change over time”. Which is why I’d consider it “hard” to run my own mail server. The complexity of running a daemon with a config file isn’t the issue, nor is it that any individual task of SPF/DKIM/etc is complex. It’s that I generally want to put 0% of my time into making sure my emails are being received, and I’m receiving other peoples’ emails.
  - brianpgordon 7y ago
    It would be nice if these big mail services accepted actual proof of work as a way to guarantee that your mail gets through. Maybe someone could extend SMTP with a bitcoin-like challenge where the server sends the client a nonce and a difficulty factor, and the client has to reply with a suffix that you can append to the nonce so that the whole string hashes to something with the corresponding number of zeroes at the end. People sending personal emails will be happy to burn a penny of CPU time as "postage" but spammers won't be able to send their spam profitably, especially if you increase the difficulty factor based on how many emails you've sent in the last hour.
    - esotericn 7y ago
      https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash > Hashcash was proposed in 1997 by Adam Back[1] and described more formally in Back's paper "Hashcash - A Denial of Service Counter-Measure".[2]
      - brianpgordon 7y ago
        I guess the idea was too obvious to be original. I'm surprised that it inspired Bitcoin instead of the other way around though.
    - deleted 7y ago
      [deleted]
    - jcranmer 7y ago
      > People sending personal emails will be happy to burn a penny of CPU time as "postage" but spammers won't be able to send their spam profitably The problem with this idea is that spammers don't use their own computer, they borrow other people's computers to send the spam. It also heavily penalizes people who legitimately need to send bulk email.
      - Nullabillity 7y ago
        > It also heavily penalizes people who legitimately need to send bulk email. Aka spammers.
        dasil003 7y ago
        Believe it or not there are email newsletters and mailing lists that people legitimately want to receive.
        jcranmer 7y ago
        Such as the Linux kernel mailing list.
  - jdboyd 7y ago
    In my experience of over 15 years, the "rules" have only changed 3 times since I started. First to require good reverse DNS, then to require SPF, last to require DKIM. The transition on each of those was generally several years with plenty of publicity.
    - bt848 7y ago
      That hardly covers it. The problem is that who "you" are, your identity as a sender, is not entirely within your control. The big mail operators figure in a ton of stuff, like the reputation of your subnet - /24 maybe, or /96, it's an approximation - or the reputation of your AS, your domain registrar, all kinds of stuff. If the guy on the next IP over is sending spam that's going to pollute your reputation. If you have your own ASN and your information is registered with reputable parties in lawful countries, you might not have much to worry about. If you are using a cheap cloud VPS service whose network is registered to a Romanian holding company, nobody will deliver your mail.
    - toomim 7y ago
      You're forgetting DMARC.
      - teddyh 7y ago
        If you already have SPF and/or DKIM, adding DMARC is a matter of setting one static DNS record and never changing it again. Hardly difficult.
      - scarejunba 7y ago
        I don’t have DMARC setup and can still send to Gmail.
  - amelius 7y ago
    > Which is a fancy way of saying “it requires a bunch of work, continuously as the rules change over time”. Wouldn't this be a good task for an open-source package to handle? If you'd just updated the package regularly, your mail would always be sent according to the rules. No need for any third-party handling your mail.
    - jasode 7y ago
      >Wouldn't this be a good task for an open-source package to handle? If you'd just updated the package regularly, your mail would always be sent according to the rules. Some of the "rules" for reliable outgoing email delivery cannot be encoded inside the source code files of a github repo, or email setup bash script, or a Docker image, or a EC2 virtual image, etc. An example of an unspecified "rule" outside the boundaries of a local machine holding the email server is recovering from an unexpected blacklisting of ip addresses. Example.[1] A preconfigured Docker image of Dovecot isn't going to magically ask Amazon support staff why the ip address is blacklisted and/or move the server to a different ISP etc. If a bad actor (that you are unaware of and have no control over) happens to share your ip address block and sends out spam which then causes Gmail/Hotmail/Yahoo to reject your server's emails, there's no open-source programming code that can detect and fix those problems happening outside of your control. Or put another way, if HonestJoeBlow could download a constantly updated Docker image that has the so-called "correct rules" for sent emails to always be accepted by Gmail, it would mean the DishonestSpammers could also download that same Docker image to get their emails delivered. The issue is that "trust" in the email ecosystem is an emergent property among participants and therefore, it can't all be embedded inside email server configs, or in a DIY blog article trying to explain the exact 12 steps to make email delivery work perfectly with all receivers. Eventually, something can break (because a receiver changes their idea of "trust" and "spam") and it requires troubleshooting & debugging to get email delivery working again. A tldr would be: You can't put "sender reputation" into a downloadable software package. [1] https://forums.aws.amazon.com/message.jspa?messageID=724690 https://forums.aws.amazon.com/message.jspa?messageID=724690
      - temptemptemp111 7y ago
        However, would be sweet for a bunch of mailserver experts to converge on a docker-compose or swarm file for an optimal setup.
dwd 7y ago
I would also recommend setting up a dmarc record so you can monitor your deliverability. Spam however is still a hassle and you're never going to match Gmail or Outlook when you're small. Say it takes a few hundred identical spammy emails to trigger a filter. For Gmail it means the chance of receiving that spam is less than 1 in a million. If you're small and managing say a 1000 users - a large percentage will end up with receiving the spam.
- dorgo 7y ago
  >Say it takes a few hundred identical spammy emails to trigger a filter Wait, is this really the way how spam filters work? All I need to do is to add a random number to all emails to bypass spam filters?
  - mr__y 7y ago
    In essence this is one of the principles, the thing is the emails don't need to be perfectly identical, just identical-ish with the identicalish matching implementation being not public (as this would immediately lead to spammers adding just enough randomness to whatever they are sending)
  - al_form2000 7y ago
    In short, no.
- baobabKoodaa 7y ago
  You can't actually measure deliverability with dmarc reports. They will only tell you the portion of emails that are bounced, but they don't differentiate between email in inbox and email in spam folder.
  - dwd 7y ago
    Wrong word - delivery rather than deliverability. You just want to know about the SPF and DKIM fails. Last thing you need someone on a self-hosted email server is running EDM campaigns.
  - 7y ago
ogoffart 7y ago
> I think the biggest pain in hosting your own mail server is getting your outbound mail delivered into the mailboxes of the large providers without being marked as spam Why do you think that? The article mention exactly this could just be a myth that perpetuates because people repeat it without actually trying it. In fact, I used to think exactly like you when I originally set up my personal mail server and opted for the hybrid setup you suggest. However, when the third party service I used to send mail shut down a couple of years ago, I quiclky set up outbound mail on my own server. None of my emails are considered as spam, and I haven't touched the mail configuration since.
- defanor 7y ago
  Not GP, but it matches my experience: with reverse DNS, DKIM, SPF, being in DNSWL (and not in DNSBLs) some of my messages end up in Gmail spam folders (very recently received a reply to a message I sent 3 years ago because of that, for instance). A while ago there were strange issues with Google-hosted mailing lists as well, with some messages not being visible even to an ML administrator, while Gmail servers accepted mail just fine (maybe it still happens, I'm just not writing to those). I hear similar stories from other mail server administrators too. I think it's more of a Gmail/BigMailerCorp issue, and still using a private mail server myself, but I don't think it's fair to say that it's all easy and works flawlessly, even with strangely behaving mail servers on the other end.
  - pw6hv 7y ago
    > some of my messages end up in Gmail spam folders (very recently received a reply to a message I sent 3 years ago because of that, for instance) How did this happen if Gmail deletes spam after some weeks?
    - defanor 7y ago
      Indeed, a quick search suggests that spam is automatically removed every 30 days there. The wording was "Google hid it from me", so either I have wrongly assumed that it's about the spam folder, or it is possible to disable autoremoval. Edit: or, as the sibling comment suggests, it behaves strangely with respect to removal as well.
anteatersa 7y ago
This is true, I run a small mailserver for myself, friends and family. I eventually gave up trying to get mail delivered to the major mail services and now just send postfix mail through Amazon Simple Email Service.

SahAssar 7y ago

This is also talked about in the article, is your experience to the contrary?

akerl_ 7y ago
The article handwaves this by pointing out that it’s “proof-of-work”. Which is a fancy way of saying “it requires a bunch of work, continuously as the rules change over time”. Which is why I’d consider it “hard” to run my own mail server. The complexity of running a daemon with a config file isn’t the issue, nor is it that any individual task of SPF/DKIM/etc is complex. It’s that I generally want to put 0% of my time into making sure my emails are being received, and I’m receiving other peoples’ emails.
- brianpgordon 7y ago
  It would be nice if these big mail services accepted actual proof of work as a way to guarantee that your mail gets through. Maybe someone could extend SMTP with a bitcoin-like challenge where the server sends the client a nonce and a difficulty factor, and the client has to reply with a suffix that you can append to the nonce so that the whole string hashes to something with the corresponding number of zeroes at the end. People sending personal emails will be happy to burn a penny of CPU time as "postage" but spammers won't be able to send their spam profitably, especially if you increase the difficulty factor based on how many emails you've sent in the last hour.
  - esotericn 7y ago
    https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash > Hashcash was proposed in 1997 by Adam Back[1] and described more formally in Back's paper "Hashcash - A Denial of Service Counter-Measure".[2]
    - brianpgordon 7y ago
      I guess the idea was too obvious to be original. I'm surprised that it inspired Bitcoin instead of the other way around though.
  - deleted 7y ago
    [deleted]
  - jcranmer 7y ago
    > People sending personal emails will be happy to burn a penny of CPU time as "postage" but spammers won't be able to send their spam profitably The problem with this idea is that spammers don't use their own computer, they borrow other people's computers to send the spam. It also heavily penalizes people who legitimately need to send bulk email.
    - Nullabillity 7y ago
      > It also heavily penalizes people who legitimately need to send bulk email. Aka spammers.
      - dasil003 7y ago
        Believe it or not there are email newsletters and mailing lists that people legitimately want to receive.
        jcranmer 7y ago
        Such as the Linux kernel mailing list.
- jdboyd 7y ago
  In my experience of over 15 years, the "rules" have only changed 3 times since I started. First to require good reverse DNS, then to require SPF, last to require DKIM. The transition on each of those was generally several years with plenty of publicity.
  - bt848 7y ago
    That hardly covers it. The problem is that who "you" are, your identity as a sender, is not entirely within your control. The big mail operators figure in a ton of stuff, like the reputation of your subnet - /24 maybe, or /96, it's an approximation - or the reputation of your AS, your domain registrar, all kinds of stuff. If the guy on the next IP over is sending spam that's going to pollute your reputation. If you have your own ASN and your information is registered with reputable parties in lawful countries, you might not have much to worry about. If you are using a cheap cloud VPS service whose network is registered to a Romanian holding company, nobody will deliver your mail.
  - toomim 7y ago
    You're forgetting DMARC.
    - teddyh 7y ago
      If you already have SPF and/or DKIM, adding DMARC is a matter of setting one static DNS record and never changing it again. Hardly difficult.
    - scarejunba 7y ago
      I don’t have DMARC setup and can still send to Gmail.
- amelius 7y ago
  > Which is a fancy way of saying “it requires a bunch of work, continuously as the rules change over time”. Wouldn't this be a good task for an open-source package to handle? If you'd just updated the package regularly, your mail would always be sent according to the rules. No need for any third-party handling your mail.
  - jasode 7y ago
    >Wouldn't this be a good task for an open-source package to handle? If you'd just updated the package regularly, your mail would always be sent according to the rules. Some of the "rules" for reliable outgoing email delivery cannot be encoded inside the source code files of a github repo, or email setup bash script, or a Docker image, or a EC2 virtual image, etc. An example of an unspecified "rule" outside the boundaries of a local machine holding the email server is recovering from an unexpected blacklisting of ip addresses. Example.[1] A preconfigured Docker image of Dovecot isn't going to magically ask Amazon support staff why the ip address is blacklisted and/or move the server to a different ISP etc. If a bad actor (that you are unaware of and have no control over) happens to share your ip address block and sends out spam which then causes Gmail/Hotmail/Yahoo to reject your server's emails, there's no open-source programming code that can detect and fix those problems happening outside of your control. Or put another way, if HonestJoeBlow could download a constantly updated Docker image that has the so-called "correct rules" for sent emails to always be accepted by Gmail, it would mean the DishonestSpammers could also download that same Docker image to get their emails delivered. The issue is that "trust" in the email ecosystem is an emergent property among participants and therefore, it can't all be embedded inside email server configs, or in a DIY blog article trying to explain the exact 12 steps to make email delivery work perfectly with all receivers. Eventually, something can break (because a receiver changes their idea of "trust" and "spam") and it requires troubleshooting & debugging to get email delivery working again. A tldr would be: You can't put "sender reputation" into a downloadable software package. [1] https://forums.aws.amazon.com/message.jspa?messageID=724690 https://forums.aws.amazon.com/message.jspa?messageID=724690
    - temptemptemp111 7y ago
      However, would be sweet for a bunch of mailserver experts to converge on a docker-compose or swarm file for an optimal setup.

dwd 7y ago

I would also recommend setting up a dmarc record so you can monitor your deliverability. Spam however is still a hassle and you're never going to match Gmail or Outlook when you're small. Say it takes a few hundred identical spammy emails to trigger a filter. For Gmail it means the chance of receiving that spam is less than 1 in a million. If you're small and managing say a 1000 users - a large percentage will end up with receiving the spam.

dorgo 7y ago
>Say it takes a few hundred identical spammy emails to trigger a filter Wait, is this really the way how spam filters work? All I need to do is to add a random number to all emails to bypass spam filters?
- mr__y 7y ago
  In essence this is one of the principles, the thing is the emails don't need to be perfectly identical, just identical-ish with the identicalish matching implementation being not public (as this would immediately lead to spammers adding just enough randomness to whatever they are sending)
- al_form2000 7y ago
  In short, no.
baobabKoodaa 7y ago
You can't actually measure deliverability with dmarc reports. They will only tell you the portion of emails that are bounced, but they don't differentiate between email in inbox and email in spam folder.
- dwd 7y ago
  Wrong word - delivery rather than deliverability. You just want to know about the SPF and DKIM fails. Last thing you need someone on a self-hosted email server is running EDM campaigns.
- 7y ago

ogoffart 7y ago

> I think the biggest pain in hosting your own mail server is getting your outbound mail delivered into the mailboxes of the large providers without being marked as spam Why do you think that? The article mention exactly this could just be a myth that perpetuates because people repeat it without actually trying it. In fact, I used to think exactly like you when I originally set up my personal mail server and opted for the hybrid setup you suggest. However, when the third party service I used to send mail shut down a couple of years ago, I quiclky set up outbound mail on my own server. None of my emails are considered as spam, and I haven't touched the mail configuration since.

defanor 7y ago
Not GP, but it matches my experience: with reverse DNS, DKIM, SPF, being in DNSWL (and not in DNSBLs) some of my messages end up in Gmail spam folders (very recently received a reply to a message I sent 3 years ago because of that, for instance). A while ago there were strange issues with Google-hosted mailing lists as well, with some messages not being visible even to an ML administrator, while Gmail servers accepted mail just fine (maybe it still happens, I'm just not writing to those). I hear similar stories from other mail server administrators too. I think it's more of a Gmail/BigMailerCorp issue, and still using a private mail server myself, but I don't think it's fair to say that it's all easy and works flawlessly, even with strangely behaving mail servers on the other end.
- pw6hv 7y ago
  > some of my messages end up in Gmail spam folders (very recently received a reply to a message I sent 3 years ago because of that, for instance) How did this happen if Gmail deletes spam after some weeks?
  - defanor 7y ago
    Indeed, a quick search suggests that spam is automatically removed every 30 days there. The wording was "Google hid it from me", so either I have wrongly assumed that it's about the spam folder, or it is possible to disable autoremoval. Edit: or, as the sibling comment suggests, it behaves strangely with respect to removal as well.

anteatersa 7y ago

This is true, I run a small mailserver for myself, friends and family. I eventually gave up trying to get mail delivered to the major mail services and now just send postfix mail through Amazon Simple Email Service.