8 ms·

I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender p

I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender program' and they still don't actually bother to contact postmaster@, but a few emails is usually enough to unblock the block within 24h. Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do. Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.

yomkippur 4y ago
was just gonna ask how you would handle DKIM and SPF stuff. Hetzner? Digitalocean?
- zahllos 4y ago
  I use a small datacentre in my country, actually not far from where I live. DKIM/SPF are independent of the provider. The easiest way to understand is to consider how receiving works. If I'm getting an email from hnemail.example, the first thing I do is consider the IP address. Oh, 257.257.257.1? Ok. So I then ask DNS "what is the SPF record for hnemail.example?" and it returns v=spf1 mx -all This tells me only to accept emails from 'MX' entries for that domain. So I query 'MX' against the DNS server and I get a list of A records, which I can get IPs from. If the IP is in the list, spf passes. Otherwise it fails, mark as spam. For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam. This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware. Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges. Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in. I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
  - aaronmdjones 4y ago
    DKIM selectors aren't UUIDs. You can of course use a UUID as a selector, but you don't have to. My selectors are named S-YYYYMM (when I rotate the keys), so my current public key is at S-202001._domainkey.example.com.
    - dspillett 4y ago
      A lot of tools generate UUIDs for the selector, just to get something unique without having to ask the user for something relevant or defining some other heuristic. For instance: the built-in helper tool for Zimbra generates a UUID by default, unless you provide something specific. I think a lot of people assume it should be a UUID just because they see UUIDs used in common examples. Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
- jhugo 4y ago
  Find a clueful small provider, local to you if possible. On huge providers like Hetzner and DO, you are guaranteed to have spammers as neighbours some of the time, even if the provider rapidly shuts them down. On the other hand, a good-quality small provider may rarely if ever host spammers.
  - Godel_unicode 4y ago
    Counterpoint, our mail admins spend a lot of time trying to convince small-scale providers to shutdown the spam email coming from them. Lots of people who host at small scale providers don’t care about patches, so they send tons of spam.
    - hkt 4y ago
      I suppose that's why the person you're replying to specified clueful!
    - jhugo 4y ago
      Doesn't seem like a counterpoint to me. The provider you're describing isn't clueful. A clueful provider pays attention to what's happening on their network and knows how to make themselves an unattractive host for spammers.
bornfreddy 4y ago
Agree completely. To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
- Avamander 4y ago
  EU's MECSA is nice as well https://mecsa.jrc.ec.europa.eu/en/ https://mecsa.jrc.ec.europa.eu/en/
- yonrg 4y ago
  I used this one: https://www.mail-tester.com/ https://www.mail-tester.com/ I will check out your recommendations. Thanks
flyinghamster 4y ago
I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs. Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
- DanAtC 4y ago
  At least you get notified. Microsoft/Outlook on the other hand silently drops emails leaving you and the recipient in the dark.
  - pas 4y ago
    that seems new, or maybe a different beast from the MS zoo of madness. gmail on the other hand does what others said, report smtp 250 and silently discard some emails. (mostly those that lack DKIM)
  - raxxorraxor 4y ago
    The correct response would be to set them on the blacklist for a time. People would complain and they would quickly change their behavior. Microsoft currently really wants everyone to have a Microsoft account.
PinguTS 4y ago
I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time. For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists. Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
- jimmydorry 4y ago
  If you are using external blocklists, you are perpetuating the problem here. Small mail operators end up on blocklists through no fault of their own, and it sounds like your server would reject them just like the big servers do.
  - EvanAnderson 4y ago
    You don’t have to use blocklists to block messages. I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
    - jimmydorry 4y ago
      That sounds like a better solution and not at all what I would assume someone means when they say that they use blocklists to filter out spam.
  - koheripbal 4y ago
    Not using block lists will inundate mail servers with spam. It's a necessary evil.
    - 4y ago
LMYahooTFY 4y ago
Would using a VPS with a static IP not work? Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?
- kodah 4y ago
  Yes, as long as the IP doesn't already have some reputation attached to it. Here's a list of resources: https://sendgrid.com/blog/5-ways-check-sending-reputation/ https://sendgrid.com/blog/5-ways-check-sending-reputation/
867-5309 4y ago
>VPS provider that disallows things like VPN endpoints this can't exactly be policed
- Avamander 4y ago
  It can be enforced when complaints come in.
raxxorraxor 4y ago
I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers. We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
bullen 4y ago
Everyone that can should host their own HTTP/1.1, DNS and SMTP at home on a battery backed Raspberry 1, 2 or Zero if they have an external IP. To host anything beyond those protocols and/or on more powerful hardware is often counter productive. The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term. Fight for external IP, ports and static IP in that order.

yomkippur 4y ago

was just gonna ask how you would handle DKIM and SPF stuff. Hetzner? Digitalocean?

zahllos 4y ago
I use a small datacentre in my country, actually not far from where I live. DKIM/SPF are independent of the provider. The easiest way to understand is to consider how receiving works. If I'm getting an email from hnemail.example, the first thing I do is consider the IP address. Oh, 257.257.257.1? Ok. So I then ask DNS "what is the SPF record for hnemail.example?" and it returns v=spf1 mx -all This tells me only to accept emails from 'MX' entries for that domain. So I query 'MX' against the DNS server and I get a list of A records, which I can get IPs from. If the IP is in the list, spf passes. Otherwise it fails, mark as spam. For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam. This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware. Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges. Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in. I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
- aaronmdjones 4y ago
  DKIM selectors aren't UUIDs. You can of course use a UUID as a selector, but you don't have to. My selectors are named S-YYYYMM (when I rotate the keys), so my current public key is at S-202001._domainkey.example.com.
  - dspillett 4y ago
    A lot of tools generate UUIDs for the selector, just to get something unique without having to ask the user for something relevant or defining some other heuristic. For instance: the built-in helper tool for Zimbra generates a UUID by default, unless you provide something specific. I think a lot of people assume it should be a UUID just because they see UUIDs used in common examples. Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
jhugo 4y ago
Find a clueful small provider, local to you if possible. On huge providers like Hetzner and DO, you are guaranteed to have spammers as neighbours some of the time, even if the provider rapidly shuts them down. On the other hand, a good-quality small provider may rarely if ever host spammers.
- Godel_unicode 4y ago
  Counterpoint, our mail admins spend a lot of time trying to convince small-scale providers to shutdown the spam email coming from them. Lots of people who host at small scale providers don’t care about patches, so they send tons of spam.
  - hkt 4y ago
    I suppose that's why the person you're replying to specified clueful!
  - jhugo 4y ago
    Doesn't seem like a counterpoint to me. The provider you're describing isn't clueful. A clueful provider pays attention to what's happening on their network and knows how to make themselves an unattractive host for spammers.

bornfreddy 4y ago

Agree completely. To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).

Avamander 4y ago
EU's MECSA is nice as well https://mecsa.jrc.ec.europa.eu/en/ https://mecsa.jrc.ec.europa.eu/en/
yonrg 4y ago
I used this one: https://www.mail-tester.com/ https://www.mail-tester.com/ I will check out your recommendations. Thanks

flyinghamster 4y ago

I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs. Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.

DanAtC 4y ago
At least you get notified. Microsoft/Outlook on the other hand silently drops emails leaving you and the recipient in the dark.
- pas 4y ago
  that seems new, or maybe a different beast from the MS zoo of madness. gmail on the other hand does what others said, report smtp 250 and silently discard some emails. (mostly those that lack DKIM)
- raxxorraxor 4y ago
  The correct response would be to set them on the blacklist for a time. People would complain and they would quickly change their behavior. Microsoft currently really wants everyone to have a Microsoft account.

PinguTS 4y ago

I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time. For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists. Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.

jimmydorry 4y ago
If you are using external blocklists, you are perpetuating the problem here. Small mail operators end up on blocklists through no fault of their own, and it sounds like your server would reject them just like the big servers do.
- EvanAnderson 4y ago
  You don’t have to use blocklists to block messages. I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
  - jimmydorry 4y ago
    That sounds like a better solution and not at all what I would assume someone means when they say that they use blocklists to filter out spam.
- koheripbal 4y ago
  Not using block lists will inundate mail servers with spam. It's a necessary evil.
  - 4y ago

LMYahooTFY 4y ago

Would using a VPS with a static IP not work? Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?

kodah 4y ago
Yes, as long as the IP doesn't already have some reputation attached to it. Here's a list of resources: https://sendgrid.com/blog/5-ways-check-sending-reputation/ https://sendgrid.com/blog/5-ways-check-sending-reputation/

867-5309 4y ago

>VPS provider that disallows things like VPN endpoints this can't exactly be policed

Avamander 4y ago
It can be enforced when complaints come in.

raxxorraxor 4y ago

I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers. We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.

bullen 4y ago

Everyone that can should host their own HTTP/1.1, DNS and SMTP at home on a battery backed Raspberry 1, 2 or Zero if they have an external IP. To host anything beyond those protocols and/or on more powerful hardware is often counter productive. The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term. Fight for external IP, ports and static IP in that order.