5 ms·

THIS. Spam is a solved problem, thanks to SPF and DKIM. But despite doing all the right things, Microsoft and Google continuously block and rate-limit deliver

THIS. Spam is a solved problem, thanks to SPF and DKIM. But despite doing all the right things, Microsoft and Google continuously block and rate-limit delivery. Case in point: we deliver 20,000 booking confirmation emails every day, all requested by users and not spam. We have perfect Postmaster Tools metrics: absolutely zero reported spam, 100% IP reputation, high domain reputation, zero feedback loop spam, 100% encryption. But Gmail still rate-limits us daily, so their customers get their booking confirmations several hours late. It's getting to the point where running an independent mail server is impossible. Yet another part of the open internet bites the dust.

brightball 4y ago
How many servers do you have? Just curious if the rate limit is per IP address?
- TedDoesntTalk 4y ago
  We have a Microsoft email subscription (all our mail is hosted with them), and their smtp server rate limits how much mail we send to our customers. We can’t even send 30 emails at once. We had to implement retry logic with back off.
  - exclusiv 4y ago
    Why not just use Sendgrid or similar for SMTP emails to customers, get a dedicated IP with them and have way less concern. I always prefer to isolate those things from the main corporate email and reputation anyway. You don't want your main domain getting hacked and affecting customer emails or vice versa.
  - ryanjshaw 4y ago
    Have you spoken to an account manager there? I send way more than that on a regular basis. I do it sequentially for simplicity, I wonder if that makes a difference.
  - KyeRussell 4y ago
    You should be using a dedicated transactional email provider, and accept the additional cost. At most scales, this is the correct business decision.
hanniabu 4y ago
Really hoping ENS can step in here and fill this gap. There's a lot of work being done on social and messaging infrastructure. Using that as an email and chat handle and social handle would be huge for maintaining open messaging. Unfortunately this relies on adoption and people living past their bias and admitting there's use cases.
- n3storm 4y ago
  What is ENS?
  - kristianp 4y ago
    ethereum mail service? https://www.skiff.com/blog/skiff-ens-ethereum-mail https://www.skiff.com/blog/skiff-ens-ethereum-mail
    - kristianp 4y ago
      Whoops, Ethereum Naming Service
- kkielhofner 4y ago
  I’ve been in the Ethereum ecosystem for over six years. I’ve heard “there’s a lot of work being done on…” literally thousands of times. Yet outside of the crypto-sphere I’ve only met a handful of people who’s life would be impacted in the slightest if the entire thing disappeared. Ethereum is eight years old and debates about use cases, etc aside I think even someone towards the middle of the debate can readily acknowledge progress, utility, value, and adoption has been incredibly slow. Consider ENS - the last time I registered with ENS the process was just as confusing and convoluted as ever, only in this instance with gas fees I paid around $300 for the name. Oh, and the various transactions kept failing - leaving me wondering (per usual with crypto) if my funds went up in smoke, off to some scammer, etc. Like so many times in crypto before I just assumed the funds were gone but like I said I DO follow this space and DO find it interesting so I consider burning crypto left and right a relatively cheap education/hobby. Plus I get to tell cool stories like this one on HN ;). I can’t imagine how creaky, bug ridden, impossible to scale, nearly impossible for mortals to use, and inevitably completely overrun by scammers and frauds an Ethereum (or any crypto) based mail and messaging system would be. No. Just no. With every day, week, month, and year that passes hearing “We’re building! People are working on XYZ!” when meanwhile the only thing people actually see is the latest headline for some crypto criminal, fraud, scam, collapse of last years “use case” (NFTs anyone?), etc. I’m not saying it’s dead but I think it’s pretty safe to say the window for anything approaching mass adoption, credibility, and real utility has closed. “Stop trying to make fetch happen! It’s not going to happen!”
wyclif 4y ago
I'm a former sysadmin and as such can remember the days when it was almost considered fun (at least by nerds) to run your own mail server. Those days are long gone, and the reason why is that monopolistic companies hate open protocols like email and want to 'monetize' everything they can.
franga2000 4y ago
How is spam related to SPF and DKIM? Those prevent forgery, but if a spammer actually owns a domain, they can send you whatever they want. That's where the majority of spam comes from, so it's far from a solved problem.
- Avamander 4y ago
  A lot of spammers forge, requiring SPF/DKIM does help a bit. But yeah, not a solution.
- kevincox 4y ago
  SPF and DKIM allow using domain reputation instead of IP reputation. Especially with IPv6 IPs are fairly easy to aquire and can be "stolen" either by spoofing/highjacking the network or more easily by compromising some hosts and sending from there. Hijacking a domain is generally harder as you need to compromise a mail server to get both the IP and DKIM key. And there are less mail servers than WordPress instances to hijack. Sure, you can buy domains but that leaves some paper trail and generally you will see added spam scores for new domains. So SPF and DKIM don't improve spam scores on their own, but the give a much more reliable path to building reputation.
mittermayr 4y ago
Spam is only solved by the large mail providers (who have enough data to act on it within a split second). After the great Gmail exodus two years ago (when they killed then unkilled legacy domain accounts), I moved to MXRoute — and the amount of unfiltered spam I get is insane. It's a daily nuisance. I have added about 200 filter words now, blocked hundreds of e-mail addresses, but it's next to impossible to filter out sometimes. From conversations and threads online by the MX Route admin, it seems like this is a nearly unsolvable problem, still. Spam evolves so fast and uses throwaway one-time approaches only, so once you've filtered for a pattern, the next campaign looks completely different. So, yeah, sorry about the long response, but I don't think Spam has been solved, at all, at least not if you're using an e-mail that's been around for longer than a few years (and appeared at least in one leak).
- fesc 4y ago
  I am using a small German email provider and I can't remember the last time I got spam. What's even crazier is that they are so confident in their spam filter that they simply reject mails classified as spam. But that hasn't been a problem since the few years I have been using them as well.
- account42 4y ago
  I have a wildcard email on my domain. In fact multiple domains. Multiple of these have emails listed in bug trackers, git repos and in plaintext on my website. The only filter I have is a couple of regular expressions to block the most lazy of spam at submission time. Even without that I got less than 50 spam mails a day, usually quite a bit less - and almost all of those are automatically sorted into my spam folder. How long does it take you to glance over a low two digit number of emails each day? Not a lot for me. I think the "spam" problem is overrated. Do you get more spam than me or do you just have different expectations?
KyeRussell 4y ago
Impersonation is “””solved””” by SPF and DKIM, not spam.
fuzzy2 4y ago
Most spam I receive (and that continues to somehow bypass my spam filters) originates from Gmail. So no, spam is not solved, not at all. Spam is only “solved” on big providers because they mostly accept mails only from other big providers. --- I would argue that “running an independent mail server” and mass-mailing are two entirely separate concerns.
- GTP 4y ago
  > Most spam I receive [...] originates from Gmail > Spam is only “solved” on big providers because they mostly accept mails only from other big providers. I would say that only accepting emails form other big providers isn't a good solution then.
- adr1an 4y ago
  Of course I can't prove this and is just from my subjectiviy and limited experience, yet... I've reasons to believe that Gmail users get more spam emails as they report these. It would be nice if such outsourcing went into other parts of the net (e.g. captcha).
nedt 4y ago
Not only are the blocking. They also send bounce spam although they know with SPF record, that the original mail was send by a server that's not allowed in that domain.
Avamander 4y ago
Spam is one of the most difficult content classification challenges for the past 20+ years. There are approaches that work well, on a small scale, but "solved" is a step too far.

brightball 4y ago

How many servers do you have? Just curious if the rate limit is per IP address?

TedDoesntTalk 4y ago
We have a Microsoft email subscription (all our mail is hosted with them), and their smtp server rate limits how much mail we send to our customers. We can’t even send 30 emails at once. We had to implement retry logic with back off.
- exclusiv 4y ago
  Why not just use Sendgrid or similar for SMTP emails to customers, get a dedicated IP with them and have way less concern. I always prefer to isolate those things from the main corporate email and reputation anyway. You don't want your main domain getting hacked and affecting customer emails or vice versa.
- ryanjshaw 4y ago
  Have you spoken to an account manager there? I send way more than that on a regular basis. I do it sequentially for simplicity, I wonder if that makes a difference.
- KyeRussell 4y ago
  You should be using a dedicated transactional email provider, and accept the additional cost. At most scales, this is the correct business decision.

hanniabu 4y ago

Really hoping ENS can step in here and fill this gap. There's a lot of work being done on social and messaging infrastructure. Using that as an email and chat handle and social handle would be huge for maintaining open messaging. Unfortunately this relies on adoption and people living past their bias and admitting there's use cases.

n3storm 4y ago
What is ENS?
- kristianp 4y ago
  ethereum mail service? https://www.skiff.com/blog/skiff-ens-ethereum-mail https://www.skiff.com/blog/skiff-ens-ethereum-mail
  - kristianp 4y ago
    Whoops, Ethereum Naming Service
kkielhofner 4y ago
I’ve been in the Ethereum ecosystem for over six years. I’ve heard “there’s a lot of work being done on…” literally thousands of times. Yet outside of the crypto-sphere I’ve only met a handful of people who’s life would be impacted in the slightest if the entire thing disappeared. Ethereum is eight years old and debates about use cases, etc aside I think even someone towards the middle of the debate can readily acknowledge progress, utility, value, and adoption has been incredibly slow. Consider ENS - the last time I registered with ENS the process was just as confusing and convoluted as ever, only in this instance with gas fees I paid around $300 for the name. Oh, and the various transactions kept failing - leaving me wondering (per usual with crypto) if my funds went up in smoke, off to some scammer, etc. Like so many times in crypto before I just assumed the funds were gone but like I said I DO follow this space and DO find it interesting so I consider burning crypto left and right a relatively cheap education/hobby. Plus I get to tell cool stories like this one on HN ;). I can’t imagine how creaky, bug ridden, impossible to scale, nearly impossible for mortals to use, and inevitably completely overrun by scammers and frauds an Ethereum (or any crypto) based mail and messaging system would be. No. Just no. With every day, week, month, and year that passes hearing “We’re building! People are working on XYZ!” when meanwhile the only thing people actually see is the latest headline for some crypto criminal, fraud, scam, collapse of last years “use case” (NFTs anyone?), etc. I’m not saying it’s dead but I think it’s pretty safe to say the window for anything approaching mass adoption, credibility, and real utility has closed. “Stop trying to make fetch happen! It’s not going to happen!”

wyclif 4y ago

I'm a former sysadmin and as such can remember the days when it was almost considered fun (at least by nerds) to run your own mail server. Those days are long gone, and the reason why is that monopolistic companies hate open protocols like email and want to 'monetize' everything they can.

franga2000 4y ago

How is spam related to SPF and DKIM? Those prevent forgery, but if a spammer actually owns a domain, they can send you whatever they want. That's where the majority of spam comes from, so it's far from a solved problem.

Avamander 4y ago
A lot of spammers forge, requiring SPF/DKIM does help a bit. But yeah, not a solution.
kevincox 4y ago
SPF and DKIM allow using domain reputation instead of IP reputation. Especially with IPv6 IPs are fairly easy to aquire and can be "stolen" either by spoofing/highjacking the network or more easily by compromising some hosts and sending from there. Hijacking a domain is generally harder as you need to compromise a mail server to get both the IP and DKIM key. And there are less mail servers than WordPress instances to hijack. Sure, you can buy domains but that leaves some paper trail and generally you will see added spam scores for new domains. So SPF and DKIM don't improve spam scores on their own, but the give a much more reliable path to building reputation.

mittermayr 4y ago

Spam is only solved by the large mail providers (who have enough data to act on it within a split second). After the great Gmail exodus two years ago (when they killed then unkilled legacy domain accounts), I moved to MXRoute — and the amount of unfiltered spam I get is insane. It's a daily nuisance. I have added about 200 filter words now, blocked hundreds of e-mail addresses, but it's next to impossible to filter out sometimes. From conversations and threads online by the MX Route admin, it seems like this is a nearly unsolvable problem, still. Spam evolves so fast and uses throwaway one-time approaches only, so once you've filtered for a pattern, the next campaign looks completely different. So, yeah, sorry about the long response, but I don't think Spam has been solved, at all, at least not if you're using an e-mail that's been around for longer than a few years (and appeared at least in one leak).

fesc 4y ago
I am using a small German email provider and I can't remember the last time I got spam. What's even crazier is that they are so confident in their spam filter that they simply reject mails classified as spam. But that hasn't been a problem since the few years I have been using them as well.
account42 4y ago
I have a wildcard email on my domain. In fact multiple domains. Multiple of these have emails listed in bug trackers, git repos and in plaintext on my website. The only filter I have is a couple of regular expressions to block the most lazy of spam at submission time. Even without that I got less than 50 spam mails a day, usually quite a bit less - and almost all of those are automatically sorted into my spam folder. How long does it take you to glance over a low two digit number of emails each day? Not a lot for me. I think the "spam" problem is overrated. Do you get more spam than me or do you just have different expectations?

KyeRussell 4y ago

Impersonation is “””solved””” by SPF and DKIM, not spam.

fuzzy2 4y ago

Most spam I receive (and that continues to somehow bypass my spam filters) originates from Gmail. So no, spam is not solved, not at all. Spam is only “solved” on big providers because they mostly accept mails only from other big providers. --- I would argue that “running an independent mail server” and mass-mailing are two entirely separate concerns.

GTP 4y ago
> Most spam I receive [...] originates from Gmail > Spam is only “solved” on big providers because they mostly accept mails only from other big providers. I would say that only accepting emails form other big providers isn't a good solution then.
adr1an 4y ago
Of course I can't prove this and is just from my subjectiviy and limited experience, yet... I've reasons to believe that Gmail users get more spam emails as they report these. It would be nice if such outsourcing went into other parts of the net (e.g. captcha).

nedt 4y ago

Not only are the blocking. They also send bounce spam although they know with SPF record, that the original mail was send by a server that's not allowed in that domain.

Avamander 4y ago

Spam is one of the most difficult content classification challenges for the past 20+ years. There are approaches that work well, on a small scale, but "solved" is a step too far.