7 ms·

So under the guise of spam protection, we have created an oligopoly of email gateways, despite sending an email being an utterly trivial, long solved problem,

So under the guise of spam protection, we have created an oligopoly of email gateways, despite sending an email being an utterly trivial, long solved problem, and now those companies are putting the squeeze on. The best part about this particular item on the list of "things that have only gotten worse over time": they charge companies more to deliver what is essentially spam, and they conspired to make that spam show up in inboxes and "enhance" it with all sorts of tracking.

Eikon 7y ago
> despite sending an email being an utterly trivial, long solved problem. It is absolutely not. Even if you are not considering the whole spam thing. Sending an email with the multitude of clients, MIME which is very complex in itself, network issues, message queuing, retries, unsubscription, bounces, providers feedback loops, rate limiting and what not is very hard. If you add the whole spam folder + blacklists situation into consideration, sending emails becomes a more than painful thing you don't want to deal with.
- kick 7y ago
  Sending an email is absolutely a solved problem. I can send a message to my address in 2020 using a 2003 Fedora Core 1 box without a domain using only what's part of the POSIX standard, and I'll receive it just fine. I just did a few days ago, actually. It even got past my provider's spam filter. I could probably have done it using earlier software than 2003-era mail/mailx, but that was the quickest way I could find to send an attachment given the software on hand.
  - ebiester 7y ago
    You can still do that. However, I, as a receiver of email, get thousands of spam messages a day. So, I use services that block them before they even reach my view, or get sent directly to the spam folder. If you want to send me an email, that's solved. If you want to send it to ten thousand people on your mailing list, and not have it end up in the spam folder, you may have more work.
    - bigtunacan 7y ago
      Please explain the difference to me. Mailgun is sending tons and tons of spam email; what is so special about them that they aren't ending up in people's spam/junk folders?
      - slig 7y ago
        They probably send much, much more valid emails, respond quickly to abuse reports, and since they're large/known enough, they won't get blacklisted because that would be disruptive to large, legit business.
      - aseipp 7y ago
        Mail providers like Mailgun live or die based on their reputation. If they as a sender get marked as spam by "The Algorithms", then their product is useless. They almost certainly respond to abuse reports quickly and have enough reputation, and enough legitimate clients, that they have "leeway". But that leeway isn't unlimited. There was a big fit thrown on this website a year or two ago when one transactional mail provider (MailChimp?) said they were banning crypto companies with ICOs from their service. They literally put "ICOs are not welcome" in their Terms of Service. Why? Because first off, ICOs are scams. And that means they buy email lists, and then spam the shit out of people with "buy into our ICO!" emails, which get immediately shitcanned into the spam folder by 99.99% of users -- they're no different than Nigerian Princes, as far as most people care. That behavior tanks the reputation of the sender, and you cannot reason with The Algorithm once it "recognizes" you as spam. That kind of problem is not a minor inconvenience to companies like Mailgun, it's an existential threat.
        lolc 7y ago
        They really care with their reputation on the line. Here for example is their response to the only complaint I've had to send Mailgun: "Thanks for sending this over. I was able to tie this sender to a compromised account, which has been disabled in our system to prevent further messages." Pity we'll have to move off them. It was good while it lasted.
    - 7y ago
  - space_ghost 7y ago
    You're both right, but you're solving different problems. Just because your message got past your spam filter doesn't mean it would get past all spam filters.
    - kick 7y ago
      It is absolutely not. Even if you are not considering the whole spam thing. They specified that it wasn't even if you could ignore the spam issue. I said that it was if you ignored the spam issue. The first post in this thread was "We've made sending electronic mail way too complicated under the guise of fighting spam." I'm aware that it won't get past every spam filter, but it gets past both Gmail's and my own provider (a smaller, non-American one), which is good enough for my own use. My claim was that sending electronic mail is a solved problem, because it is.
      - bigiain 7y ago
        > My claim was that sending electronic mail is a solved problem, because it is. Agreed. I mean, I can _probably_ remember enough to be able to send mail using just telnet without even needing to look anything up. (And that mail, without any mime parts or urls - it very very likely to not fall into any of the spam filtering on my inbound email accounts...)
        greglindahl 7y ago
        In the modern era, lots of mailservers run software that looks for weirdnesses in incoming email. I don't know, but I'd suspect that the typing delays of using telnet to send mail are likely to trigger their defenses.
        bigiain 7y ago
        SO I just tested. After working out that on macOS I now need to use nc -c instead of telnet... I _can_ send mail manually, but it _does_ end up in gmails's spam folder saying "Why is this message in spam? It is similar to messages that were identified as spam in the past." Iain-2:~ iain$ nc -c gmail-smtp-in.l.google.com 25 220 mx.google.com ESMTP k5si5682618pls.209 - gsmtp EHLO elided.org 250-mx.google.com at your service, [202.171.181.100] 250-SIZE 157286400 250-8BITMIME 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-CHUNKING 250 SMTPUTF8 MAIL FROM: <bigiain@elided.org> 250 2.1.0 OK k5si5682618pls.209 - gsmtp RCPT TO: <bigiain@gmail.com> 250 2.1.5 OK k5si5682618pls.209 - gsmtp DATA 354 Go ahead k5si5682618pls.209 - gsmtp Subject: Test From: bigiain@elided.org To: bigiain@gmail.com Date: Fri, 31 Jan 2020 13:07:00 +1100 Foo baz nah . 250 2.0.0 OK 1580436443 k5si5682618pls.209 - gsmtp QUIT 221 2.0.0 closing connection k5si5682618pls.209 - gsmtp Iain-2:~ iain$
  - cosmotic 7y ago
    Most ISPs block SMTP traffic, which creates a pretty significant barrier to sending email. Even VPS services send through a secret gateways with all sorts of filters and controls on it.
  - gwittel 7y ago
    Sending at scale is the problem. A provider like Mailgun (aka an ESP), is a huge target for abuse. Free and low priced tiers are tricky since they need to draw in customers, but then get heavily abused. tl;dr: Abuse ruins it for everyone. I work on the filtering side of the world so I see a lot of the challenges that even purportedly good actors face. Each provider does better or worse at controlling abuse. These things all have a cost: * IPs -- You need to send from a variety of IPs. They have warmup time before they can be safely used. One bad guy can burn that IP and everyone who shares. Most ESPs aren't going to have dedicated IPs for each (or those are for high tiers). * Dmarc/dkim/spf -- Passing validation can be hard since it requires your client to work with you. * Abuse reports -- Likely huge volume * Bounces -- They can count against the customer. How much is OK (e.g. AWS SES will cut you off if your bounce rate gets too high)? * Throwaway account abuse -- Huge issue, this threshold is a simple hammer (so now bad guys need ~10x more throwaway accounts). * Account takeover -- That good guy is now sending out crap. Now what? * Post-abuse cleanup -- Good luck going around and working with major providers to get yourself unblocked. Its a huge time sink. By using Mailgun, etc all of the above becomes their problem. On top of it they'll offer analytics, help crafting content that works in varying mail clients, etc. A small business type sender doesn't need much of this (other than maybe the technical help). But at some point, the scales tip.
    - webbie917 7y ago
      ^ Couldn't agree more, especially "Abuse ruins it for everyone."
    - quicksilver03 7y ago
      > IPs -- You need to send from a variety of IPs. They have warmup time before they can be safely used. One bad guy can burn that IP and everyone who shares. Most ESPs aren't going to have dedicated IPs for each (or those are for high tiers). I'm curious, is that specific to IPv4 or you see the same phenomenon with IPv6 where providers allocate /64 or /48 to customers? Or there simply isn't enough sending volume from IPv6 at the moment to warrant specific filtering?
      - gwittel 7y ago
- NightlyDev 7y ago
  I've sent a lot of mail in the last 14 years, and it's not hard. I spammed with invite mails to users friends back in 2008 and mail started to end up in spam for a while, but after switching to double opt-in for all mail and making unsubscribing easily available it went back to good. Add some periodic list cleaning of long time inactive subscribers and it works perfectly! It does get harder if you have to account for bad users, but if you're the only user then you can make sure everything is good. So, is it hard? That depends entirely on how wanted or unwanted the sent mail is.
xwowsersx 7y ago
It is clearly not trivial though. I get that it feels like it should be and probably is if you're sending a few emails, but "at scale" (pardon the buzzword) it is not trivial hence why people use these services.
ruffrey 7y ago
I've been running a SaaS in the email space since 2011. It is not surprising at all, to me, that Mailgun decided to do this. Sending an email _is_ trivial - you're right - but getting it delivered to the inbox, instead of the spam folder, is far from trivial. At scale, it is incredibly difficult. Even worse, spammers and fraudsters are utterly relentless. They hammer services like mine and Mailgun's constantly with fraud and phishing and porn and other garbage, inbound and outbound.
omarhaneef 7y ago
Yes, sending email is indeed a solved issue, but that isn't what mailgun (re?) solves. Asking someone's permissions first, tracking their desires and immediately updating your response is what they solve. Seems trivial? Try doing it for (looks at note) 626 people.
- hnick 7y ago
  I think their point was that it's been drummed into our heads that ISPs and mail services only accept email from 'trusted' IP blocks, so you can't do it yourself, better use 'the experts'.
nemothekid 7y ago
>despite sending an email being an utterly trivial, long solved problem AFAIK, SPAM is a still a hard problem, and getting emails delivered isn't trivial. "Under the guise of Spam" is a very reductionist viewpoint. It's like saying "Under the guise of the measles, we now have an oligopoly of pharmaceuticals." The latter might be true, but its important not to understate the threat of the former.
luckylion 7y ago
No, we've created something far better. ~50% of the Spam that makes it to my inbox is sent via Amazon SES. Amazon does not give a fuck, I've reported it to abuse, they ignore it. I can't just block Amazon SES though, because there's also a ton of legit services that use them for mail transport. We've really helped spammers evade blocking and decrease their infrastructure costs, and because there's so much "normal" traffic on SES, spammers will blend right in. And as long as they pay for services rendered, Amazon couldn't be happier.
ljm 7y ago
These providers are just as responsible for spam by running opt-out mailing lists and, even worse... spamming you with surveys and confirmations to see why you unsubscribed. With tracking pixels and rewritten URLs that go through their system before you get to the real destination. You don’t get a choice. That was made for you.
ttul 7y ago
If it's an oligopoly and presumably throwing off cash, what's stopping anyone from creating their own transactional email service? Answer: It's very difficult. Email is adversarial. Getting to the inbox is incredibly difficult to do _at scale_.
- NightlyDev 7y ago
  A private service where you are very sure that subscribers wants and opens the mail? Easy. A public service where a lot will be spam and unsolicited emails? Must be hell to run.
bdcravens 7y ago
> despite sending an email being an utterly trivial, long solved problem Then why do companies need transactional email providers?
mike-cardwell 7y ago
We have our own domain in G-Suite for our mail. I just checked my GMail Spam box. In it are 6 emails from this week. All 6 of them were sent through Mailgun, from and to the same domain (the one in G Suite). One was a cron email from one of our servers and the other 5 are transactional emails about people signing up to our service. All of them passed both SPF and DKIM. How. Just how did these emails get spam filtered by Google? Everything about them screams that they're legit emails from ourselves to ourselves.
- Keverw 7y ago
  Wow, sounds like even if you pay for a company to do then still might not go to inbox 100%... I noticed marking things not as spam still sends things to spam, I figured be doing that would tell it it's not spam... I was recently thinking about the email problem myself, and was reading about IP Warmups. Seems like every provider has different recommendations too. So sounds like not an exact science either. But maybe if it was an exact science it'd be hard to tell the good and bad guys apart. I guess one of the tricky problems with decentralized systems.
pyuser583 7y ago
You want to try an impressive open source software project? Create a framework that takes an email forwarded to a given address, turns it into an HTTP request with a JSON body, and sends that HTTP request to webhook. It’s easy to make a proof of concept. Production grade is an entirely different story. Remember to preserve conversations and forwarding. Make sure file attachments are handled and scanned for viruses. Make sure you watch out for recursive automatic forwarding. Make sure that any html is secure. And make sure it will scale well ... and I mean billions of emails a day well.
- cx42net 7y ago
  You are 100% right. It started the same with our service; we bought back ImprovMX (I don't put the link as I'm not looking for publicity - just for reference) when it was just a 5$ droplet at Digital Ocean, and now we have 10 big servers, more than 700$ monthly cost (and I honestly think it's not that much) and have to handle all the messy parts of delivering an email. It's easy to send an email, but it's way harder to ensure that the emails you send are delivered properly.

Eikon 7y ago

> despite sending an email being an utterly trivial, long solved problem. It is absolutely not. Even if you are not considering the whole spam thing. Sending an email with the multitude of clients, MIME which is very complex in itself, network issues, message queuing, retries, unsubscription, bounces, providers feedback loops, rate limiting and what not is very hard. If you add the whole spam folder + blacklists situation into consideration, sending emails becomes a more than painful thing you don't want to deal with.

kick 7y ago
Sending an email is absolutely a solved problem. I can send a message to my address in 2020 using a 2003 Fedora Core 1 box without a domain using only what's part of the POSIX standard, and I'll receive it just fine. I just did a few days ago, actually. It even got past my provider's spam filter. I could probably have done it using earlier software than 2003-era mail/mailx, but that was the quickest way I could find to send an attachment given the software on hand.
- ebiester 7y ago
  You can still do that. However, I, as a receiver of email, get thousands of spam messages a day. So, I use services that block them before they even reach my view, or get sent directly to the spam folder. If you want to send me an email, that's solved. If you want to send it to ten thousand people on your mailing list, and not have it end up in the spam folder, you may have more work.
  - bigtunacan 7y ago
    Please explain the difference to me. Mailgun is sending tons and tons of spam email; what is so special about them that they aren't ending up in people's spam/junk folders?
    - slig 7y ago
      They probably send much, much more valid emails, respond quickly to abuse reports, and since they're large/known enough, they won't get blacklisted because that would be disruptive to large, legit business.
    - aseipp 7y ago
      Mail providers like Mailgun live or die based on their reputation. If they as a sender get marked as spam by "The Algorithms", then their product is useless. They almost certainly respond to abuse reports quickly and have enough reputation, and enough legitimate clients, that they have "leeway". But that leeway isn't unlimited. There was a big fit thrown on this website a year or two ago when one transactional mail provider (MailChimp?) said they were banning crypto companies with ICOs from their service. They literally put "ICOs are not welcome" in their Terms of Service. Why? Because first off, ICOs are scams. And that means they buy email lists, and then spam the shit out of people with "buy into our ICO!" emails, which get immediately shitcanned into the spam folder by 99.99% of users -- they're no different than Nigerian Princes, as far as most people care. That behavior tanks the reputation of the sender, and you cannot reason with The Algorithm once it "recognizes" you as spam. That kind of problem is not a minor inconvenience to companies like Mailgun, it's an existential threat.
      - lolc 7y ago
        They really care with their reputation on the line. Here for example is their response to the only complaint I've had to send Mailgun: "Thanks for sending this over. I was able to tie this sender to a compromised account, which has been disabled in our system to prevent further messages." Pity we'll have to move off them. It was good while it lasted.
  - 7y ago
- space_ghost 7y ago
  You're both right, but you're solving different problems. Just because your message got past your spam filter doesn't mean it would get past all spam filters.
  - kick 7y ago
    It is absolutely not. Even if you are not considering the whole spam thing. They specified that it wasn't even if you could ignore the spam issue. I said that it was if you ignored the spam issue. The first post in this thread was "We've made sending electronic mail way too complicated under the guise of fighting spam." I'm aware that it won't get past every spam filter, but it gets past both Gmail's and my own provider (a smaller, non-American one), which is good enough for my own use. My claim was that sending electronic mail is a solved problem, because it is.
    - bigiain 7y ago
      > My claim was that sending electronic mail is a solved problem, because it is. Agreed. I mean, I can _probably_ remember enough to be able to send mail using just telnet without even needing to look anything up. (And that mail, without any mime parts or urls - it very very likely to not fall into any of the spam filtering on my inbound email accounts...)
      - greglindahl 7y ago
        In the modern era, lots of mailservers run software that looks for weirdnesses in incoming email. I don't know, but I'd suspect that the typing delays of using telnet to send mail are likely to trigger their defenses.
        bigiain 7y ago
        SO I just tested. After working out that on macOS I now need to use nc -c instead of telnet... I _can_ send mail manually, but it _does_ end up in gmails's spam folder saying "Why is this message in spam? It is similar to messages that were identified as spam in the past." Iain-2:~ iain$ nc -c gmail-smtp-in.l.google.com 25 220 mx.google.com ESMTP k5si5682618pls.209 - gsmtp EHLO elided.org 250-mx.google.com at your service, [202.171.181.100] 250-SIZE 157286400 250-8BITMIME 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-CHUNKING 250 SMTPUTF8 MAIL FROM: <bigiain@elided.org> 250 2.1.0 OK k5si5682618pls.209 - gsmtp RCPT TO: <bigiain@gmail.com> 250 2.1.5 OK k5si5682618pls.209 - gsmtp DATA 354 Go ahead k5si5682618pls.209 - gsmtp Subject: Test From: bigiain@elided.org To: bigiain@gmail.com Date: Fri, 31 Jan 2020 13:07:00 +1100 Foo baz nah . 250 2.0.0 OK 1580436443 k5si5682618pls.209 - gsmtp QUIT 221 2.0.0 closing connection k5si5682618pls.209 - gsmtp Iain-2:~ iain$
- cosmotic 7y ago
  Most ISPs block SMTP traffic, which creates a pretty significant barrier to sending email. Even VPS services send through a secret gateways with all sorts of filters and controls on it.
- gwittel 7y ago
  Sending at scale is the problem. A provider like Mailgun (aka an ESP), is a huge target for abuse. Free and low priced tiers are tricky since they need to draw in customers, but then get heavily abused. tl;dr: Abuse ruins it for everyone. I work on the filtering side of the world so I see a lot of the challenges that even purportedly good actors face. Each provider does better or worse at controlling abuse. These things all have a cost: * IPs -- You need to send from a variety of IPs. They have warmup time before they can be safely used. One bad guy can burn that IP and everyone who shares. Most ESPs aren't going to have dedicated IPs for each (or those are for high tiers). * Dmarc/dkim/spf -- Passing validation can be hard since it requires your client to work with you. * Abuse reports -- Likely huge volume * Bounces -- They can count against the customer. How much is OK (e.g. AWS SES will cut you off if your bounce rate gets too high)? * Throwaway account abuse -- Huge issue, this threshold is a simple hammer (so now bad guys need ~10x more throwaway accounts). * Account takeover -- That good guy is now sending out crap. Now what? * Post-abuse cleanup -- Good luck going around and working with major providers to get yourself unblocked. Its a huge time sink. By using Mailgun, etc all of the above becomes their problem. On top of it they'll offer analytics, help crafting content that works in varying mail clients, etc. A small business type sender doesn't need much of this (other than maybe the technical help). But at some point, the scales tip.
  - webbie917 7y ago
    ^ Couldn't agree more, especially "Abuse ruins it for everyone."
  - quicksilver03 7y ago
    > IPs -- You need to send from a variety of IPs. They have warmup time before they can be safely used. One bad guy can burn that IP and everyone who shares. Most ESPs aren't going to have dedicated IPs for each (or those are for high tiers). I'm curious, is that specific to IPv4 or you see the same phenomenon with IPv6 where providers allocate /64 or /48 to customers? Or there simply isn't enough sending volume from IPv6 at the moment to warrant specific filtering?
    - gwittel 7y ago
NightlyDev 7y ago
I've sent a lot of mail in the last 14 years, and it's not hard. I spammed with invite mails to users friends back in 2008 and mail started to end up in spam for a while, but after switching to double opt-in for all mail and making unsubscribing easily available it went back to good. Add some periodic list cleaning of long time inactive subscribers and it works perfectly! It does get harder if you have to account for bad users, but if you're the only user then you can make sure everything is good. So, is it hard? That depends entirely on how wanted or unwanted the sent mail is.

xwowsersx 7y ago

It is clearly not trivial though. I get that it feels like it should be and probably is if you're sending a few emails, but "at scale" (pardon the buzzword) it is not trivial hence why people use these services.

ruffrey 7y ago

I've been running a SaaS in the email space since 2011. It is not surprising at all, to me, that Mailgun decided to do this. Sending an email _is_ trivial - you're right - but getting it delivered to the inbox, instead of the spam folder, is far from trivial. At scale, it is incredibly difficult. Even worse, spammers and fraudsters are utterly relentless. They hammer services like mine and Mailgun's constantly with fraud and phishing and porn and other garbage, inbound and outbound.

omarhaneef 7y ago

Yes, sending email is indeed a solved issue, but that isn't what mailgun (re?) solves. Asking someone's permissions first, tracking their desires and immediately updating your response is what they solve. Seems trivial? Try doing it for (looks at note) 626 people.

hnick 7y ago
I think their point was that it's been drummed into our heads that ISPs and mail services only accept email from 'trusted' IP blocks, so you can't do it yourself, better use 'the experts'.

nemothekid 7y ago

>despite sending an email being an utterly trivial, long solved problem AFAIK, SPAM is a still a hard problem, and getting emails delivered isn't trivial. "Under the guise of Spam" is a very reductionist viewpoint. It's like saying "Under the guise of the measles, we now have an oligopoly of pharmaceuticals." The latter might be true, but its important not to understate the threat of the former.

luckylion 7y ago

No, we've created something far better. ~50% of the Spam that makes it to my inbox is sent via Amazon SES. Amazon does not give a fuck, I've reported it to abuse, they ignore it. I can't just block Amazon SES though, because there's also a ton of legit services that use them for mail transport. We've really helped spammers evade blocking and decrease their infrastructure costs, and because there's so much "normal" traffic on SES, spammers will blend right in. And as long as they pay for services rendered, Amazon couldn't be happier.

ljm 7y ago

These providers are just as responsible for spam by running opt-out mailing lists and, even worse... spamming you with surveys and confirmations to see why you unsubscribed. With tracking pixels and rewritten URLs that go through their system before you get to the real destination. You don’t get a choice. That was made for you.

ttul 7y ago

If it's an oligopoly and presumably throwing off cash, what's stopping anyone from creating their own transactional email service? Answer: It's very difficult. Email is adversarial. Getting to the inbox is incredibly difficult to do _at scale_.

NightlyDev 7y ago
A private service where you are very sure that subscribers wants and opens the mail? Easy. A public service where a lot will be spam and unsolicited emails? Must be hell to run.

bdcravens 7y ago

> despite sending an email being an utterly trivial, long solved problem Then why do companies need transactional email providers?

mike-cardwell 7y ago

We have our own domain in G-Suite for our mail. I just checked my GMail Spam box. In it are 6 emails from this week. All 6 of them were sent through Mailgun, from and to the same domain (the one in G Suite). One was a cron email from one of our servers and the other 5 are transactional emails about people signing up to our service. All of them passed both SPF and DKIM. How. Just how did these emails get spam filtered by Google? Everything about them screams that they're legit emails from ourselves to ourselves.

Keverw 7y ago
Wow, sounds like even if you pay for a company to do then still might not go to inbox 100%... I noticed marking things not as spam still sends things to spam, I figured be doing that would tell it it's not spam... I was recently thinking about the email problem myself, and was reading about IP Warmups. Seems like every provider has different recommendations too. So sounds like not an exact science either. But maybe if it was an exact science it'd be hard to tell the good and bad guys apart. I guess one of the tricky problems with decentralized systems.

pyuser583 7y ago

You want to try an impressive open source software project? Create a framework that takes an email forwarded to a given address, turns it into an HTTP request with a JSON body, and sends that HTTP request to webhook. It’s easy to make a proof of concept. Production grade is an entirely different story. Remember to preserve conversations and forwarding. Make sure file attachments are handled and scanned for viruses. Make sure you watch out for recursive automatic forwarding. Make sure that any html is secure. And make sure it will scale well ... and I mean billions of emails a day well.

cx42net 7y ago
You are 100% right. It started the same with our service; we bought back ImprovMX (I don't put the link as I'm not looking for publicity - just for reference) when it was just a 5$ droplet at Digital Ocean, and now we have 10 big servers, more than 700$ monthly cost (and I honestly think it's not that much) and have to handle all the messy parts of delivering an email. It's easy to send an email, but it's way harder to ensure that the emails you send are delivered properly.