4 ms·

> The services you mention all have entire large teams devoted to tackling spam, reputation, uptime, etc. Can you provide specific numbers to the count of peop

> The services you mention all have entire large teams devoted to tackling spam, reputation, uptime, etc. Can you provide specific numbers to the count of people devoted to tackling spam, reputation, uptime, etc? I suspect the number's a lot smaller than you think. I also suspect the peoples' duties are less devoted than you think. If Google, for example, truly worked to tackle spam then it wouldn't have a spam problem today. If Yahoo truly worked to tackle reputation, people wouldn't have trouble sending email using an email client instead of a browser. I'll grant you that uptime does have dedicated people. But it's not to tackle spam or reputation. It's because offline services don't make a profit.

joshuamorton 7y ago
I work with these teams at Google. There are teams that actively (and solely) work on spam and abuse detection. They're larger than you seem to believe, though I won't give exact numbers. There's also obviously sre teams that maintain uptime. (Note I said teams, and there's a public approximate minimum size for an are team at Google of 8-12 people) The problem is that spam can't be "solved". Reputation is easy to solve: only accept email from a known list of good senders. Gmail, Yahoo, MailChimp (or not), etc. But that makes people on HN complain. So your have to try and infer reputation of mailservers on shared hosts. And spammers are always trying to beat you, and there are thousands, maybe tens of thousands of spam outfits. Af they're sneaky. They try to use awa or gcp to send email, or even send spam from Gmail, prevent trickier things. So you're left to defend from a spam campaign from Yahoo while also trying to not block everyone at Yahoo, and detect the spammers who are using Gmail to spam Yahoo too. And the spammers are always innovating, so you have to as well. My personal belief is that Google likely considers spam detection to be an area of competitive advantage so investments are warranted.
- inetknght 7y ago
  > The problem is that spam can't be "solved". Reputation is easy to solve: only accept email from a known list of good senders. Gmail, Yahoo, MailChimp (or not), etc. But that makes people on HN complain. So your have to try and infer reputation of mailservers on shared hosts. And spammers are always trying to beat you, and there are thousands, maybe tens of thousands of spam outfits. Af they're sneaky. They try to use awa or gcp to send email, or even send spam from Gmail, prevent trickier things. So you're left to defend from a spam campaign from Yahoo while also trying to not block everyone at Yahoo, and detect the spammers who are using Gmail to spam Yahoo too. Spam absolutely can be solved. 1) enforce identity. If the sender isn't authentic, then the sender is spam. 2) enforce reportability. If the user reports the sender as spam, then don't permit the sender to send more messages to the person who complained. if a lot of people report the problem, then block the sender. 3) enforce liability. if an ISP hosts spammers, then block the ISP. If someone complains then walk them through the process. Just like people shouldn't drive vehicles without understanding that vehicles are dangerous, the same should be done with computers.
  - joshuamorton 7y ago
    > If the sender isn't authentic, then the sender is spam. What's your definition of authentic? Is a self-hosted email server authentic? How do you decide? > If the user reports the sender as spam, then don't permit the sender to send more messages to the person who complained. If 10000 yahoo accounts are sending spam emails to other websites, what do you do? Block all yahoo senders? Try to block the yahoo accounts as they appear? > 3) enforce liability. if an ISP hosts spammers, then block the ISP. All major ISPs host spammers. Often they don't know that they do. Is it worth cutting off all comcast users nationwide from being able to use email? If anything, this would further centralize on one or two trustworthy email hosts, because those providers are essentially their own ISPs.
    - inetknght 7y ago
      > What's your definition of authentic? Is a self-hosted email server authentic? How do you decide? Authentic in terms of DNS. That means using and enforcing DKIM at the minimum. Also in terms of from: and reply-to: addresses matching each other. > If 10000 yahoo accounts are sending spam emails to other websites, what do you do? Block all yahoo senders? Try to block the yahoo accounts as they appear? If 10000 yahoo accounts are sending spam emails, then that's a Yahoo problem. Yes, I would refuse to accept incoming mail from @yahoo.com until they've fixed their complicity. > All major ISPs host spammers. Often they don't know that they do. I disagree about not knowing that they do. ISPs must respond to fraud and abuse reports or they would lose the ability to do business. ISPs not responding to spam reports are offloading the cost of policing their users onto you.
      - joshuamorton 7y ago
        > Authentic in terms of DNS. That means using and enforcing DKIM at the minimum. Sure, these are basic things that are generally used as strong signals, but all this does is filter out the incompetent spammers. If you're sending from yahoo or from gmail, you've already solved the reputation problem. And there are other ways of doing the same. > If 10000 yahoo accounts are sending spam emails, then that's a Yahoo problem. Yes, I would refuse to accept incoming mail from @yahoo.com until they've fixed their complicity. I'd expect that this is approximately the baseline number of yahoo accounts sending spam when they aren't being actively targeted. Its less than 1% of 1% of the active monthly accounts on yahoo. So you'd like to just block yahoo constantly? > I disagree about not knowing that they do. Sure they know, in the sense that I also know that there are always people spamming from every major ISP. That doesn't mean that they can immediately address things. And while you're busy blocking all comcast users from sending your users email, your users are busy moving to a different email provider that identifies individual spam senders so that they can still receive legitimate email. In closing, a simple question: if solving spam is this straightforward, why hasn't an upstart competitor (yahoo, protonmail, etc.) taken advantage of this strategy to fix the spam problem? It appears you're presuming a centralized system, which defeats the point of email and significantly simplifies the problem.

joshuamorton 7y ago

I work with these teams at Google. There are teams that actively (and solely) work on spam and abuse detection. They're larger than you seem to believe, though I won't give exact numbers. There's also obviously sre teams that maintain uptime. (Note I said teams, and there's a public approximate minimum size for an are team at Google of 8-12 people) The problem is that spam can't be "solved". Reputation is easy to solve: only accept email from a known list of good senders. Gmail, Yahoo, MailChimp (or not), etc. But that makes people on HN complain. So your have to try and infer reputation of mailservers on shared hosts. And spammers are always trying to beat you, and there are thousands, maybe tens of thousands of spam outfits. Af they're sneaky. They try to use awa or gcp to send email, or even send spam from Gmail, prevent trickier things. So you're left to defend from a spam campaign from Yahoo while also trying to not block everyone at Yahoo, and detect the spammers who are using Gmail to spam Yahoo too. And the spammers are always innovating, so you have to as well. My personal belief is that Google likely considers spam detection to be an area of competitive advantage so investments are warranted.

inetknght 7y ago
> The problem is that spam can't be "solved". Reputation is easy to solve: only accept email from a known list of good senders. Gmail, Yahoo, MailChimp (or not), etc. But that makes people on HN complain. So your have to try and infer reputation of mailservers on shared hosts. And spammers are always trying to beat you, and there are thousands, maybe tens of thousands of spam outfits. Af they're sneaky. They try to use awa or gcp to send email, or even send spam from Gmail, prevent trickier things. So you're left to defend from a spam campaign from Yahoo while also trying to not block everyone at Yahoo, and detect the spammers who are using Gmail to spam Yahoo too. Spam absolutely can be solved. 1) enforce identity. If the sender isn't authentic, then the sender is spam. 2) enforce reportability. If the user reports the sender as spam, then don't permit the sender to send more messages to the person who complained. if a lot of people report the problem, then block the sender. 3) enforce liability. if an ISP hosts spammers, then block the ISP. If someone complains then walk them through the process. Just like people shouldn't drive vehicles without understanding that vehicles are dangerous, the same should be done with computers.
- joshuamorton 7y ago
  > If the sender isn't authentic, then the sender is spam. What's your definition of authentic? Is a self-hosted email server authentic? How do you decide? > If the user reports the sender as spam, then don't permit the sender to send more messages to the person who complained. If 10000 yahoo accounts are sending spam emails to other websites, what do you do? Block all yahoo senders? Try to block the yahoo accounts as they appear? > 3) enforce liability. if an ISP hosts spammers, then block the ISP. All major ISPs host spammers. Often they don't know that they do. Is it worth cutting off all comcast users nationwide from being able to use email? If anything, this would further centralize on one or two trustworthy email hosts, because those providers are essentially their own ISPs.
  - inetknght 7y ago
    > What's your definition of authentic? Is a self-hosted email server authentic? How do you decide? Authentic in terms of DNS. That means using and enforcing DKIM at the minimum. Also in terms of from: and reply-to: addresses matching each other. > If 10000 yahoo accounts are sending spam emails to other websites, what do you do? Block all yahoo senders? Try to block the yahoo accounts as they appear? If 10000 yahoo accounts are sending spam emails, then that's a Yahoo problem. Yes, I would refuse to accept incoming mail from @yahoo.com until they've fixed their complicity. > All major ISPs host spammers. Often they don't know that they do. I disagree about not knowing that they do. ISPs must respond to fraud and abuse reports or they would lose the ability to do business. ISPs not responding to spam reports are offloading the cost of policing their users onto you.
    - joshuamorton 7y ago
      > Authentic in terms of DNS. That means using and enforcing DKIM at the minimum. Sure, these are basic things that are generally used as strong signals, but all this does is filter out the incompetent spammers. If you're sending from yahoo or from gmail, you've already solved the reputation problem. And there are other ways of doing the same. > If 10000 yahoo accounts are sending spam emails, then that's a Yahoo problem. Yes, I would refuse to accept incoming mail from @yahoo.com until they've fixed their complicity. I'd expect that this is approximately the baseline number of yahoo accounts sending spam when they aren't being actively targeted. Its less than 1% of 1% of the active monthly accounts on yahoo. So you'd like to just block yahoo constantly? > I disagree about not knowing that they do. Sure they know, in the sense that I also know that there are always people spamming from every major ISP. That doesn't mean that they can immediately address things. And while you're busy blocking all comcast users from sending your users email, your users are busy moving to a different email provider that identifies individual spam senders so that they can still receive legitimate email. In closing, a simple question: if solving spam is this straightforward, why hasn't an upstart competitor (yahoo, protonmail, etc.) taken advantage of this strategy to fix the spam problem? It appears you're presuming a centralized system, which defeats the point of email and significantly simplifies the problem.