8 ms·

Gmail is killing independent email

235 points by foxylad 4y ago

ssgodderidge 4y ago
Besides Google playing nice and fixing their stuff, is there a possible fix for this?
- rubatuga 4y ago
  Legislation demanding interoperability and humans we can call when problems arise.
  - vanilla_nut 4y ago
    Or better yet, break it up into Gmails A-Z. Force them to interop and compete.
  - eternalban 4y ago
    Yes. Email is public infrastructure and Alphabet/Google should not be permitted to continue to stifle and control email.
    - TedDoesntTalk 4y ago
      Most teenagers don’t even use your “public infrastructure.” They never started with email and won’t unless they get a job with it. I don’t think it’s public infrastructure.
- tempestn 4y ago
  Only way I know of is to pay a company that provides SMTP service and is too big for gmail to ignore. Some have free tiers if your volume is low. I switched my personal domain email over to sendgrid for that reason.
- devmunchies 4y ago
  If you’re being rate limited by IP, then I would guess you need more IPs to spread the load. Maybe even stagger these mass emails to send over 1-2 minutes.
  - foxylad 4y ago
    We have two email servers for redundancy (like when Gmail is rate-limiting), not load. We send about 10,000 messages a day total, or about seven per minute, spread fairly evenly across the day. No mass email. I suppose it would be an interesting exercise to slowly increase servers/IP addresses until rate limiting stops, but servers cost time and money.
- ocdtrekkie 4y ago
  Unfortunately it's difficult because Google is beholden to nobody. If you have an issue with a telecom blocking legitimate communications (including email), you can complain to the FCC, and it will get fixed shockingly expediently (hours, not days), meanwhile Google will happily ignore all regulatory bodies. Google is terrified of being classified as a telecom because telecoms are held accountable. And yes, I have direct personal experience here. :)
walrus01 4y ago
It's very easy to get legitimate emails false-positived as spam by gmail/gsuite or office365 MX even if your sending IP address is not in a bad-reputation IP block, your rdns, spf, dkim and dmarc are impeccable. I know people who've been running mx/smtp servers on the internet since 1994 who have now given up on running their own, not because they're technically incapable or unwilling to continue to do so, but because they've been forced into themselves using gsuite or office365 because of the monopolistic practices of the huge email-as-a-service providers.
- jrochkind1 4y ago
  Are mid-tier players like FastMail screwed too? Is this a reason to be careful moving to, say, FastMail?
  - daydream 4y ago
    I’ve used Fastmail for email since 2015 or so, with a custom domain, and have never had any issues with email delivery to Gmail, Office365, or any other email host. Data point of one. It’s something I occasionally worry about actually but it has never happened to me. So I probably just shouldn’t worry.
  - ShamelessC 4y ago
    No.
  - jasonriddle 4y ago
    I've been using Fastmail since last year, also with a custom domain. I haven't experienced any issues with deliverability so far.
  - calvinmorrison 4y ago
    not really. providers have a lot of tools that monitor IP reputation, and also human contacts never hurt. Fastmail and other providers maintain their reputation by taking spam seriously and having prudent rules in place. source: i worked at fastmail and regularly got oncall alerts to go deal with some rbl issues
tempestn 4y ago
Yep, after over a decade of trying to run independent mail servers we finally broke down and switched to SendInBlue and SendGrid last year. Even if you do everything right, it's just not possible to get gmail to reliably deliver mail unless you use one of the big services for smtp. It's really disappointing. Edit: We can at least keep using our servers for incoming mail, and just use those for sending.
- mouse_ 4y ago
  Anyone embracing and extending anything to the point of Gmail should be held suspect.
  - echelon 4y ago
    Newsgroups, RSS, ... Google kills a lot of the bigger ecosystem.
    - MBCook 4y ago
      Web browsers…
    - m-p-3 4y ago
      At least they didn't kill instant messaging with their disastrous strategy.
      - tambourine_man 4y ago
        Not for the lack of trying, just sheer incompetence.
      - pvb 4y ago
        they did kill quiet a bit of the XMPP ecosystem imho...
    - zitterbewegung 4y ago
      Embrace, extend then extinguish.
crazygringo 4y ago
That sucks, big time. What I don't get is, what is Google trying to achieve with this? A rate limit that doesn't increase daily or check the spam rate or provide a support rep to lift it? Gmail has been around a long time. Is this the first we're hearing about this?
jfindley 4y ago
They say "Both email servers have PTR records set up, and SPF [...] DKIM, and DMARC records[...]". Yes. Great. Thing is that this is such a trivial barrier to entry that guess what? Spammers do it too! Email has become so utterly corrupted with spam that the reality is that an independent provider who has no existing reputation is, 99% of the time, going to be a spammer. It would be wonderful if we could fix this - but so far noone's come up with a workable solution.
- midoridensha 4y ago
  I agree. It really sucks that a handful of big corps control email now, and we can't all just have our own email servers like in the old days, but the spammers really ruined it. And I don't see how any technical solution could change this: anything that's free and open-source can be easily used by the spammers just like anyone else, and then subverted. The solution to spam isn't technical, it's legislative and judicial, but that's just not possible because they operate across national borders and no one's going to start a shooting war over spam.
- ggm 4y ago
  It's not that the solutions are unworkable: The problem is that they are unpalatable. There are outlier Mail-wonks who maintain pay-to-send wouldn't "work" but they are somewhat in a minority: If you forced senders to pay even tokenistic per-mail sums, the attractiveness of mail would disappear. People don't want to monetize Mail for complex reasons. The "won't work" aspect in spam pushback has always been debatable. (there's a well known checkbox list of "your proposed anti-spam mechanism won't work because.. which is a huge antipattern to having a rational debate about it) The problems are regulatory: who sets the price, who collects the price, whats done with the money, and what it does to the ecology of email internationally. But, it would within some definitions of the term "work"
  - yipbub 4y ago
    Spammers are the most willing to pay negligible amounts. Way more than regular users. They pay for tools, servers, services, etc. because they have some expected return on email.
newsclues 4y ago
It's not just gmail, Big Email is a cartel.
- drexlspivey 4y ago
  That’s how federated protocols work, you choose who you want to federate with. See mastodon instances blocking other instances
b7r6 4y ago
For me at least, existing completely without GMail / Google SSO is unrealistic, and the million email corpus that is attached to it would be painful to part with as well. But I've been trying to slowly de-tangle myself, and for it's faults ProtonMail has been working out pretty OK for me as a compromise between usability and true digital freedom.
- hedora 4y ago
  It’s not hard to move a million emails over imap between providers. However, gmail has some anti abuse throttling, so you might need to do a one time painful manually rate limited slow sync to get them out. The smaller providers generally don’t engage in such bullshit, in my experience.
teawrecks 4y ago
This is what all fediverse protocols and specs gaining traction today should be designing for. Email was lucky and got their foot in the door before the dotcom era figured out about walled gardens. Now we're slowly crawling back out of them, but if we're not careful we'll be out of the pot and into the frying pan.
- ocdtrekkie 4y ago
  Unfortunately, the fediverse likely will have many of the same problems as email and RBLs today: It's less automated, but groups of admins share servers to block en masse, and not all admins bother to vet the claims before doing so. Once your server is on enough servers' block lists, it's near impossible to get off of them. One of the most useful block lists people upload is generated based on how many other servers have blocked it. So once you're there, even if the situation has changed, you may have to petition a dozen or so servers to remove a block on you before you come off the list everyone else is using. Botspam is an increasing problem, servers can get blacklisted because too many spammers get on it faster than the admin addresses the issue. And there's really no automated tools for handling all of it yet.
appleflaxen 4y ago
This is anticompetitive, and should be enforced as such. Unfortunately regulatory capture is real.
peanut-walrus 4y ago
Microsoft is even worse. They are blocking their own customers from replying to emails because their outgoing smtp server rejects it as spam. Despite all the advancement in machine learning, spam filtering seems to keep getting worse. And both Google and Microsoft have absolutely laughable support for when your emails are getting blocked. Could we just dial things back a bit? Some spam is fine if the alternative is having an unknown amount of legitimate messages not delivered to you.
- hackerfactor1 4y ago
  I was going to say the same thing. Even if you have SPF and DKIM working correctly, they will block your email to outlook, hotmail, and other services unless you fill out their form. If you fill out the form, then you still need to wait months before they decide whether to accept your email. On my own servers, I warn users who try to use hotmail or outlook that my email to them may not work and they should use a different email address.
  - taylodl 4y ago
    Now imagine you’re an electric utility trying to email customers their bills - email is a mess!
- hedora 4y ago
  Funny. I have none of these spam problems with Fastmail. I assume the problem is because email is an afterthought at the big near-monopoly providers.
- nunez 4y ago
  This was why I left Microsoft 364. Their spam filtering is insane. I lost lots of legitimate emails to my junk mail folder on Microsoft.
thrwy_918 4y ago
I can't even get Gmail to deliver my 100% genuine, properly authenticated, sent-from-my-own-domain-which-has-never-sent-spam firebase sign-in emails - and it's the same parent company! The get sent to spam or (worse) silently have the links stripped out a decent percentage of the time
teddyh 4y ago
Off-topic: > We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+". So something like squash:club! makes an excellent password. That is terrible advice.
- deleted 4y ago
  [deleted]
- Zurrrrr 4y ago
  Not really, it's a great way to make passwords. Chose a nonsense phrase from some popucluture thing or whatever, and add in or replace some letters with numbers and symbols. Something like kRyptoni4n!muGgl342r0nin is easy to remember, and not being brute forced anytime soon.
  - kristopolous 4y ago
    Ever run a dictionary cracker like hashcat? You really really should, with modern GPU hardware. Read the documentation too. Your example is a pretty early match. letter/number cycling, single case substitutions, symbol concatenation - GPUs can cycle through those in parallel, fast. It's a guessable password + guessable ruleset. You've introduced far less entropy than you've imagined. If you want to have protection against attacks and use your technique, you have to come up with a ruleset you (1) don't see in the documentation (2) that's computationally unreasonable. For instance, combine multiple languages. Strawberry Octopus Sundae, which is pretty memorable, can become FragolaOctopusEisbecher combining Italian, English and German. And with that we just went O(N^3) where N is dictionary size (actually more than that because you've got a wide choice of latin scripted language). That gives you far more protection then say sTr4b33ry+0ct0pu5!5und43. You aren't fooling a GeForce RTX 4090 with those tricks. That's an example I just came up with though. There's lots of things like that. But really run a cracker. Try it out.
    - Zurrrrr 4y ago
      The example I gave isn't that guessable. It's combining things from different popculture into one new phrase that won't be in any dictionary. An even better attempt is to insert a whole word between two sides of a broken up word, e.g. `4mAzonij-0rdan4Npr7nces7`. And yes I use hashcat frequently. Using something like a 20 character password following that formula is going to be fine for most people - it isn't going to be brute forced before someone gets an alert that passwords are compromised and can change it. It's a much better solution IMO than using something entirely random and writing it down, or using something like LastPass.
regnull 4y ago
Yes, email has evolved from an open standard to a closed system dominated by only a few big corporations. I don't see how is this situation acceptable - such a fundamental functionality is out of reach to all but a few of us. Well, Gmail and such are free and convenient (until you get banned for no reason), so it's good enough for the most. Trying to get back to the open decentralized ecosystem is a waste of time. But we do it anyway - and we start with the identity layer. Email based on blockchain identity, free and open. I've been working on this for a while, still WIP but check it out: https://ubikom.cc https://ubikom.cc
Analemma_ 4y ago
I understand this guy's frustration but this: > And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem. is just wrong. Tons of spam comes from servers with SPF, DKIM and DMARC now. It stopped being a trustworthy signal of not-spam many years ago.
- ed_mercer 4y ago
  That explains why I'm starting to receive spam in Outlook again... Guess it's a cat and mouse game.
- cge 4y ago
  Having SPF and DKIM implemented says little about the email coming from a server. But it should allow filtering of spam from domains based on the history of emails received from them, and how users have interacted with them. They should allow the keeping of trust scores at a domain, rather than server or IP, level, and that should be feasible with even relatively few emails involved. Yet Google and Microsoft don't do that. I can send replies on a personal domain and server to people I've talked with often in the past, and quite often they'll go to spam with Gmail. I can send emails to universities running Office365 from the domain of a prominent university, and sometimes they'll go to spam. I've seen Google and Microsoft email at universities have emails from mts-nature.nature.com, Nature's manuscript tracking system, something no academic would want sent to spam, get sent to spam. SPF and DKIM should allow these things to be rather simply avoidable. But they don't.
- billpg 4y ago
  SPF et al are not anti-spam but anti-impersonation.
favsq 4y ago
[flagged]
CodeWriter23 4y ago
An open system right up until your users use the junk button as the delete button.
- foxylad 4y ago
  We are registered with Google's spam feedback loop, and Microsofts SNDS, so we get informed about any messages marked as spam. We get maybe one a week, so this isn't what triggers the rate-limiting.
jijji 4y ago
i wound up developing a pretty complex email sending infrastructure because of situations like this. it involves putting a message in a queue and then sending to a list of predefined proxies (mailjet, mailgun,sendgrid, etc, etc) and using backoff algorithms per proxy per hour....
civilized 4y ago
Meanwhile Gmail happily lets through any scam email from any hijacked account on outlook.com.
throwawaaarrgh 4y ago
Heretical opinion: the government does a good job with postal mail. Let them handle electronic mail. Most of the problems of spam have to do with who's an approved sender and who's abusive. Regulate the mail, it's much harder to be abusive. You're in this country and sending spam? We arrest you. You're in another country? We rate limit your mail, report you to foreign authorities, and flag anything that looks like your mail. You can burn me at the stake now.
- 35208654 4y ago
  I think there's room here to have a 'public option' open source implementation with appropriate governance / rules. Give everyone a basic 10 digit @ email.usa.gov. But I also think there should be a government regulated digital 'town square' for each level of government, with township / county being optional but the rest funded by taxes. Then have those groups determine what is allowed and what's not, through people they nominate in elections. This is like the 'public option' for Facebook. But mostly I just think this because I want more tax dollars to go to open source projects trying to solve problems that are very real and very much caused by the incentives of private industry.
- theamk 4y ago
  If I am a user, there is no way I am entering my government-issued ID into your website (unless it deals with money) If I am a website owner, no way I am going to use regulated service that will bring police to my door if I make a bug in my script, unless I have to.
- soraminazuki 4y ago
  How about no? Democracies have no business making it more efficient for the government to hoard or sift through emails.
  - EVa5I7bHFq9mnYK 4y ago
    Sure they have pretty efficient arrangements with gmail and outlook right now.
zinekeller 4y ago
Okay, instead of taking this at face value (with a big acknowledgment that it is genuinely frustrating that both Google and Microsoft don't have good contacts to just know what went wrong), I'll analyse this carefully. This website is operated by Virtual Industries Group (https://www.vig.co.nz/ https://www.vig.co.nz/), which is according to their website operates three services: School Bookings/School Interviews (https://www.schoolinterviews.co.nz/ https://www.schoolinterviews.co.nz/, focused on school scheduling and the service discussed here), Care Bookings (https://www.carebookings.co.nz/ https://www.carebookings.co.nz/, which is the same service as above but focuses on day-care and other similar functions), and MessageMyWay (https://www.messagemyway.com/ https://www.messagemyway.com/, which according to their website "is the communications hub for your community. It is your emergency communications plan, your telephone tree, and your email broadcast system all in one"). From a cursory glance, these three services shares this set of outbound MX servers. While it is very unlikely that someone who uses School Interview and Care Bookings would mark their message as spam, if the messages relayed by MessageMyWay is sent on the same outbound MXes then I could immediately see the problem. A large part of MX operators knows this and separates "marketing" and "operational" messages into separate servers to prevent this exact thing from happening. While I understand this dev's frustration, maybe the messages relayed by MessageMyWay are the ones marked as spam by frustrated parents who are receiving irrelevant school marketing which are sent to same MXes as their important operational messages? If I were the developer (and still insist on using on-prem email), I'll operate three groups of servers: Set A: purely for company-initiated messages, never for the customers Set B: "operational" customer-initiated messages: School Bookings, Care Bookings, and MessageMyWay mails which are marked "critical" by the users (which is apparently additional cost) Set C: "marketing" customer-initiated messages: "normal" MessageMyWay mails
- foxylad 4y ago
  I take my hat off to you - an impressive amount of research went into this! MessageMyWay is opt-in and has a robust opt-out mechanism - and is sadly moribund. It sends a maximum of a couple of hundred emails a day. And we are registered with GMail spam feedback loop and Microsoft's SNDS, both of which tell us when someone marks a message as spam. This happens less than once a week, so this isn't the signal that triggers the rate-limiting.
pentagrama 4y ago
I have a more positive experience than the OP, I worked with a news outlet that has several newsletters and sent about 200K emails/day, and we use our own servers/domain and an open source/self-hosted tool to send them (I think it was Mailman [1]). We did all those technical bits like SPF and DKIM, put the one click unsubscribe link on the message and also on the header of the message so that clients like Gmail can put the unsubscribe link on their own UI [2], all the recommended practices. I remember using the tool Mail Tester [3] and the results where all green. We doesn't have issues with being marked as SPAM by Gmail/Outlook, and have average open rates of 50%, which is a lot higher than the industry standard who is around 20% [4]. We have a good UX and an ethical way to treat our users, like all users have to opt-in to their desired newsletters when creating an account (or choose to receive newsletters without creating an account), one-click unsubscribe link in big text at the bottom of each newsletter, but also a one-click-no-need-to-be-logged-in link to opt-out of all the newsletters that the user was subscribed, and more stuff like that. But one thing that I think give us a lot of reputation for the Gmail algorithm, it was that we designed a feature that if the user doesn't opened a newseltter for about 3 months, we started to send the newsletters with an alert at the top saying like "Seems that you aren't reading this newsletter anymore, you will be automatically unsubscribed in 30[n counter] days. Click here to disable the auto-unsuscribe." (the disable the auto-unsuscribe also was for people who have images disabled and we can't track the openings, but is small percentage). So with that feature, we get sure that our users where engaged with the newsletters, and we have a system to avoid sending messages to "dead" emails, maintaining a fresh and healthy database of emails, and seems that Gmail/Outlook knows and likes that. [1] https://www.list.org/ https://www.list.org/ [2] https://www.sendinblue.com/blog/list-unsubscribe-header/ https://www.sendinblue.com/blog/list-unsubscribe-header/ [3] https://www.mail-tester.com/ https://www.mail-tester.com/ [4] https://mailchimp.com/resources/email-marketing-benchmarks/ https://mailchimp.com/resources/email-marketing-benchmarks/
- kilburn 4y ago
  It is important to point out the several unusual things that went right in this story: 1. Your decision makers recognized that the important number to pursue is "engaged users", not "number of emails in the list". 2. You have a large enough subscriber base so that Gmail/Outlook realize you're a good player. This wouldn't happen if you only sent a few thousand mails per week. 3. You have the technical expertise to properly maintain the list and the surrounding (unsubscribe, etc.) infrastructure. Usually the budget is allocated to _create_ the list/service and then assumed that it will continue working forever with 0 investment. IF you can get these things right too you'll (eventually) have a similar experience. If you fail on any of them... you're screwed like everyone else is saying. Of course, having to send large volumes of email to ensure that "the big guys" treat you fairly is why many people running small/personal email servers complains about them.
foxylad 4y ago
OP here. For those thinking that Gmail MUST have valid reasons, and that we abusing the system in some way, consider... 1. We are registered with Gmail, Microsoft and Yahoo to provide feedback when someone marks our messages as spam (we DO do everything right). We get reports less than once a week, so if I've got my maths right, less than 0.002% of recipients think we're spammers. 1. Our two servers send around 10,000 messages a day, spread fairly evenly over time. That works out at three or four messages per server per minute. If we were spammers, we'd be ashamed of ourselves.
rkagerer 4y ago
Could their behavior towards legitimate email from small-time outfits like this be useful evidence in an anti-compete lawsuit? Alternatively, some kind of class action? (Asking any lawyers in the room)
studentik 4y ago
Why is server spam filtering even needed with Junk/NotJunk feature in Thunderbird?
- Proven 4y ago
  [dead]
- beej71 4y ago
  It's what I use, but it means I have to have Thunderbird running all the time or I see all the spam on my phone. It's just kind of kludgey compared to a server-based solution.
deleted 4y ago
[deleted]
giomasce 4y ago
The promise behind SPF, DKIM and DMARC is that they allowed positive reputation mechanisms, which cannot work if any domain is easy to spoof. So if you're careful, only send good email and stop sending when the recipient asks, that should be a golden star on your review, destination domains will happily let your email through and nobody will be able to freeride on your good reputation because your domain is authenticated. However, this is still hard to establish, because it depends on the destination domain server to decide whether you're sending good stuff or not. I would like to have a mechanism by which the user can decide. I imagine something like this: each email provider, say Gmail, issues to its users a number of single use codes like "Sor7xeik". When the user wants to subscribe to a newsletter (say news@interesting.com) it gives its own email address and one of those codes. The first email from news@interesting.com contains some header like Authorized-Sender: authorize Sor7xeik When Gmail receives it, scratches the code and marks @interesting.com as an authorized domain. From that point on, all (DMARC validated) emails from @interesting.com having some header like Authorized-Sender: yes are deemed to be interesting for that specific user, and accepted without further spam filtering. The user can revoke the consent at any time on Gmail's web interface, at which point emails from @interesting.com (with that header) will be rejected. The sender at @interesting.com will see the rejection and disable mail sending for that user. With this mechanism bad practices like address harvesting and selling become much less useful (because an address alone is not that useful, if the sender is not authorized; and the authorization must be initiated by the user). BTW, I am not saying that all emails should be sent with this authorized sender mechanism. I don't expect individual users to collect authorizations for each of their contacts. Email without the Authorized-Sender header would still be subject to the usual spam filtering, but agencies that often send legitimate mass emails can have a mechanism to prove that they're doing it with the user authorization.
illiac786 4y ago
This is why Proton is extremely aggressive with its free users and close accounts seemingly arbitrarily. Any abuse of their service could cause them to be banned because they are small. Google doesn't have to care, who is going to ban Google SMTP servers? That would be suicide.
rkwasny 4y ago
I am running my own server @entropy.be for last 20 years. Maybe I'm lucky but it just works? The secret is only to have trusted users that do not send spam. OK I get the occasional spam through spamassasin, but I can live with archiving 5 spam emails/day.
deafpolygon 4y ago
killing? More like murdered.
93po 4y ago
I tried to move off fastmail back to gmail last year because I lost a job opportunity due to a random fastmail-origin email i sent getting blocked by my interviewer's google mail. the first three went through fine, it was only the fourth one that didn't make it. they thought i ghosted and had already chosen another candidate because of it. ultimately i didn't make the move because email is unusable without custom aliases i can completely bounce email from. but i think i might attempt again, but again I recently found myself insanely paranoid about my interviewing situation and had to make the awkward transitioning to my gmail account and explain why to my interviewer
rediguanayum 4y ago
Disclosure: I work in Gmail. Likely there's a reputation issue on those IPs. Consider filling out a form to investigate that: https://support.google.com/mail/contact/gmail_bulk_sender_escalation?visit_id=638158923865383305-3828756122&rd=1 https://support.google.com/mail/contact/gmail_bulk_sender_es...
- foxylad 4y ago
  OP here: thanks for poking your head over the parapet. But we've submitted that form three times over five months, with no discernible effect. I have wondered if a pattern-matching algorithm is penalising us for having a similar IP address to a spammer. But given that practically all servers are now located at hosting companies, IP address is a completely valueless signal - and an arbitrarily damaging one. We have no control over other servers near us in IP space, and can't even ask our hosting company to deal with the spammer because we have no idea which IP address they are on. If you have access to an internal directory, could I prevail on you to pass this concern on to the Gmail spam filter team? I think there is a genuine bug here, and if they fix it and let me know, I promise to post a followup article saying that fixing this shows GMail isn't killing independant email on purpose.

235 points by foxylad 4y ago

ssgodderidge 4y ago

Besides Google playing nice and fixing their stuff, is there a possible fix for this?

rubatuga 4y ago
Legislation demanding interoperability and humans we can call when problems arise.
- vanilla_nut 4y ago
  Or better yet, break it up into Gmails A-Z. Force them to interop and compete.
- eternalban 4y ago
  Yes. Email is public infrastructure and Alphabet/Google should not be permitted to continue to stifle and control email.
  - TedDoesntTalk 4y ago
    Most teenagers don’t even use your “public infrastructure.” They never started with email and won’t unless they get a job with it. I don’t think it’s public infrastructure.
tempestn 4y ago
Only way I know of is to pay a company that provides SMTP service and is too big for gmail to ignore. Some have free tiers if your volume is low. I switched my personal domain email over to sendgrid for that reason.
devmunchies 4y ago
If you’re being rate limited by IP, then I would guess you need more IPs to spread the load. Maybe even stagger these mass emails to send over 1-2 minutes.
- foxylad 4y ago
  We have two email servers for redundancy (like when Gmail is rate-limiting), not load. We send about 10,000 messages a day total, or about seven per minute, spread fairly evenly across the day. No mass email. I suppose it would be an interesting exercise to slowly increase servers/IP addresses until rate limiting stops, but servers cost time and money.
ocdtrekkie 4y ago
Unfortunately it's difficult because Google is beholden to nobody. If you have an issue with a telecom blocking legitimate communications (including email), you can complain to the FCC, and it will get fixed shockingly expediently (hours, not days), meanwhile Google will happily ignore all regulatory bodies. Google is terrified of being classified as a telecom because telecoms are held accountable. And yes, I have direct personal experience here. :)

walrus01 4y ago

It's very easy to get legitimate emails false-positived as spam by gmail/gsuite or office365 MX even if your sending IP address is not in a bad-reputation IP block, your rdns, spf, dkim and dmarc are impeccable. I know people who've been running mx/smtp servers on the internet since 1994 who have now given up on running their own, not because they're technically incapable or unwilling to continue to do so, but because they've been forced into themselves using gsuite or office365 because of the monopolistic practices of the huge email-as-a-service providers.

jrochkind1 4y ago
Are mid-tier players like FastMail screwed too? Is this a reason to be careful moving to, say, FastMail?
- daydream 4y ago
  I’ve used Fastmail for email since 2015 or so, with a custom domain, and have never had any issues with email delivery to Gmail, Office365, or any other email host. Data point of one. It’s something I occasionally worry about actually but it has never happened to me. So I probably just shouldn’t worry.
- ShamelessC 4y ago
  No.
- jasonriddle 4y ago
  I've been using Fastmail since last year, also with a custom domain. I haven't experienced any issues with deliverability so far.
- calvinmorrison 4y ago
  not really. providers have a lot of tools that monitor IP reputation, and also human contacts never hurt. Fastmail and other providers maintain their reputation by taking spam seriously and having prudent rules in place. source: i worked at fastmail and regularly got oncall alerts to go deal with some rbl issues

tempestn 4y ago

Yep, after over a decade of trying to run independent mail servers we finally broke down and switched to SendInBlue and SendGrid last year. Even if you do everything right, it's just not possible to get gmail to reliably deliver mail unless you use one of the big services for smtp. It's really disappointing. Edit: We can at least keep using our servers for incoming mail, and just use those for sending.

mouse_ 4y ago
Anyone embracing and extending anything to the point of Gmail should be held suspect.
- echelon 4y ago
  Newsgroups, RSS, ... Google kills a lot of the bigger ecosystem.
  - MBCook 4y ago
    Web browsers…
  - m-p-3 4y ago
    At least they didn't kill instant messaging with their disastrous strategy.
    - tambourine_man 4y ago
      Not for the lack of trying, just sheer incompetence.
    - pvb 4y ago
      they did kill quiet a bit of the XMPP ecosystem imho...
  - zitterbewegung 4y ago
    Embrace, extend then extinguish.

crazygringo 4y ago

That sucks, big time. What I don't get is, what is Google trying to achieve with this? A rate limit that doesn't increase daily or check the spam rate or provide a support rep to lift it? Gmail has been around a long time. Is this the first we're hearing about this?

jfindley 4y ago

They say "Both email servers have PTR records set up, and SPF [...] DKIM, and DMARC records[...]". Yes. Great. Thing is that this is such a trivial barrier to entry that guess what? Spammers do it too! Email has become so utterly corrupted with spam that the reality is that an independent provider who has no existing reputation is, 99% of the time, going to be a spammer. It would be wonderful if we could fix this - but so far noone's come up with a workable solution.

midoridensha 4y ago
I agree. It really sucks that a handful of big corps control email now, and we can't all just have our own email servers like in the old days, but the spammers really ruined it. And I don't see how any technical solution could change this: anything that's free and open-source can be easily used by the spammers just like anyone else, and then subverted. The solution to spam isn't technical, it's legislative and judicial, but that's just not possible because they operate across national borders and no one's going to start a shooting war over spam.
ggm 4y ago
It's not that the solutions are unworkable: The problem is that they are unpalatable. There are outlier Mail-wonks who maintain pay-to-send wouldn't "work" but they are somewhat in a minority: If you forced senders to pay even tokenistic per-mail sums, the attractiveness of mail would disappear. People don't want to monetize Mail for complex reasons. The "won't work" aspect in spam pushback has always been debatable. (there's a well known checkbox list of "your proposed anti-spam mechanism won't work because.. which is a huge antipattern to having a rational debate about it) The problems are regulatory: who sets the price, who collects the price, whats done with the money, and what it does to the ecology of email internationally. But, it would within some definitions of the term "work"
- yipbub 4y ago
  Spammers are the most willing to pay negligible amounts. Way more than regular users. They pay for tools, servers, services, etc. because they have some expected return on email.

newsclues 4y ago

It's not just gmail, Big Email is a cartel.

drexlspivey 4y ago
That’s how federated protocols work, you choose who you want to federate with. See mastodon instances blocking other instances

b7r6 4y ago

For me at least, existing completely without GMail / Google SSO is unrealistic, and the million email corpus that is attached to it would be painful to part with as well. But I've been trying to slowly de-tangle myself, and for it's faults ProtonMail has been working out pretty OK for me as a compromise between usability and true digital freedom.

hedora 4y ago
It’s not hard to move a million emails over imap between providers. However, gmail has some anti abuse throttling, so you might need to do a one time painful manually rate limited slow sync to get them out. The smaller providers generally don’t engage in such bullshit, in my experience.

teawrecks 4y ago

This is what all fediverse protocols and specs gaining traction today should be designing for. Email was lucky and got their foot in the door before the dotcom era figured out about walled gardens. Now we're slowly crawling back out of them, but if we're not careful we'll be out of the pot and into the frying pan.

ocdtrekkie 4y ago
Unfortunately, the fediverse likely will have many of the same problems as email and RBLs today: It's less automated, but groups of admins share servers to block en masse, and not all admins bother to vet the claims before doing so. Once your server is on enough servers' block lists, it's near impossible to get off of them. One of the most useful block lists people upload is generated based on how many other servers have blocked it. So once you're there, even if the situation has changed, you may have to petition a dozen or so servers to remove a block on you before you come off the list everyone else is using. Botspam is an increasing problem, servers can get blacklisted because too many spammers get on it faster than the admin addresses the issue. And there's really no automated tools for handling all of it yet.

appleflaxen 4y ago

This is anticompetitive, and should be enforced as such. Unfortunately regulatory capture is real.

peanut-walrus 4y ago

Microsoft is even worse. They are blocking their own customers from replying to emails because their outgoing smtp server rejects it as spam. Despite all the advancement in machine learning, spam filtering seems to keep getting worse. And both Google and Microsoft have absolutely laughable support for when your emails are getting blocked. Could we just dial things back a bit? Some spam is fine if the alternative is having an unknown amount of legitimate messages not delivered to you.

hackerfactor1 4y ago
I was going to say the same thing. Even if you have SPF and DKIM working correctly, they will block your email to outlook, hotmail, and other services unless you fill out their form. If you fill out the form, then you still need to wait months before they decide whether to accept your email. On my own servers, I warn users who try to use hotmail or outlook that my email to them may not work and they should use a different email address.
- taylodl 4y ago
  Now imagine you’re an electric utility trying to email customers their bills - email is a mess!
hedora 4y ago
Funny. I have none of these spam problems with Fastmail. I assume the problem is because email is an afterthought at the big near-monopoly providers.
nunez 4y ago
This was why I left Microsoft 364. Their spam filtering is insane. I lost lots of legitimate emails to my junk mail folder on Microsoft.

thrwy_918 4y ago

I can't even get Gmail to deliver my 100% genuine, properly authenticated, sent-from-my-own-domain-which-has-never-sent-spam firebase sign-in emails - and it's the same parent company! The get sent to spam or (worse) silently have the links stripped out a decent percentage of the time

teddyh 4y ago

Off-topic: > We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+". So something like squash:club! makes an excellent password. That is terrible advice.

deleted 4y ago
[deleted]
Zurrrrr 4y ago
Not really, it's a great way to make passwords. Chose a nonsense phrase from some popucluture thing or whatever, and add in or replace some letters with numbers and symbols. Something like kRyptoni4n!muGgl342r0nin is easy to remember, and not being brute forced anytime soon.
- kristopolous 4y ago
  Ever run a dictionary cracker like hashcat? You really really should, with modern GPU hardware. Read the documentation too. Your example is a pretty early match. letter/number cycling, single case substitutions, symbol concatenation - GPUs can cycle through those in parallel, fast. It's a guessable password + guessable ruleset. You've introduced far less entropy than you've imagined. If you want to have protection against attacks and use your technique, you have to come up with a ruleset you (1) don't see in the documentation (2) that's computationally unreasonable. For instance, combine multiple languages. Strawberry Octopus Sundae, which is pretty memorable, can become FragolaOctopusEisbecher combining Italian, English and German. And with that we just went O(N^3) where N is dictionary size (actually more than that because you've got a wide choice of latin scripted language). That gives you far more protection then say sTr4b33ry+0ct0pu5!5und43. You aren't fooling a GeForce RTX 4090 with those tricks. That's an example I just came up with though. There's lots of things like that. But really run a cracker. Try it out.
  - Zurrrrr 4y ago
    The example I gave isn't that guessable. It's combining things from different popculture into one new phrase that won't be in any dictionary. An even better attempt is to insert a whole word between two sides of a broken up word, e.g. `4mAzonij-0rdan4Npr7nces7`. And yes I use hashcat frequently. Using something like a 20 character password following that formula is going to be fine for most people - it isn't going to be brute forced before someone gets an alert that passwords are compromised and can change it. It's a much better solution IMO than using something entirely random and writing it down, or using something like LastPass.

regnull 4y ago

Yes, email has evolved from an open standard to a closed system dominated by only a few big corporations. I don't see how is this situation acceptable - such a fundamental functionality is out of reach to all but a few of us. Well, Gmail and such are free and convenient (until you get banned for no reason), so it's good enough for the most. Trying to get back to the open decentralized ecosystem is a waste of time. But we do it anyway - and we start with the identity layer. Email based on blockchain identity, free and open. I've been working on this for a while, still WIP but check it out: https://ubikom.cc https://ubikom.cc

Analemma_ 4y ago

I understand this guy's frustration but this: > And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem. is just wrong. Tons of spam comes from servers with SPF, DKIM and DMARC now. It stopped being a trustworthy signal of not-spam many years ago.

ed_mercer 4y ago
That explains why I'm starting to receive spam in Outlook again... Guess it's a cat and mouse game.
cge 4y ago
Having SPF and DKIM implemented says little about the email coming from a server. But it should allow filtering of spam from domains based on the history of emails received from them, and how users have interacted with them. They should allow the keeping of trust scores at a domain, rather than server or IP, level, and that should be feasible with even relatively few emails involved. Yet Google and Microsoft don't do that. I can send replies on a personal domain and server to people I've talked with often in the past, and quite often they'll go to spam with Gmail. I can send emails to universities running Office365 from the domain of a prominent university, and sometimes they'll go to spam. I've seen Google and Microsoft email at universities have emails from mts-nature.nature.com, Nature's manuscript tracking system, something no academic would want sent to spam, get sent to spam. SPF and DKIM should allow these things to be rather simply avoidable. But they don't.
billpg 4y ago
SPF et al are not anti-spam but anti-impersonation.

favsq 4y ago

[flagged]

CodeWriter23 4y ago

An open system right up until your users use the junk button as the delete button.

foxylad 4y ago
We are registered with Google's spam feedback loop, and Microsofts SNDS, so we get informed about any messages marked as spam. We get maybe one a week, so this isn't what triggers the rate-limiting.

jijji 4y ago

i wound up developing a pretty complex email sending infrastructure because of situations like this. it involves putting a message in a queue and then sending to a list of predefined proxies (mailjet, mailgun,sendgrid, etc, etc) and using backoff algorithms per proxy per hour....

civilized 4y ago

Meanwhile Gmail happily lets through any scam email from any hijacked account on outlook.com.

throwawaaarrgh 4y ago

Heretical opinion: the government does a good job with postal mail. Let them handle electronic mail. Most of the problems of spam have to do with who's an approved sender and who's abusive. Regulate the mail, it's much harder to be abusive. You're in this country and sending spam? We arrest you. You're in another country? We rate limit your mail, report you to foreign authorities, and flag anything that looks like your mail. You can burn me at the stake now.

35208654 4y ago
I think there's room here to have a 'public option' open source implementation with appropriate governance / rules. Give everyone a basic 10 digit @ email.usa.gov. But I also think there should be a government regulated digital 'town square' for each level of government, with township / county being optional but the rest funded by taxes. Then have those groups determine what is allowed and what's not, through people they nominate in elections. This is like the 'public option' for Facebook. But mostly I just think this because I want more tax dollars to go to open source projects trying to solve problems that are very real and very much caused by the incentives of private industry.
theamk 4y ago
If I am a user, there is no way I am entering my government-issued ID into your website (unless it deals with money) If I am a website owner, no way I am going to use regulated service that will bring police to my door if I make a bug in my script, unless I have to.
soraminazuki 4y ago
How about no? Democracies have no business making it more efficient for the government to hoard or sift through emails.
- EVa5I7bHFq9mnYK 4y ago
  Sure they have pretty efficient arrangements with gmail and outlook right now.

zinekeller 4y ago

Okay, instead of taking this at face value (with a big acknowledgment that it is genuinely frustrating that both Google and Microsoft don't have good contacts to just know what went wrong), I'll analyse this carefully. This website is operated by Virtual Industries Group (https://www.vig.co.nz/ https://www.vig.co.nz/), which is according to their website operates three services: School Bookings/School Interviews (https://www.schoolinterviews.co.nz/ https://www.schoolinterviews.co.nz/, focused on school scheduling and the service discussed here), Care Bookings (https://www.carebookings.co.nz/ https://www.carebookings.co.nz/, which is the same service as above but focuses on day-care and other similar functions), and MessageMyWay (https://www.messagemyway.com/ https://www.messagemyway.com/, which according to their website "is the communications hub for your community. It is your emergency communications plan, your telephone tree, and your email broadcast system all in one"). From a cursory glance, these three services shares this set of outbound MX servers. While it is very unlikely that someone who uses School Interview and Care Bookings would mark their message as spam, if the messages relayed by MessageMyWay is sent on the same outbound MXes then I could immediately see the problem. A large part of MX operators knows this and separates "marketing" and "operational" messages into separate servers to prevent this exact thing from happening. While I understand this dev's frustration, maybe the messages relayed by MessageMyWay are the ones marked as spam by frustrated parents who are receiving irrelevant school marketing which are sent to same MXes as their important operational messages? If I were the developer (and still insist on using on-prem email), I'll operate three groups of servers: Set A: purely for company-initiated messages, never for the customers Set B: "operational" customer-initiated messages: School Bookings, Care Bookings, and MessageMyWay mails which are marked "critical" by the users (which is apparently additional cost) Set C: "marketing" customer-initiated messages: "normal" MessageMyWay mails

foxylad 4y ago
I take my hat off to you - an impressive amount of research went into this! MessageMyWay is opt-in and has a robust opt-out mechanism - and is sadly moribund. It sends a maximum of a couple of hundred emails a day. And we are registered with GMail spam feedback loop and Microsoft's SNDS, both of which tell us when someone marks a message as spam. This happens less than once a week, so this isn't the signal that triggers the rate-limiting.

pentagrama 4y ago

I have a more positive experience than the OP, I worked with a news outlet that has several newsletters and sent about 200K emails/day, and we use our own servers/domain and an open source/self-hosted tool to send them (I think it was Mailman [1]). We did all those technical bits like SPF and DKIM, put the one click unsubscribe link on the message and also on the header of the message so that clients like Gmail can put the unsubscribe link on their own UI [2], all the recommended practices. I remember using the tool Mail Tester [3] and the results where all green. We doesn't have issues with being marked as SPAM by Gmail/Outlook, and have average open rates of 50%, which is a lot higher than the industry standard who is around 20% [4]. We have a good UX and an ethical way to treat our users, like all users have to opt-in to their desired newsletters when creating an account (or choose to receive newsletters without creating an account), one-click unsubscribe link in big text at the bottom of each newsletter, but also a one-click-no-need-to-be-logged-in link to opt-out of all the newsletters that the user was subscribed, and more stuff like that. But one thing that I think give us a lot of reputation for the Gmail algorithm, it was that we designed a feature that if the user doesn't opened a newseltter for about 3 months, we started to send the newsletters with an alert at the top saying like "Seems that you aren't reading this newsletter anymore, you will be automatically unsubscribed in 30[n counter] days. Click here to disable the auto-unsuscribe." (the disable the auto-unsuscribe also was for people who have images disabled and we can't track the openings, but is small percentage). So with that feature, we get sure that our users where engaged with the newsletters, and we have a system to avoid sending messages to "dead" emails, maintaining a fresh and healthy database of emails, and seems that Gmail/Outlook knows and likes that. [1] https://www.list.org/ https://www.list.org/ [2] https://www.sendinblue.com/blog/list-unsubscribe-header/ https://www.sendinblue.com/blog/list-unsubscribe-header/ [3] https://www.mail-tester.com/ https://www.mail-tester.com/ [4] https://mailchimp.com/resources/email-marketing-benchmarks/ https://mailchimp.com/resources/email-marketing-benchmarks/

kilburn 4y ago
It is important to point out the several unusual things that went right in this story: 1. Your decision makers recognized that the important number to pursue is "engaged users", not "number of emails in the list". 2. You have a large enough subscriber base so that Gmail/Outlook realize you're a good player. This wouldn't happen if you only sent a few thousand mails per week. 3. You have the technical expertise to properly maintain the list and the surrounding (unsubscribe, etc.) infrastructure. Usually the budget is allocated to _create_ the list/service and then assumed that it will continue working forever with 0 investment. IF you can get these things right too you'll (eventually) have a similar experience. If you fail on any of them... you're screwed like everyone else is saying. Of course, having to send large volumes of email to ensure that "the big guys" treat you fairly is why many people running small/personal email servers complains about them.

foxylad 4y ago

OP here. For those thinking that Gmail MUST have valid reasons, and that we abusing the system in some way, consider... 1. We are registered with Gmail, Microsoft and Yahoo to provide feedback when someone marks our messages as spam (we DO do everything right). We get reports less than once a week, so if I've got my maths right, less than 0.002% of recipients think we're spammers. 1. Our two servers send around 10,000 messages a day, spread fairly evenly over time. That works out at three or four messages per server per minute. If we were spammers, we'd be ashamed of ourselves.

rkagerer 4y ago

Could their behavior towards legitimate email from small-time outfits like this be useful evidence in an anti-compete lawsuit? Alternatively, some kind of class action? (Asking any lawyers in the room)

studentik 4y ago

Why is server spam filtering even needed with Junk/NotJunk feature in Thunderbird?

Proven 4y ago
[dead]
beej71 4y ago
It's what I use, but it means I have to have Thunderbird running all the time or I see all the spam on my phone. It's just kind of kludgey compared to a server-based solution.

deleted 4y ago

[deleted]

giomasce 4y ago

The promise behind SPF, DKIM and DMARC is that they allowed positive reputation mechanisms, which cannot work if any domain is easy to spoof. So if you're careful, only send good email and stop sending when the recipient asks, that should be a golden star on your review, destination domains will happily let your email through and nobody will be able to freeride on your good reputation because your domain is authenticated. However, this is still hard to establish, because it depends on the destination domain server to decide whether you're sending good stuff or not. I would like to have a mechanism by which the user can decide. I imagine something like this: each email provider, say Gmail, issues to its users a number of single use codes like "Sor7xeik". When the user wants to subscribe to a newsletter (say news@interesting.com) it gives its own email address and one of those codes. The first email from news@interesting.com contains some header like Authorized-Sender: authorize Sor7xeik When Gmail receives it, scratches the code and marks @interesting.com as an authorized domain. From that point on, all (DMARC validated) emails from @interesting.com having some header like Authorized-Sender: yes are deemed to be interesting for that specific user, and accepted without further spam filtering. The user can revoke the consent at any time on Gmail's web interface, at which point emails from @interesting.com (with that header) will be rejected. The sender at @interesting.com will see the rejection and disable mail sending for that user. With this mechanism bad practices like address harvesting and selling become much less useful (because an address alone is not that useful, if the sender is not authorized; and the authorization must be initiated by the user). BTW, I am not saying that all emails should be sent with this authorized sender mechanism. I don't expect individual users to collect authorizations for each of their contacts. Email without the Authorized-Sender header would still be subject to the usual spam filtering, but agencies that often send legitimate mass emails can have a mechanism to prove that they're doing it with the user authorization.

illiac786 4y ago

This is why Proton is extremely aggressive with its free users and close accounts seemingly arbitrarily. Any abuse of their service could cause them to be banned because they are small. Google doesn't have to care, who is going to ban Google SMTP servers? That would be suicide.

rkwasny 4y ago

I am running my own server @entropy.be for last 20 years. Maybe I'm lucky but it just works? The secret is only to have trusted users that do not send spam. OK I get the occasional spam through spamassasin, but I can live with archiving 5 spam emails/day.

deafpolygon 4y ago

killing? More like murdered.

93po 4y ago

I tried to move off fastmail back to gmail last year because I lost a job opportunity due to a random fastmail-origin email i sent getting blocked by my interviewer's google mail. the first three went through fine, it was only the fourth one that didn't make it. they thought i ghosted and had already chosen another candidate because of it. ultimately i didn't make the move because email is unusable without custom aliases i can completely bounce email from. but i think i might attempt again, but again I recently found myself insanely paranoid about my interviewing situation and had to make the awkward transitioning to my gmail account and explain why to my interviewer

rediguanayum 4y ago

Disclosure: I work in Gmail. Likely there's a reputation issue on those IPs. Consider filling out a form to investigate that: https://support.google.com/mail/contact/gmail_bulk_sender_escalation?visit_id=638158923865383305-3828756122&rd=1 https://support.google.com/mail/contact/gmail_bulk_sender_es...

foxylad 4y ago
OP here: thanks for poking your head over the parapet. But we've submitted that form three times over five months, with no discernible effect. I have wondered if a pattern-matching algorithm is penalising us for having a similar IP address to a spammer. But given that practically all servers are now located at hosting companies, IP address is a completely valueless signal - and an arbitrarily damaging one. We have no control over other servers near us in IP space, and can't even ask our hosting company to deal with the spammer because we have no idea which IP address they are on. If you have access to an internal directory, could I prevail on you to pass this concern on to the Gmail spam filter team? I think there is a genuine bug here, and if they fix it and let me know, I promise to post a followup article saying that fixing this shows GMail isn't killing independant email on purpose.