11 ms·
An email list’s open rate drops over 50% after changes to Gmail spam filtering
- IanCal 9y agoGoogle sends me reports on the spam we get to one of our mailing lists. I hadn't seem them in a while and found that they themselves were being sent to the spam folder. edit - I'd forgotten about this actually, and checked, it happened again today. It's apparently happening because people send me spam, so I mark it as spam. Because I've marked things sent to that email address as spam, they're saying their own report is now spam. The description of the problem is sounding more python-esque as I continue so I'll stop there.
- charred_toast 9y agoCan confirm — Google's own Spam reports are going to the Spam folder. Was it the word Spam that triggered it? lol.
- nilsocket 9y agoIf Google wasn't able to find the problem, then they probably should rollback. They can't hurt their customers this way.
- snuxoll 9y agoYou say that like Google cares, over the years they've amassed a large swatch of consumer email users (probably due to Android) - they have basically gotten to a position where they set the rules and if it hurts you tough luck. Email deliverability is a hard problem these days, spam has caused us so much grief that large email providers are doing really annoying things in their spam filters to combat the problem but I've noticed more and more legitimate email is getting caught in the crossfire. Unfortunately, I don't see this being fixed anytime soon - and it's personally making my life increasingly difficult as someone who had to give up self-hosting his email and switching to a hosted Zimbra service just to stop getting flagged by Google as spam (even though I passed DMARC/DKIM and SPF validation and had a non-blacklisted IP).
- kuschku 9y ago> as someone who had to give up self-hosting his email and switching to a hosted Zimbra service just to stop getting flagged by Google as spam (even though I passed DMARC/DKIM and SPF validation and had a non-blacklisted IP). There is a trick how you can get out of the spamfilter: Send mail, regularly (like a few mails per week) to friends/family that use Googlemail, and tell them to mark your mail as "not spam". After roughly a month, you’ll have gathered enough points that Googlemail won’t mark you as spam anymore.
- AznHisoka 9y agoThe people sending these newsletters aren't even their customers, for the most part. They're customers of AWeber, Mailchimp, etc. And for the people receiving these emails? This is a nuisance at best. Although if 99% of the population stopped receiving emails from their friends, etc, THEN this would be a bigger mainstream issue. Right now? Nobody's worrying (especially in Google right now) except a few newsletter senders.
- kazinator 9y agoYou mean their product? People don't pay for gmail, so they're not customers. If you can't figure out what is being sold in the situation to keep the business afloat, what is being sold is you.
- mikl 9y agoWhen I saw the friend was using Mailchimp, it made a lot more sense. I get lots of spam from Mailchimp (ie. people subscribing me to their newsletter, without my consent), and complaining to Mailchimp about it seems to have no effect. So since Mailchimp allows bad actors to send mail with their service, it’s not entirely unfair that Mailchimp mail gets flagged as spam.
- gk1 9y agoThe author uses Aweber, not Mailchimp. Besides, Mailchimp is quite strict with their opt-in policy and they don't hesitate to suspend accounts that ignore it.
- tyingq 9y agoOne of the author's peers uses MailChimp and saw the drop. https://detailed.com/wp-content/uploads/2017/07/paul-jarvis.jpg https://detailed.com/wp-content/uploads/2017/07/paul-jarvis..... That's in the article. "Did you read the post"...that always shuts down discussion.
- gk1 9y agoYou're right, it's unnecessarily hostile. I edited that out.
- mikl 9y ago> The author uses Aweber, not Mailchimp. Sorry, I wasn’t aware that was a mail service, my bad. His friend (from the iMessage convo) was using Mailchimp, though. > Mailchimp is quite strict with their opt-in policy and they don't hesitate to suspend accounts that ignore it. They might say they’re strict, but almost any time I give my e-mail address some where online, I’m automatically signed up for some Mailchimp newsletter, and Mailchimp can’t even be bothered to reply if you report it.
- chinathrow 9y agoMailchimp does not care at all. I unsubscribe daily for lists never subscribed to. They come back and spam again...
- creo 9y agoI bought electronic content that was sent to me via signed-email and it got into spam ... somethings not right.
- danvoell 9y agoGoogle could get really evil in this regard. But then again Facebook has been doing this for years. "We didn't think the content was relevant to this audience. Pay us some money and we will allow your email to reach more people..."
- AznHisoka 9y agoThis is already the case today. I get so much irrelevant mail in my main inbox that I ignore a lot of legitimate email in the process. And you can buy Gmail ads which appear in the top of your inbox.
- dwringer 9y agoI've had emails from family members go straight to gmail's spam folder on two different occasions, and a third occasion in which the mail appeared to have sent according to the sender's mailserver, but never showed up anywhere in my gmail account. From what I could tell glancing around some forum posts, mine was hardly a unique experience and is reliably solved (well, in the first two cases) only by explicitly whitelisting every single address from which you expect to receive correspondence. I'm glad to see more attention to this issue of overzealous filtering.
- threepipeproblm 9y agoI only skimmed this so forgive me if I missed something. But could this simply be what happens to you when a sufficient number of email recipients classify you as spam, and you then reach some critical threshold? >> I changed my sender email address to see if would help But if the pattern of content got flagged as spam by Google users, the sender address might not be the only determinant. >> I was relieved to discover that others with newsletters were having similar troubles very recently. But if this results from crossing some sort of user-classified spam threshold, one would probably always be able to find some recent "victims". This is not intended as a criticism -- I'm sensitive to Google's seemingly arbitrary power -- just an idea to consider or rule out.
- ivanbakel 9y agoThat seems an unlikely thing to have happen to multiple high-profile newsletters at once in a short span of time. Particularly, if these people aren't seeing big changes in their users, why would the spam threshold be hit now? Did a large chunk of email readers suddenly decide enough is enough?
- threepipeproblm 9y agoWhat if it happens to some high-profile newsletters over every/most any short span of time you might pick?
- ivanbakel 9y agoThen you'd expect this article to appear on the FP every week, surely? It seems more likely to me that whenever people write about something out of the ordinary, it would not happen ordinarily - otherwise there wouldn't be anything to write about.
- threepipeproblm 9y agoNo you'd expect people to be complaining about it somewhere at some frequencly...and it hasn't been established that isn't the case. However, if you search for "Google newsletter suddenly spam folder", you can easily find lots posts that are superficially similar-looking, from various time periods.
- victorNicollet 9y agoThe spam problem has never been solved, no matter how good spam filters may have become in the recent years. It's still an arms race, with a front that is moving one way or another every few months. Even today, a spam filter is a piece of software that you trust to throw away e-mails without letting you read them, based on its own decision that you don't want to read them. It seems insane to me that people would willingly let a program decide to throw away inbound mail based on unspecified rules. And I nearly start frothing at the mouth when someone suggests that, since their software decided to throw away my e-mail, it is somehow my responsibility to fix that. It isn't. If there were thousands of different anti-spam solutions, and one of them behaved badly, then senders wouldn't care, and users would vote with their feet (or wallet, or app store) until the behaviour was fixed. But since Google has a near-monopoly on e-mail, it's now up to the senders to do the work and jump through the badly-documented ever-changing hoops.
- draw_down 9y agoDo they really have a "near monopoly" on email? How near are we talking here?
- mikl 9y agoIn the younger, private mail account demographic, I’ve seen as much as 80% of some e-mail lists being Gmail accounts. For business, Exchange is still king, but Gmail is gaining on them.
- saul_goodman 9y agoIt's scarey stupid how much the "big 3" own email now. Google, Yahoo, Hotmail. When any of them change up their spam fighting techniques it causes ripple effects across the entire email ecosystem now. Have Yahoo mail subscribers on a Mailman mailing list? Not if you're not running the latest Mailman which has a new workaround just for them. Yahoo started bouncing legitimate mail a couple years ago due to the normal way mailing lists have ALWAYS handled sender addresses, since before Yahoo existed. I get it, sometimes desparate times call for desparate measures (for fighting spam). But jeeze, it's super annoying to have to break your package manager to get an upstream mail server to support one user who uses one certain mail hosting service.
- alyandon 9y agoThe hilarious part about Google's spam filtering is that I have crons that send logs in emails to my own address using Google's own smtp servers (that require authentication with my own credentials) and they end up getting placed in spam.
- paulsutter 9y agoThat's hilarious: A newsletter about how-to-spam is treated as spammy and the author is upset. In the end, everyone thinks they're one of the good guys. > The aim of this blog is to help anyone with a website find proven content ideas that they can use to drive more traffic and links to their own sites. The concept is already working, with one recent article helping a reader get on the front page of Product Hunt.
- SamUK96 9y agoThis is instantly what I thought too. Essentially this guy is teaching how to get rich by using SEO, catchy gimmicks/ideas, and the likes. The spam teacher themselves get filtered, how ironic.
- martokus 9y agoYes, welcome to the world of Viperchill - the shadow king of get rich by teaching others how to get rich schemes. I wouldn't be surprised if Google personally targeted him, I would do :)
- viperchill 9y agoSadly this is happening to a lot more people than just me otherwise I would not have written about it.
- Bartweiss 9y agoHeh - I wouldn't be surprised if the language used sets off the filter. Machine learning isn't great at abstractions like "writing spam" versus "writing samples of spam someone else might write", so I imagine this sort of topic is particularly prone to filtering.
- viperchill 9y agoAbout how to spam? Lol, that's quite the jump you made there. If you read the article, you'll find there are many other industries which (suddenly) have the same issue.
- 9y ago
- ryandrake 9y agoAs much as I'd like to reduce my reliance on "cloud" services, Gmail is one of those that I just can't quit, since they've pretty much solved the spam problem for me. I'd rather a few false positives get filtered than the deluge that comes with inadequate (self-hosted) E-mail. Sorry if this guy's "audience building" business is affected, but as an end user not worried about "my open rate", I appreciate Google's aggressive filtering and urge them not to let up.
- mikl 9y agoThat’s fine for you, but believe it or not, there are actually people who sign up for newsletters voluntarily because they want to stay informed about something. Google shouldn’t try to override the user’s preference about receiving newsletters.
- deleted 9y ago[deleted]
- CydeWeys 9y agoMark a few of the messages as important and you won't miss future ones. Given how pervasive getting signed up for email lists is even when I don't really want to be on them, the opt-in model makes more sense for me than the opt-out model.
- rtx 9y agoTrue, but the solution should be double optin not something managed by Google.
- CydeWeys 9y agoMailing lists that aren't double opt-in are way more likely to be managed by Google on behalf of the user than those that are. The spammy mailing lists are way more likely to get spam-filtered by users who were "opted-in" to them by the mailer. Google's solution is necessary because bad actors in email abound.
- deleted 9y ago[deleted]
- raverbashing 9y agoThe core of the issue seems to be an anti-phishing feature that Google seems to have dialed too high in some cases But yeah, communication channels are getting more and more clogged with irrelevant communications (not even span alone) so it's getting tough to get across the barrier
- deleted 9y ago[deleted]
- coding123 9y agoAll the spam I get is newsletters from the perspective of the spammer. F that. My email should be for people.
- eli 9y agoWell, if you didn't sign up for it then it probably is spam. But lots of people want to get newsletters and if you want to get it, then it isn't spam.
- Spivak 9y agoEven if you did sign up for it it's probably spam. That's the real issue here. Given how aggressive businesses are with signing up people for unwanted newsletters it seems completely natural for Google to be suspicious of them.
- ReidZB 9y agoYep. I waste an hour (probably) every week unsubscribing from various newsletters and marketing campaigns that I don't care about. Most corporations have largely taken "this person has given me an email address" -- even if one is necessary to use their service! -- as free license to hit them with all sorts of marketing messages. Really, these are spam -- unsolicited, unwanted email. It's just that "spam" has taken a stronger, more negative meaning.
- zippergz 9y agoI wonder if they've started handing SPF or DMARC misalignment differently. I've noticed a lot of those kinds of problems with newsletter-type mail in the past.
- eli 9y agoHuh, weird. I haven't noticed this and I send a lot of email newsletters.
- kazinator 9y agoGmail is not magically going to white-list recipients for receiving some spam that they opted into on some random website it knows nothing about. People sometimes opt-in in good faith, and then when they get the e-mail, they realize it is garbage. They tell their e-mail client that it is junk, rather than going through the ritual of unsubscribing. Probably, in may cases, they probably don't even make the mental connection between what they received and the site they visited where they checkboxed something; they don't recognize that what they're receiving is something they signed up for. Their reaction is "WTF is this crap?" and report it as spam 500 milliseconds later without giving it another thought. If enough users flag the posting as spam, Gmail will probably learn that and help the remaining users.
- RichardHeart 9y agoGood thing everyone uses gmail right? Centralize everything under one company! Centralize all the things! I use gmail :(
- Nazzareno 9y agoIt's not a Gmail issue! By using an ESP (like Aweber, Mailchimp, etc) means that you are sharing part of your "reputation" with thousands (or even millions) other senders. So even thou most of the ESP are quite efficient in cleaning-up their customer base from spammers, the more clients they have, the more it's difficult. Here you can find an pretty accurate and updated analysis of ESP's clients "quality": http://mainsleaze.spambouncer.org/may-2017-in-spamtraps-esps/ http://mainsleaze.spambouncer.org/may-2017-in-spamtraps-esps... Aweber is #6, which is not a good signal. That said, you may switch ESP by choosing one that provides a you custom "identity" such as dedicated IP, DKIM, PTR, SPF records (which usually means quite a lot of setup and costs) or choose an ESP that doesn't have tons and tons of clients, eventually with a free version. http://www.emailvendorselection.com/ http://www.emailvendorselection.com/ is an updated ESP directory. You are of course also welcomed to try mine: http://www.mailup.com http://www.mailup.com
- mrbunnyman 9y agoAfter reading this I checked my spam folder. Found an email from one of my best friends from highschool, who I've sent 100s of emails back and forth with, letting me know he would be in town in my spam folder... Well guess I didn't want to see him over the holiday weekend. GAAHAHAHHAH. See as how a lot of my friends use their own domains / hosting for email I guess I'll be spending a lot of time in my spam folder.
- rstupek 9y agoWhat's really bad is your friend had no idea you never saw his email because it ends up in a spam folder and not bounced back instead
- davidgerard 9y agotl;dr have your SPF, DMARC and DKIM in perfect order, 'cos Gmail has started filtering hard on those.
- wolco 9y agoGmail is not a good provider if you care about seeing all of your emails all of the time. It does a good job getting rid of spam but if you need to check the spam folder for missed emails then it makes sense to leave the spam in the main inbox and save a few clicks
- II2II 9y agoPerhaps businesses are becoming too agressive at requesting email addresses, people are providing fake email addresses, and the recipients are responding by flagging it as spam? I bring this up because I started receiving suspiciously legitimate spam a few years back. No red flags were raised by the email itself (e.g. no indications of phishing), the companies exist, the recipients exist, everything was geographically consistent. It was as though certain individuals were providing my email address as their own to businesses yet not to friends or colleagues. When you consider how agressive some of these websites are with newsletters, I would not be surprised if people were providing fake email addresses just to get rid of the prompts. I would not be surprised if Google started flagging those newsletters as spam, even if the odd user genuinely wanted them.
- 77pt77 9y agoSome years ago I got a reimbursement from google regarding expenses they were paying me. I though they had totally forgotten about it and contacted them. Even the google representative knew their own financial related emails were systematically being tagged as SPAM on gmail. That's how aggressive it was. I couldn't stop laughing for minutes.
- Ardren 9y agoJust a anecdote, but this month there was a significant phishing attempt emulating an Australian electrical provider which got through Gmail's spam filter. Gmail has now updated their filters to detect it, but it's now catching genuine email as well. That said, it's been a long time since Gmail has had a false positive that I was surprised about (i.e. not a spammy newsletter that I subscribed too)
- deleted 9y ago[deleted]
- jordigh 9y agoThey're measuring "open rate" by putting links or images in the email that have uniquely identifying information, right? As much as I'm frustrated with Gmail's spam filtering (I also use a personal email server, and it's annoying to get my personal email filtered by Gmail), I absolutely loathe seeing that tracking information in links, even if it's from a mailing list I specifically subscribed to. Often times they're newsletters that are also available on the web. I'd rather just go to ddg to search for the web version to read. Of course, that's probably a case of https://xkcd.com/1105/ https://xkcd.com/1105/
- dhekimian 9y agoDETAILED.COM has a bunch of issues that would easily trip SPAM filters. ------------------------------------------------- TL;DR: DNS, SPF, DKIM & DMARC are not properly configured for DETAILED.COM. Fix and test the issues below. ------------------------------------------------- Start here: https://mxtoolbox.com/domain/detailed.com/ https://mxtoolbox.com/domain/detailed.com/ 1. SPF Records Issue: You should only have 1 SPF record but you have 2. Combine them. https://www.spfwizard.net/ https://www.spfwizard.net/ Current: detailed.com. 3600 IN TXT "v=spf1 include:smtp.groovehq.com ~all" detailed.com. 3600 IN TXT "v=spf1 mx include:send.aweber.com -all" Should be: detailed.com. IN TXT "v=spf1 mx include:send.aweber.com include:smtp.groovehq.com ~all" 2. You SOA record is incorrect Your current authoritative DNS servers are: detailed.com. 172800 IN NS ns.sourcedns.com. detailed.com. 172800 IN NS ns1.sourcedns.com. Your current SOA is: detailed.com. 3600 IN SOA ns.liquidweb.com. admin.liquidweb.com. 2017042600 86400 7200 3600000 14400 In your SOA, ns.liquidweb.com is incorrect. It should be ns.sourcedns.com 3. Missing DKIM records Check if AWEBER is sending the correct ones. Use the steps outlined here https://www.phishingscorecard.com/ https://www.phishingscorecard.com/ to find your DKIM key and test to see if it's working properly. 4. Setup DMARC to get reports on your failed email If you had set this up, you would be getting notifications from Gmail, Yahoo, etc as to why your email is not being delivered properly. I suggest doing a 14-day free trial via https://dmarcian.com/ https://dmarcian.com/ Once you create an account, they will provide you with RUA & RUF addresses and process the XML responses for you. Don't forget to add the _DMARC TXT record to your DNS. 5. Test your email using https://www.mail-tester.com/ https://www.mail-tester.com/ This site is fantastic and will help you narrow down on actual delivery issues.
- f4rker 9y agoGoogle aside, plenty of email servers regularly filter or silently drop customer requested emails from well established companies.
- hamstercat 9y agoIn a previous job, my team was responsible (among other things) of the company's newsletter. Nothing great but nothing spammy either, and we made sure to follow anti-spam law by having the newsletter subscription being opt-in and having a clear and easy unsubscribe flow. We didn't send the emails ourselves, but rather used a big player in that area. What I learned is that sending emails and not getting treated as spam by any of the big email providers is an absolute mess of tinkering and hoping it'll work. Not all of them would categorize our emails as spam, but Outlook.com started doing it out of the blue, and even a year later we didn't have a clue as to why or how to make it stop. Support for email senders is as good as for users, meaning getting anything else than a canned response wasn't happening.
- brihas 9y agoI work for a small non-profit and we've had the same issue. We use SendGrid to send out our email newsletters and the open rates have fallen from ~27% to ~13% over the past month. Does anyone here have contacts at Google who might be able to help? [Edit: I've also reached out to SendGrid support to look into this] Looking at the data more, the drop is due to a ~73% drop in Gmail opens. Hotmail and Yahoo opens remain stable: 1) See drop in overall open rate here: https://www.dropbox.com/s/j8grg3r684zshhz/Drop%20in%20open%20rate.png?dl=0 https://www.dropbox.com/s/j8grg3r684zshhz/Drop%20in%20open%2... 2) See drop in Gmail open rate vs. Yahoo and Hotmail here: https://www.dropbox.com/s/0fdvbddwno27mgf/Drop%20in%20Gmail%20opens.png?dl=0 https://www.dropbox.com/s/0fdvbddwno27mgf/Drop%20in%20Gmail%...
- dhekimian 9y agoSince you didn't provide your domain name, I can't troubleshoot for you. I recommend reading my comment https://news.ycombinator.com/item?id=14720344 https://news.ycombinator.com/item?id=14720344 and following the troubleshooting steps I outlined.
- brihas 9y agoWe use the news.pariyatti.org domain for these emails. If you're willing to help troubleshoot, that would be much appreciated. Otherwise I'll try to work through your recommendations, although a lot of it is very new to me so I'm not sure how far I'll be able to get. You can reach me via email at my HN username @pariyatti.org if you wish to discuss further.
- anoetudh3tnh 9y agoI have zero sympathy for these people. My spouse for example often gets added to mailing lists when purchasing a product (just like we all do). Instead of clicking the unsubscribe link, they'll just "open" the email without reading it to mark it as read in their inbox, and immediately go on to the next one. These are essentially spam that my spouse is too lazy to mark as such. I have to wonder how many of the "opens" the author's lists get are just like this. "Oh, that stupid list I got subscribed to somehow. Mark as read." I have a very deep suspicion that they're mailings are essentially spam to a lot of people who receive them and this is correct classification by Google.