9 ms·

Why does Gmail hate my domain?

310 points by stbenjam 11y ago

dasil003 11y ago
I'm very curious about that as well as I am thinking of moving my personal domains away from Gmail and it would really suck to start landing in spam simply on the basis of not being with a major mail provider. I hope this story gets traction and someone on the Gmail team finds it and comments.
ceejayoz 11y ago
If the domain is bitbin.de and the mail server is hosted on the same server, it's probably Hetzner being scorched-earth for deliverability, just like any other major hosting provider will be. Email from AWS, Rackspace, Heroku etc. get a drastically higher starting spam score because people fire up servers there to spam. Running the email through something like Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.
- quadrature 11y ago
  This seems to also be the case with DO, I setup a mail server on an instance there and can't escape people's spam boxes.
  - arrty88 11y ago
    FWIW, I am able to send mail effectively from my Linode box, (passing SPF and DKIM).
    - kijin 11y ago
      Linode has always been very strict about keeping their IP ranges clean. The fact that Linode's lowest tier has been $20/mo until recently ($10/mom nowadays) also helps. That makes Linode unnecessarily expensive for people who just want to burn IPs, since they can easily get an IP for $5/mo with DO or even less with low-end VPS providers.
  - brobinson 11y ago
    Check the IP against the spam blacklists: http://mxtoolbox.com/blacklists.aspx http://mxtoolbox.com/blacklists.aspx You might have to spin up a bunch of servers to find one which doesn't have a blacklisted IP. Then you have to build the IP's "reputation" with the various providers before you can send a significant amount of email through it.
    - quadrature 11y ago
      Thanks, I just assumed their whole block would be blacklisted. I'll try some more.
      - ceejayoz 11y ago
        Not blacklisted, just treated with higher suspicion by default.
    - massaman_yams 11y ago
      These days, there are only a handful of IP-based blacklists where a listing result in more than low single-digit impact to your delivery rate; moreover, among industry insiders, Gmail isn't known to use any public IP-based blacklist for deliverability.
      - brobinson 11y ago
        >Gmail isn't known to use any public IP-based blacklist for deliverability Good to know! I bet they have their own internal ranking system which is far more accurate than the public blacklists.
        stonogo 11y ago
        I'm curious to know why you think they'd be any more accurate than public blacklists. About a fifth of the spam that hits my domains originates from gmail servers, and the majority of people I know complain (when asked) about gmail's tendency to spam-bin legit mail.
  - mattkrea 11y ago
    I ran my personal mail server on DO for about a year or so without issue. I would definitely try a new instance--I didn't even know this was a problem.
- cmwelsh 11y ago
  Amazon SES has a pretty poor set of IPs due to the issue you mentioned about firing up servers to send spam. Mandrill is top notch, just like other professional email providers such as Mailgun and Sendgrid.
  - ceejayoz 11y ago
    Are you thinking of EC2? SES (as with Mandrill, Sendgrid, etc.) runs on a separate block of IPs and is very strict about spam/bounce reports. Very easy to get yourself throttled or cut off entirely.
- jsnell 11y ago
  I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery. It doesn't even have SPF or DKIM set up. And I'm pretty sure that if messages were not getting delivered, users would complain. At least that's exactly what happened the one time there was a configuration screwup [1] that made some email providers reject mail from that server. So I don't know if that theory explains the issue either, though it could be a component. [1] https://www.snellman.net/blog/archive/2014-12-05-how-buying-a-ssl-certificate-broke-my-email-setup/ https://www.snellman.net/blog/archive/2014-12-05-how-buying-...
  - ceejayoz 11y ago
    > I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery. That volume (assuming non-spammy content, of course) would over time build up a positive reputation for your IPs, outbound addresses, and content. A new or low-volume IP on Hetzner is going to have a harder time overcoming the initial suspicion Gmail saddles it with.
    - massaman_yams 11y ago
      That's probably more true in general than it is in this particular case. Gmail does IP reputation differently than other major ISPs, and domain/content reputation are very different beasts entirely. Gmail's approach with a low-reputation IP is generally to throttle it aggressively; mail still gets through, just not very quickly if you're sending in bulk. Domain reputation is not initially distrusted unless it's a brand-new domain (registered in the last 30 days or so), and generally doesn't come into play unless there's very unusually poor metrics (think sky-high spam complaints) associated with the domain. Content reputation is applied in a similar manner. The headers I'm looking at should most likely give Gmail enough distinct identifiers to work with to isolate it from broader reputation issues at Hetzner, but the post only included partial headers, so I can't quite say that conclusively.
    - fuzzix 11y ago
      Always worth checking if the IP you inherit is in some DNSBL. A number of Hetzner IPs I've gotten were tainted. It's oh so much fun trying to get them removed.
- seiji 11y ago
  Amazon SES or Mandrill would probably be helpful. Both have generous free tiers. But you are still sending all your outgoing email through an external 3rd party. Not ideal. Also, these days every email is a top-reply orgy of all correspondence ever sent, so your SMTP relay essentially gets a copy of your entire conversational inbox (via "QUOTE ALL THE THINGS" reply garbage) as well as your outbox.
- drzaiusapelord 11y ago
  While I don't doubt this, I often see from DIY types some really lazy smtp habits. If they don't have their reverse DNS matching the forward DNS as well as the HELO name matching that as well, then, yes, that will be a major red flag for anti-spam systems. Reverse DNS is pretty important and weighed heavily. smtp delivery is something of a black art, but at the very least those things need to be set correctly. Its also worth mentioning that if you have 'spammy' applications that opt people in to mailings then a lot of people just get pissed off and click the 'junk' button over and over instead of unsubscribing properly. So think twice before auto-subing customers or enabling 'mail me replies' by default on forums and such. I also learned the hard way that once you're on Cisco Ironport's spam list, then you really can't get off. There's no one to talk to. Its supposed to be automated. In practice, that means months before your domain is unblocked. I find most Ironport admins just use that sole list even while Cisco recommends they use many and weigh the average. After a stupid config change that opened our smtp to relaying, we got off everyone's blacklist but IronPort. IronPort is pretty much the defacto standard in several types of enterprise. My fix? Get a different IP address for the mail server.
slg 11y ago
The problem might be the IP address of the mail server and not the domain. Where are you hosting the server? Is it is your personal server? If so, what is the ISP and is it a residential connection or a business connection? If it is hosted by someone else, what is their reputation as a source of spam.
runin2k1 11y ago
That's all a bit presumptive and inflammatory for what amounts to pure speculation on the part of the author. Google stands to lose a lot more from a potential PR disaster for burning former customers who move away from hosting than they do from trying to convert a tiny portion of users to a free mail hosting service.
- MichaelCrawford 11y ago
  My own cousin blackholed my gmail because gmail sends so mych spam. I had to ask him to configure an exception just for me. If you dont want to receive spam use lafn.org. I dont know how one gets an account there but I expect it doesnt cost anything. If you want a full VPS use http://prgmr.com/ http://prgmr.com/ - "we dont assume you are stupid.". Its a hosting service operated by neckbeards for the benefit of their fellow neckbeards. Quite cheap, I get mine for free because I help them with their marketing.
  - kbar13 11y ago
    i got mine for writing a tutorial for setting up archlinux on prgmr on their wiki. :D
  - alanpost 11y ago
    I run a personal mail server there too. I'm very happy with them: I was in the free tier for years, after they acquired the hosting provider I had been using. I recently asked them to start billing and am very happy to be a paying customer.
  - aaronem 11y ago
    If you run your own MDA, you could do a lot worse than to filter incoming mail against https://www.spamhaus.org/zen/; https://www.spamhaus.org/zen/; the free tier is extremely broad, and the coverage is excellent -- I get maybe one spam email a week, tops.
    - _delirium 11y ago
      I'm also happy with Spamhaus Zen generally, but spam sent via Gmail is one category of email it won't stop, since it only blacklists pure spam sources, and Gmail is obviously a legitimate mailhost. I get a steady trickle of spam from Gmail addresses, mostly SEO-consultant spammers who grabbed my email from the whois info.
petercooper 11y ago
I send a LOT of emails each month (email newsletter business - yes, legit!) and ran into an separate but topically related and amusing problem recently. My newsletters are aimed at developers, and one issue went out and was considered by Gmail to be a 'phishing' attempt. I couldn't figure it out. Several issues later, another one was picked up the same way and I figured it out.. In both issues, one of the items was linking to domains that looked a bit like this "www.0x10abcdef.com" (this is NOT the actual domain) - basically a domain that looked like a hexadecimal number. I ran numerous tests and Gmail always considered mails with links to domains like this to be phishing attempts. I reported this as a bug (since nothing was wrong or reported with the domains in question, it was basically Gmail's filter being in error) but no idea if it was ever resolved.
- brobinson 11y ago
  It's probably specifically penalizing anything matching /https?:\/\/0/ because you can specify an IP address using hex or octal in most common browsers. hex: http://0xD83AC02E http://0xD83AC02E octal: http://033016540056 http://033016540056 That's my guess, at any rate.
  - petercooper 11y ago
    Yes, URLs like that are commonly used in phishing emails, so that's my hunch as well. They're just not recognizing that such strings can appear in domains so I suspect it's a simple bug but probably not one they'll fix.
  - hobarrera 11y ago
    OT: What does the red-flower-on-the-sun doodle mean?
    - deadbeef404 11y ago
      Argentina National Day 2015 Edit: You can usually find out what they mean by going to https://www.google.com/doodles https://www.google.com/doodles
- 11y ago
bitJericho 11y ago
Seems like author hasn't done enough troubleshooting and has jumped to a conclusion. My recommendation is to run through google's troubleshooting steps on the issue: https://support.google.com/mail/troubleshooter/2920052?hl=en https://support.google.com/mail/troubleshooter/2920052?hl=en Also make sure a DMARC record is setup.
- gknoy 11y ago
  At the end of that, it asks about whether your server passes "the diagnostic tests" -- but I didn't see a link to which tests it was talking about.
  - blfr 11y ago
    It's MXToolBox[1]. To be fair to OP, it's very popular. [1] http://mxtoolbox.com/diagnostic.aspx http://mxtoolbox.com/diagnostic.aspx
- stbenjam 11y ago
  I have been through this, and the end result is a contact form that no one responds to. I filled it out a few months ago and again this week. DMARC is enabled (and in one of the screenshots, Google shows it passing). I get reports from them, but it provides no insight into their decisions.
arihant 11y ago
"The needs of the many outweigh the needs of the few." -- Spock. I'd rather have a few geeky people like me not have the fun of self hosting an e-mail server, than waste millions of man hours round the world because of people dealing with spam. If you must hack, however, use something like Mailgun. It is more hacky in a way that you can program your incoming mail the way you want, not just install a mail server with a few commands.
- techsupporter 11y ago
  That...doesn't make any sense. The author doesn't appear to be spamming or doing any kind of mailing list traffic. He is an individual communicating with other individuals. That's kind of the whole point of this Internet thing we have going here. Besides, he's only being singled out by Google; no other providers seem to have a problem with his domain and Google's own headers show that everything is fine.
  - pekk 11y ago
    It's a coherent opinion which makes sense, even if we don't agree with it. The claim isn't that the author is spamming, the claim is that the availability of effective spam filtering ("needs of the many") trades off against and ultimately outweighs the author's reasons for running his own small web server for his personal email ("needs of the few").
    - hueving 11y ago
      No, the opinion is not even logically valid because there is nothing that shows Google's punishment of his server is required to reduce spam. It's like eliminating poverty by killing all poor people. Yes, it would work, but the side effects are worse than the outcome.
      - pekk 11y ago
        Logical validity only means that the argument works given its premises, not that the premises are correct.
thrownaway2424 11y ago
The author jumps to an unfounded conclusion which is pretty irritating and probably will make everybody who could help him not want to help him. That said I've never seen a DSN like the one in the screenshot. It certainly is not generated by the gmail spam checking system, because gmail does not bounce spam. Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.
- stbenjam 11y ago
  > The author jumps to an unfounded conclusion which is pretty irritating and probably will make everybody who could help him not want to help him. Apologies if you took it as a conclusion, it was only hand waving speculation, but google does have an incentive to make it difficult for the little guy... > Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders. They do if you have DMARC enabled, and a REJECT policy, although they still seem to ignore it sometimes and I have no clue my mail is sitting in a spam box.
- massaman_yams 11y ago
  It's entirely true that Gmail accepts mail at much higher rates than other major ISPs, and that their default handling of unwanted mail is to place it into the spam folder - but the truth is they can and will bounce mail if it's bad enough. It's just extremely uncommon for most people - even email marketers - to ever run into a situation where that happens.
- hobarrera 11y ago
  > Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders. This is a million times better. Meanwhile, microsoft accepts it, sends it at passed on DMARC reports, but discards the mail silently without it even reaching the spam mailbox.
lm741 11y ago
I hit a similar problems when sending automated internal emails to a Google Groups address at my company. The problem was fixed by adding the following footer: "To unsubscribe, email <my-email-address>."
- spinlock 11y ago
  Uh oh. I created an "unsubscribe" filter a while ago to get rid of spam. It's been great for keeping my inbox clean but I just don't see stuff with an unsubscribe line anymore.
  - Lorento 11y ago
    It's not a good idea to manually create filters that 100% block. You can't think of all the special cases like this.
    - raverbashing 11y ago
      Exactly, especially based on an not-so-unique word Doing a quick search on my inbox, there are some relevant emails containing the word 'unsubscribe'
    - spinlock 11y ago
      I just filter it into a separate folder and skip the inbox but I've definitely missed some emails because of it.
- danieltillett 11y ago
  Wow I have to try this trick. I have endless problems with some emails going missing (hotmail and the rest of microsoft’s domains are the worst).
logicallee 11y ago
>What can I do except move to a hosted provider ? To answer your question about what you can do, you can send mail to Google legal asking them to accept your mail gmail users - just copy your whole blog post, I'm sure they'll action it right away: Antitrust is serious business. (They already have trouble in the EU for it - https://www.google.com/search?q=antitrust+eu+google https://www.google.com/search?q=antitrust+eu+google - by the way I think it's completely unfounded.) It takes them seconds to whitelist you, and this really is a "15 seconds could save you $15 million on your next antitrust case" and anyone in Google legal can probably see that. EDIT: I don't see how this got downvoted. This literally answers OP's question about what else he can do.
- chc 11y ago
  Antitrust is serious business, but seeing as there is no clear antitrust issue here, I'm not sure what you think getting lawyers involved will accomplish other than making it impossible to help you since your case is now an impending legal action.
  - deleted 11y ago
    [deleted]
    - chc 11y ago
      > What do you mean there is "no clear antitrust issue" here? Most of OP's contacts use gmail, but gmail refuses his legitimate outside mail. But Google does accept almost everybody else's email, including many actual commercial competitors, and there is no reason to believe anyone at Google is deliberating singling OP out or even knows who OP is, so I can't imagine where you're seeing anticompetitive behavior. > Since email is an open protocol, and since his mail server is easy to verify and sends no spam (as he outlines at long length in his blog post), this really is quite blatant. Imagine if, early in the millennium, Internet Explorer refused to connect over http with servers that weren't IIS. This is more like if Internet Explorer failed to connect to some random Joe's homebrew personal server but could connect to most other people's just fine whether or not they were using IIS. That's not anticompetitive behavior, it's just a bug, and in fact there were many things that Internet Explorer did not handle correctly.
tedunangst 11y ago
Spam filtering/tagging is more annoying than outright bans. Microsoft banned my IP from their email services, but helpfully sent bounces so I could appeal. This is the email equivalent of hellbanning.
_cbdev 11y ago
I've had mail sent via my own server rejected by gmail because of a missing Message-ID header. The 550 reject message was the standard "Unsolicited Mail detected" text, the same mail was accepted without causing any fuzz once the Message-ID was added.
- baudehlo 11y ago
  Why on earth wouldn't your mail server add that for you?
  - _cbdev 11y ago
    Because I'm writing my own ;) https://github.com/cmail-mta/cmail https://github.com/cmail-mta/cmail if you're interested.
jmount 11y ago
Network effect. Google blocks non-GMail email, eventually you are forced to use them as your service provider. Can either be deliberate or happy unexpected outcome for Google.
- rebelde 11y ago
  I use Google for my email, and they still count some as spam. Hell, it is sent from Google's own servers! It still doesn't help.
raintrees 11y ago
Anecdotal information: I run my own mail server and manage a number of small business clients who have mail servers. Email trust is getting more and more tedious. Recently I was resolving a domain registrar issue with Network Solutions. They required forms filled out and signed, a copy of a utility bill from my client, a copy of my ID... I bundled up everything and emailed them the scans. I contacted them 5 business days later, they claim to never have received it. I sent it again while I had a rep on the phone, it went into their spam hole, probably due to size of attachment. They helpfully suggested I get a GMail account to send the same message. They are my registrar, they host my DNS, including my MX record. I have an spf record... I thought it was pretty farcical, and a sad statement of digital trust/authenticity. Some of my clients are giving up and just going with the flow, I have had several conversions to Google/Microsoft cloud-hosted solutions for email...
- arihant 11y ago
  An spf record only helps preventing emails sent from phishing servers. It is inherently meaningless to spam filters if the trust with the domain owner itself is misplaced.
  - blfr 11y ago
    True but if they're raintrees' registrar, they should already be able to connect the domain with the person/company who registered it.
codezero 11y ago
I've had the same problem myself. My best guesses as to why my domain has been dinged: 1) It's on a VPS, that IP may be flagged already 2) I often use a VPN, and sometimes send emails out through my server using it. This probably raises the red flags. 3) It's a non-standard TLD, .co 4) I don't use any Google services with that email address/domain (I assume doing so adds some level of measurable trust) 5) I'm not in the address book of a lot of gmail users, because this is my private server, that I use only for job seeking, and personal communication. This has really done damage in the past. I've applied for jobs, and heard back weeks after they hired someone, letting me know, woops, I ended up in their spam folder. Trying to meet with a friend? I can't email them, I need to use Facebook or Gmail... welp. This is super annoying. I kind of understand why it happens, but it's just a little sad that building your own fort, so to speak, is so impossible. Things I tried to do to mitigate this: 1) Made my web domain https only (why not?) 2) set up DKIM and SPF (didn't seem to have any effect) 3) proper SMTP authentication, secure port only 4) Reached out to Google via the typical forms they offer, and heard nothing, obviously.
Animats 11y ago
Notice the comment on the story: "I had the exact same problems in the past. For me it helped just to register and immediately cancel a trial of Google Apps for the domain. It’s annoying to have this as a necessary step but at least it was like that done in just a few minutes (after I tried to find a contact email of Google for a much longer time). I tried it because I had other domains on the same server which didn’t had this issue. All domains which weren’t filtered had a Google Apps account in the past. So I thought it’s worth a try, and yeah it was solved."
- stbenjam 11y ago
  Thanks, may have to pay Google the $5 (or thereabouts?) to host my mail for a month and consider it a cost of freedom from them long term. After 2+ years of hosting my domain myself, it doesn't seem possible to build up enough reputation with them personally.
  - massaman_yams 11y ago
    If the Google Apps anecdote is true, it's likely to be a design flaw in their anti-spam systems - think of the number of spammers who could register domains, associate them with a $5 google apps account, and spam millions of Gmail recipients. If you think about the kind of metrics/outcomes the Gmail anti-spam team uses to measure success, it's hard to imagine a scenario where that would be good for them. What this looks like to me is a reputation problem with one or more domains, either in your headers or your message content. From a domain reputation perspective, you don't generally need to 'build' reputation in the same way you often do with IP reputation; rather, Gmail tends to tag domains with negative reputation only when they've observed unusually bad behavior associated with the domain. Sometimes this can happen due to domain forgery or spoofing. Sometimes it's because a sender is doing overly aggressive email marketing. Sometimes it's just because your domain shows up in someone else's bulk email. With full authentication, spoofing is unlikely; I also don't see any indication you're doing email marketing here, and your IP 88.198.5.136 doesn't appear to generate much more than a trickle of volume. So - perhaps the latter? Has anyone included a link to one of your blog posts in an email newsletter, for example? It would also be helpful to see full headers and content; it's great you're doing SPF/DKIM/DMARC, but they're not going to prevent all issues - there might be something elsewhere in your message that gives us a better idea where the problem is. Finally, keep in mind that anti-spam systems are highly dynamic, and results can change - sometimes in as little as a few minutes - based on the latest data feeding into the system. If you're lucky, you might see this resolve on its own fairly quickly.
krick 11y ago
I'm still wondering how does it happen that I never ever had as many problems with spam as I have with anti-spam systems and goddamn google in the first place. Actually, at this point it well might be only google. And it's hard to ignore when more and more emails in your contact book have "gmail" in them. Makes it seem that something as simple and fundamental as e-mail now belongs to google. Just ridiculous.
loupgarou21 11y ago
I never had an issue with gmail blocking my mail server, but I was getting flagged as spam by AOL every sixish month because my server wasn't sending them enough mail... The company I work for resells an antispam solution with outbound filtering, so I just relay through that now
dools 11y ago
I had a problem when I sent a message to about 20 associates and used BCC. Google apps for business then just marked me as spam. I would get lots of rejected email messages. The people I sent to were all people I knew who were attending an event. Just gmail being a dick about BCC. So now I send email through sendgrid from within gmail. Have you tried using an external smtp relay? It's more likely your IP address than your domain that is flagged.
cvs268 11y ago
Part of the gmail's spam filtering appears to be "crowd-sourced". People clicking on "Report Spam" or "Not Spam" on your emails in their gmail-inbox. How about asking a few of your friends to select your emails in their gmail spam folders and click "Not Spam". Hopefully that gets the ball rolling and the situation improves...
mortenlarsen 11y ago
I feel your pain. I have had similar problems over the last 15 years that I have been running my own mail server, but mostly with hotmail when they were the gmail of their time. (they would accept the mail and then silently drop it) I would maybe change the SPF record to fail instead of soft-fail (~all to -all). I have a @gmail.com account that I use for testing whenever I change something.
ciaoben 11y ago
Hi, I'm Nicolò and I work in the deliverability team of Qboxmail.com. We have run into similar problems in the past. The reason of the behavior of Gmail with your domain is not easy to understand, a theory could be that in the past someone has used it for sending spam, and even if it was not your server, you are paying for this. The solution we have used, it was bring Gmail to understand that your emails have been sent from a real user. You can simulate a conversation between your domain and other Gmail accounts. Send a first email, with a realistic text, to a Gmail account, if it goes in spam, mark as safe (remove it from spam), and reply to it, always with a realistic text. Continue to simulate a normal conversation, sending 4/5 emails between the 2 account. Re-do the operation from another Gmail account, but this time do the opposite: start from the Gmail one. It not bulletproof, but it has worked in the past for us.
- leni536 11y ago
  So the way to get off from spam blacklists is to send spam mails between two addresses? Looks reasonable.
  - ciaoben 11y ago
    You send emails that Gmail looks wrongly like spam. If everything is clean, it is reasonable to think that it is matter of little fixes, and to "help" Gmail to understand that domain doesn't send spam. There is no way that this method would work id the domain it does actually send spam mails.
  - giancarlostoro 11y ago
    Could always start legitimate conversations with a friend who marks you as safe. :)
  - joering2 11y ago
    Sending about 250,000 emails per month for over a year now (all double optins), we found out that both gmail and yahoo has what we call "do you listen" rule. Basically even if user expressly add you to their contact list and pull you out of spam, etc, if you sending them just your email ignoring what they reply back to you, eventually you get back to spam. I think this is very clever and is based on simple human rule that most follow when conversating over email: when people reply to email the do not start a new email, they hit reply and previous email from their convo lands on the bottom of this new email. Once we started emailing new emails with adding content from addessee previous email to the bottom, our emails start getting back to addressee major mailbox. And this make perfect sense on how to detect a real conversation (gmail has your sent items so if incoming email contains something you wrote before, its an indication you have a real live conversation with someone)
Tepix 11y ago
What network are you hosted at? Perhaps your netblock is notorious?
digital-rubber 11y ago
Funny how when somebody else mentions the poor mail filtering at google few days ago, people assume it's that persons mistake. And now everybody reads to be understanding and trying to provide solutions. Must be the position of the moon or something.
fahadalie 11y ago
Please make sure that your domain or email address is not blacklisted by Spamhaus. You can check the status here: https://www.spamhaus.org/ https://www.spamhaus.org/ It is the blacklist removal center. I removed my email address from this list and since then, everything seems to be working perfect.
y0ghur7_xxx 11y ago
I know that on HN "me too" comments are not well seen, but I just have to this time, because I am really frustrated by this as well (as said before: https://news.ycombinator.com/item?id=9812157 https://news.ycombinator.com/item?id=9812157). This part sums up as I see it as well: I can only think this is intentional on Google’s part – they have a near monopoly; the vast majority of mail I send these days goes to Google – and if a small company is running their own mail server is too much of a hassle, then maybe they’d buy Google Apps. It’s bad, anti-competitive behavior on Google’s part. Shame on them if its true. I don’t know if it is, I can only guess, but they certainly have an incentive to make it difficult for the little guy. I’m just a geek that likes running my own servers. My pleas to Google’s impersonal forms fall on deaf ears, and I’m getting tired of telling everyone I e-mail to check their spam folders.
- sneak 11y ago
  I just tell people I email that their free email provider is fucking them out of their legitimate messages.
xedarius 11y ago
Of course it's deliberate, it's part of Google's ongoing business model of ring fencing the internet. People tell you not to run your own mail server as it's a nightmare to keep on top of all the security aspects, and yes that's a thing. The greater problem is getting your mail delivered to the vast majority of people with gmail/hotmail/yahooo mail.
teekert 11y ago
I have the same with my own server and Yahoo, after my first announcement of switching from Gmail to my own server Yahoo sent me a message that I was blocked forever. I have SPF set up but not DKIM. I also don't have a valid cert (I'd have to renew every year). Gmail never gave me any problems though. Yahoo apparently has procedures to deal with this but they are difficult to find. The annoying thing is that now I'm not even receiving any notification of the blocking anymore. The mail just disappears. Btw, mail from my Drupal system also always ended up in spam. But I kept removing the spam tag and now it is just goes into regular mail.
amelius 11y ago
I always wonder how companies like Mailchimp stay out of Google's blacklists.
- pbhjpbhj 11y ago
  Money?
pbhjpbhj 11y ago
I've had similar problems over the last year or two with Gmail and with Outlook/Live Mail. Our 2 person small time company sends a few hundred mails a month at most. We replay to someone on Gmail and get spam-binned. They whitelist us, we reply again and get spam binned. We send the mail via a major provider and it gets through. On Outlook a website mail form sending emails to a hotmail.co.uk address was getting blocked - the server has the same IP it's had for years, the form has been used for years, the recipient has whitelisted the email address. I forget what eventually fixed it, think it was addition of a reply-to address; quite ridiculous. In both cases they are long term domains with real ID info that hasn't changed, the domains have been on the same IP held by the same ISP for at least 3 years and owned by the same owner used for the same businesses for at least 10 years. Both domains are long-term registered in (Google|Microsoft) analytics. Yes I can see that such domains could be purchased by spammers and the prior owners may not change their ID info and the new holders may be able to purchase space on the old server and so keep the IP address (despite the established ISP having strong anti-spam policies) and may be able to then send out spam emails, but who would whitelist those emails??? IMO on either Outlook or Gmail if you whitelist something, even if it were spam from a known spammer, then they should let it through (sanitised if needs be). If they wanted to they could add a "99.999996% of others blocked this but you have whitelisted it, do you want to block emails from YourBestFriendWhoSendsSpam@theirISP.com in the future?". /rant
trusche 11y ago
I've had similar, unfathomable problems with Gmail delivery for a long time, until I added a PTR record for reverse IP lookup for the IPv6 version of my IP address. That did the trick, I haven't had delivery trouble since, even without DKIM (but with SPF).
kazinator 11y ago
> A few days ago, I attempted to e-mail a company regarding an online e-commerce order I had placed, from my personal address. What? How does e-mailing a company end up with a bounce from ... Google Groups? (See attached screenshot of the bounce.) Maybe the original outbound message had something funny in the To: or Cc: recipient lists.
z3t4 11y ago
Some years ago I ran a mail server on a dynamic IP. I basically contacted Google and had them white-list me. And at the same time, some small providers totally refused to accept mail from me because I used a dynamic IP :P Check-list before you attempt to get white-listed/ban-lifted on Gmail: Send bulk with "precedence: bulk" header. Use spf and dkim (optional). Always send from the same domain. Have your users "opt-in" to receive mail. Have a public e-mail policy. Basically, don't require users to enter their e-mail address.
cnst 11y ago
My 2c: I know it sucks when your mail doesn't get through, I've had the same issue with att.net, but gmail.com has never been a real problem for me. I have a lot of mail that gets forwarded to gmail, including from cron that goes to my own mailbox at gmail. I sometimes have to unmark it as spam, but not too often. My IPv4 doesn't even have a custom rDNS -- only provider-specific one -- nor have I bothered to implement DKIM, although I do have SPF and am also registered for Webmaster Tools (although I somehow doubt that really matters).
sashk 11y ago
I've had and still have similar issue like you do. I have my own smtp server, which sends some alerts to my google apps account. I've setup my Google Apps account to white -list all email arriving from my mail server, whitelisted source domain, added spf records, etc - but my mail still being blocked. First, Google told me that's because I use IMAP and I should stop using it - only webmail, or exchange on iOS devices - fine, still blocking. Then I was told, that I can't attempt connection to the second MX server if first fails - done, still have problems. Two or three month later, they confirmed issue on their end with firewall blocking DDoS by blocking whole /24 subnet, with no fix in sight. Remember, I am paid user and I went through rounds of bs with their support. I feel sorry for those, who have to send email to Google Apps/Gmail as part of their business. (I've posted this as a comment on original story, but decided to duplicate here, if someone will find this info helpful)
EmployedRussian 11y ago
Google just posted http://gmailblog.blogspot.com/2015/07/the-mail-you-want-not-spam-you-dont.html http://gmailblog.blogspot.com/2015/07/the-mail-you-want-not-... "but sometimes these wanted messages are mistakenly classified as spam. When this happens, you might have to wade through your spam folder to find that one important email (yuck!). We can help senders to do better, so today we’re launching the Gmail Postmaster Tools." https://gmail.com/postmaster/ https://gmail.com/postmaster/

310 points by stbenjam 11y ago

I'm very curious about that as well as I am thinking of moving my personal domains away from Gmail and it would really suck to start landing in spam simply on the basis of not being with a major mail provider. I hope this story gets traction and someone on the Gmail team finds it and comments.

ceejayoz 11y ago

If the domain is bitbin.de and the mail server is hosted on the same server, it's probably Hetzner being scorched-earth for deliverability, just like any other major hosting provider will be. Email from AWS, Rackspace, Heroku etc. get a drastically higher starting spam score because people fire up servers there to spam. Running the email through something like Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.

quadrature 11y ago
This seems to also be the case with DO, I setup a mail server on an instance there and can't escape people's spam boxes.
- arrty88 11y ago
  FWIW, I am able to send mail effectively from my Linode box, (passing SPF and DKIM).
  - kijin 11y ago
    Linode has always been very strict about keeping their IP ranges clean. The fact that Linode's lowest tier has been $20/mo until recently ($10/mom nowadays) also helps. That makes Linode unnecessarily expensive for people who just want to burn IPs, since they can easily get an IP for $5/mo with DO or even less with low-end VPS providers.
- brobinson 11y ago
  Check the IP against the spam blacklists: http://mxtoolbox.com/blacklists.aspx http://mxtoolbox.com/blacklists.aspx You might have to spin up a bunch of servers to find one which doesn't have a blacklisted IP. Then you have to build the IP's "reputation" with the various providers before you can send a significant amount of email through it.
  - quadrature 11y ago
    Thanks, I just assumed their whole block would be blacklisted. I'll try some more.
    - ceejayoz 11y ago
      Not blacklisted, just treated with higher suspicion by default.
  - massaman_yams 11y ago
    These days, there are only a handful of IP-based blacklists where a listing result in more than low single-digit impact to your delivery rate; moreover, among industry insiders, Gmail isn't known to use any public IP-based blacklist for deliverability.
    - brobinson 11y ago
      >Gmail isn't known to use any public IP-based blacklist for deliverability Good to know! I bet they have their own internal ranking system which is far more accurate than the public blacklists.
      - stonogo 11y ago
        I'm curious to know why you think they'd be any more accurate than public blacklists. About a fifth of the spam that hits my domains originates from gmail servers, and the majority of people I know complain (when asked) about gmail's tendency to spam-bin legit mail.
- mattkrea 11y ago
  I ran my personal mail server on DO for about a year or so without issue. I would definitely try a new instance--I didn't even know this was a problem.
cmwelsh 11y ago
Amazon SES has a pretty poor set of IPs due to the issue you mentioned about firing up servers to send spam. Mandrill is top notch, just like other professional email providers such as Mailgun and Sendgrid.
- ceejayoz 11y ago
  Are you thinking of EC2? SES (as with Mandrill, Sendgrid, etc.) runs on a separate block of IPs and is very strict about spam/bounce reports. Very easy to get yourself throttled or cut off entirely.
jsnell 11y ago
I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery. It doesn't even have SPF or DKIM set up. And I'm pretty sure that if messages were not getting delivered, users would complain. At least that's exactly what happened the one time there was a configuration screwup [1] that made some email providers reject mail from that server. So I don't know if that theory explains the issue either, though it could be a component. [1] https://www.snellman.net/blog/archive/2014-12-05-how-buying-a-ssl-certificate-broke-my-email-setup/ https://www.snellman.net/blog/archive/2014-12-05-how-buying-...
- ceejayoz 11y ago
  > I run a server on Hetzner that sends maybe 100k messages a week, and appears to have no problems with delivery. That volume (assuming non-spammy content, of course) would over time build up a positive reputation for your IPs, outbound addresses, and content. A new or low-volume IP on Hetzner is going to have a harder time overcoming the initial suspicion Gmail saddles it with.
  - massaman_yams 11y ago
    That's probably more true in general than it is in this particular case. Gmail does IP reputation differently than other major ISPs, and domain/content reputation are very different beasts entirely. Gmail's approach with a low-reputation IP is generally to throttle it aggressively; mail still gets through, just not very quickly if you're sending in bulk. Domain reputation is not initially distrusted unless it's a brand-new domain (registered in the last 30 days or so), and generally doesn't come into play unless there's very unusually poor metrics (think sky-high spam complaints) associated with the domain. Content reputation is applied in a similar manner. The headers I'm looking at should most likely give Gmail enough distinct identifiers to work with to isolate it from broader reputation issues at Hetzner, but the post only included partial headers, so I can't quite say that conclusively.
  - fuzzix 11y ago
    Always worth checking if the IP you inherit is in some DNSBL. A number of Hetzner IPs I've gotten were tainted. It's oh so much fun trying to get them removed.
seiji 11y ago
Amazon SES or Mandrill would probably be helpful. Both have generous free tiers. But you are still sending all your outgoing email through an external 3rd party. Not ideal. Also, these days every email is a top-reply orgy of all correspondence ever sent, so your SMTP relay essentially gets a copy of your entire conversational inbox (via "QUOTE ALL THE THINGS" reply garbage) as well as your outbox.
drzaiusapelord 11y ago
While I don't doubt this, I often see from DIY types some really lazy smtp habits. If they don't have their reverse DNS matching the forward DNS as well as the HELO name matching that as well, then, yes, that will be a major red flag for anti-spam systems. Reverse DNS is pretty important and weighed heavily. smtp delivery is something of a black art, but at the very least those things need to be set correctly. Its also worth mentioning that if you have 'spammy' applications that opt people in to mailings then a lot of people just get pissed off and click the 'junk' button over and over instead of unsubscribing properly. So think twice before auto-subing customers or enabling 'mail me replies' by default on forums and such. I also learned the hard way that once you're on Cisco Ironport's spam list, then you really can't get off. There's no one to talk to. Its supposed to be automated. In practice, that means months before your domain is unblocked. I find most Ironport admins just use that sole list even while Cisco recommends they use many and weigh the average. After a stupid config change that opened our smtp to relaying, we got off everyone's blacklist but IronPort. IronPort is pretty much the defacto standard in several types of enterprise. My fix? Get a different IP address for the mail server.

slg 11y ago

The problem might be the IP address of the mail server and not the domain. Where are you hosting the server? Is it is your personal server? If so, what is the ISP and is it a residential connection or a business connection? If it is hosted by someone else, what is their reputation as a source of spam.

runin2k1 11y ago

That's all a bit presumptive and inflammatory for what amounts to pure speculation on the part of the author. Google stands to lose a lot more from a potential PR disaster for burning former customers who move away from hosting than they do from trying to convert a tiny portion of users to a free mail hosting service.

MichaelCrawford 11y ago
My own cousin blackholed my gmail because gmail sends so mych spam. I had to ask him to configure an exception just for me. If you dont want to receive spam use lafn.org. I dont know how one gets an account there but I expect it doesnt cost anything. If you want a full VPS use http://prgmr.com/ http://prgmr.com/ - "we dont assume you are stupid.". Its a hosting service operated by neckbeards for the benefit of their fellow neckbeards. Quite cheap, I get mine for free because I help them with their marketing.
- kbar13 11y ago
  i got mine for writing a tutorial for setting up archlinux on prgmr on their wiki. :D
- alanpost 11y ago
  I run a personal mail server there too. I'm very happy with them: I was in the free tier for years, after they acquired the hosting provider I had been using. I recently asked them to start billing and am very happy to be a paying customer.
- aaronem 11y ago
  If you run your own MDA, you could do a lot worse than to filter incoming mail against https://www.spamhaus.org/zen/; https://www.spamhaus.org/zen/; the free tier is extremely broad, and the coverage is excellent -- I get maybe one spam email a week, tops.
  - _delirium 11y ago
    I'm also happy with Spamhaus Zen generally, but spam sent via Gmail is one category of email it won't stop, since it only blacklists pure spam sources, and Gmail is obviously a legitimate mailhost. I get a steady trickle of spam from Gmail addresses, mostly SEO-consultant spammers who grabbed my email from the whois info.

petercooper 11y ago

I send a LOT of emails each month (email newsletter business - yes, legit!) and ran into an separate but topically related and amusing problem recently. My newsletters are aimed at developers, and one issue went out and was considered by Gmail to be a 'phishing' attempt. I couldn't figure it out. Several issues later, another one was picked up the same way and I figured it out.. In both issues, one of the items was linking to domains that looked a bit like this "www.0x10abcdef.com" (this is NOT the actual domain) - basically a domain that looked like a hexadecimal number. I ran numerous tests and Gmail always considered mails with links to domains like this to be phishing attempts. I reported this as a bug (since nothing was wrong or reported with the domains in question, it was basically Gmail's filter being in error) but no idea if it was ever resolved.

brobinson 11y ago
It's probably specifically penalizing anything matching /https?:\/\/0/ because you can specify an IP address using hex or octal in most common browsers. hex: http://0xD83AC02E http://0xD83AC02E octal: http://033016540056 http://033016540056 That's my guess, at any rate.
- petercooper 11y ago
  Yes, URLs like that are commonly used in phishing emails, so that's my hunch as well. They're just not recognizing that such strings can appear in domains so I suspect it's a simple bug but probably not one they'll fix.
- hobarrera 11y ago
  OT: What does the red-flower-on-the-sun doodle mean?
  - deadbeef404 11y ago
    Argentina National Day 2015 Edit: You can usually find out what they mean by going to https://www.google.com/doodles https://www.google.com/doodles
11y ago

bitJericho 11y ago

Seems like author hasn't done enough troubleshooting and has jumped to a conclusion. My recommendation is to run through google's troubleshooting steps on the issue: https://support.google.com/mail/troubleshooter/2920052?hl=en https://support.google.com/mail/troubleshooter/2920052?hl=en Also make sure a DMARC record is setup.

gknoy 11y ago
At the end of that, it asks about whether your server passes "the diagnostic tests" -- but I didn't see a link to which tests it was talking about.
- blfr 11y ago
  It's MXToolBox[1]. To be fair to OP, it's very popular. [1] http://mxtoolbox.com/diagnostic.aspx http://mxtoolbox.com/diagnostic.aspx
stbenjam 11y ago
I have been through this, and the end result is a contact form that no one responds to. I filled it out a few months ago and again this week. DMARC is enabled (and in one of the screenshots, Google shows it passing). I get reports from them, but it provides no insight into their decisions.

arihant 11y ago

"The needs of the many outweigh the needs of the few." -- Spock. I'd rather have a few geeky people like me not have the fun of self hosting an e-mail server, than waste millions of man hours round the world because of people dealing with spam. If you must hack, however, use something like Mailgun. It is more hacky in a way that you can program your incoming mail the way you want, not just install a mail server with a few commands.

techsupporter 11y ago
That...doesn't make any sense. The author doesn't appear to be spamming or doing any kind of mailing list traffic. He is an individual communicating with other individuals. That's kind of the whole point of this Internet thing we have going here. Besides, he's only being singled out by Google; no other providers seem to have a problem with his domain and Google's own headers show that everything is fine.
- pekk 11y ago
  It's a coherent opinion which makes sense, even if we don't agree with it. The claim isn't that the author is spamming, the claim is that the availability of effective spam filtering ("needs of the many") trades off against and ultimately outweighs the author's reasons for running his own small web server for his personal email ("needs of the few").
  - hueving 11y ago
    No, the opinion is not even logically valid because there is nothing that shows Google's punishment of his server is required to reduce spam. It's like eliminating poverty by killing all poor people. Yes, it would work, but the side effects are worse than the outcome.
    - pekk 11y ago
      Logical validity only means that the argument works given its premises, not that the premises are correct.

thrownaway2424 11y ago

The author jumps to an unfounded conclusion which is pretty irritating and probably will make everybody who could help him not want to help him. That said I've never seen a DSN like the one in the screenshot. It certainly is not generated by the gmail spam checking system, because gmail does not bounce spam. Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.

stbenjam 11y ago
> The author jumps to an unfounded conclusion which is pretty irritating and probably will make everybody who could help him not want to help him. Apologies if you took it as a conclusion, it was only hand waving speculation, but google does have an incentive to make it difficult for the little guy... > Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders. They do if you have DMARC enabled, and a REJECT policy, although they still seem to ignore it sometimes and I have no clue my mail is sitting in a spam box.
massaman_yams 11y ago
It's entirely true that Gmail accepts mail at much higher rates than other major ISPs, and that their default handling of unwanted mail is to place it into the spam folder - but the truth is they can and will bounce mail if it's bad enough. It's just extremely uncommon for most people - even email marketers - to ever run into a situation where that happens.
hobarrera 11y ago
> Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders. This is a million times better. Meanwhile, microsoft accepts it, sends it at passed on DMARC reports, but discards the mail silently without it even reaching the spam mailbox.

lm741 11y ago

I hit a similar problems when sending automated internal emails to a Google Groups address at my company. The problem was fixed by adding the following footer: "To unsubscribe, email <my-email-address>."

spinlock 11y ago
Uh oh. I created an "unsubscribe" filter a while ago to get rid of spam. It's been great for keeping my inbox clean but I just don't see stuff with an unsubscribe line anymore.
- Lorento 11y ago
  It's not a good idea to manually create filters that 100% block. You can't think of all the special cases like this.
  - raverbashing 11y ago
    Exactly, especially based on an not-so-unique word Doing a quick search on my inbox, there are some relevant emails containing the word 'unsubscribe'
  - spinlock 11y ago
    I just filter it into a separate folder and skip the inbox but I've definitely missed some emails because of it.
danieltillett 11y ago
Wow I have to try this trick. I have endless problems with some emails going missing (hotmail and the rest of microsoft’s domains are the worst).

logicallee 11y ago

>What can I do except move to a hosted provider ? To answer your question about what you can do, you can send mail to Google legal asking them to accept your mail gmail users - just copy your whole blog post, I'm sure they'll action it right away: Antitrust is serious business. (They already have trouble in the EU for it - https://www.google.com/search?q=antitrust+eu+google https://www.google.com/search?q=antitrust+eu+google - by the way I think it's completely unfounded.) It takes them seconds to whitelist you, and this really is a "15 seconds could save you $15 million on your next antitrust case" and anyone in Google legal can probably see that. EDIT: I don't see how this got downvoted. This literally answers OP's question about what else he can do.

chc 11y ago
Antitrust is serious business, but seeing as there is no clear antitrust issue here, I'm not sure what you think getting lawyers involved will accomplish other than making it impossible to help you since your case is now an impending legal action.
- deleted 11y ago
  [deleted]
  - chc 11y ago
    > What do you mean there is "no clear antitrust issue" here? Most of OP's contacts use gmail, but gmail refuses his legitimate outside mail. But Google does accept almost everybody else's email, including many actual commercial competitors, and there is no reason to believe anyone at Google is deliberating singling OP out or even knows who OP is, so I can't imagine where you're seeing anticompetitive behavior. > Since email is an open protocol, and since his mail server is easy to verify and sends no spam (as he outlines at long length in his blog post), this really is quite blatant. Imagine if, early in the millennium, Internet Explorer refused to connect over http with servers that weren't IIS. This is more like if Internet Explorer failed to connect to some random Joe's homebrew personal server but could connect to most other people's just fine whether or not they were using IIS. That's not anticompetitive behavior, it's just a bug, and in fact there were many things that Internet Explorer did not handle correctly.

tedunangst 11y ago

Spam filtering/tagging is more annoying than outright bans. Microsoft banned my IP from their email services, but helpfully sent bounces so I could appeal. This is the email equivalent of hellbanning.

_cbdev 11y ago

I've had mail sent via my own server rejected by gmail because of a missing Message-ID header. The 550 reject message was the standard "Unsolicited Mail detected" text, the same mail was accepted without causing any fuzz once the Message-ID was added.

baudehlo 11y ago
Why on earth wouldn't your mail server add that for you?
- _cbdev 11y ago
  Because I'm writing my own ;) https://github.com/cmail-mta/cmail https://github.com/cmail-mta/cmail if you're interested.

jmount 11y ago

Network effect. Google blocks non-GMail email, eventually you are forced to use them as your service provider. Can either be deliberate or happy unexpected outcome for Google.

rebelde 11y ago
I use Google for my email, and they still count some as spam. Hell, it is sent from Google's own servers! It still doesn't help.

raintrees 11y ago

Anecdotal information: I run my own mail server and manage a number of small business clients who have mail servers. Email trust is getting more and more tedious. Recently I was resolving a domain registrar issue with Network Solutions. They required forms filled out and signed, a copy of a utility bill from my client, a copy of my ID... I bundled up everything and emailed them the scans. I contacted them 5 business days later, they claim to never have received it. I sent it again while I had a rep on the phone, it went into their spam hole, probably due to size of attachment. They helpfully suggested I get a GMail account to send the same message. They are my registrar, they host my DNS, including my MX record. I have an spf record... I thought it was pretty farcical, and a sad statement of digital trust/authenticity. Some of my clients are giving up and just going with the flow, I have had several conversions to Google/Microsoft cloud-hosted solutions for email...

arihant 11y ago
An spf record only helps preventing emails sent from phishing servers. It is inherently meaningless to spam filters if the trust with the domain owner itself is misplaced.
- blfr 11y ago
  True but if they're raintrees' registrar, they should already be able to connect the domain with the person/company who registered it.

codezero 11y ago

I've had the same problem myself. My best guesses as to why my domain has been dinged: 1) It's on a VPS, that IP may be flagged already 2) I often use a VPN, and sometimes send emails out through my server using it. This probably raises the red flags. 3) It's a non-standard TLD, .co 4) I don't use any Google services with that email address/domain (I assume doing so adds some level of measurable trust) 5) I'm not in the address book of a lot of gmail users, because this is my private server, that I use only for job seeking, and personal communication. This has really done damage in the past. I've applied for jobs, and heard back weeks after they hired someone, letting me know, woops, I ended up in their spam folder. Trying to meet with a friend? I can't email them, I need to use Facebook or Gmail... welp. This is super annoying. I kind of understand why it happens, but it's just a little sad that building your own fort, so to speak, is so impossible. Things I tried to do to mitigate this: 1) Made my web domain https only (why not?) 2) set up DKIM and SPF (didn't seem to have any effect) 3) proper SMTP authentication, secure port only 4) Reached out to Google via the typical forms they offer, and heard nothing, obviously.

Animats 11y ago

Notice the comment on the story: "I had the exact same problems in the past. For me it helped just to register and immediately cancel a trial of Google Apps for the domain. It’s annoying to have this as a necessary step but at least it was like that done in just a few minutes (after I tried to find a contact email of Google for a much longer time). I tried it because I had other domains on the same server which didn’t had this issue. All domains which weren’t filtered had a Google Apps account in the past. So I thought it’s worth a try, and yeah it was solved."

stbenjam 11y ago
Thanks, may have to pay Google the $5 (or thereabouts?) to host my mail for a month and consider it a cost of freedom from them long term. After 2+ years of hosting my domain myself, it doesn't seem possible to build up enough reputation with them personally.
- massaman_yams 11y ago
  If the Google Apps anecdote is true, it's likely to be a design flaw in their anti-spam systems - think of the number of spammers who could register domains, associate them with a $5 google apps account, and spam millions of Gmail recipients. If you think about the kind of metrics/outcomes the Gmail anti-spam team uses to measure success, it's hard to imagine a scenario where that would be good for them. What this looks like to me is a reputation problem with one or more domains, either in your headers or your message content. From a domain reputation perspective, you don't generally need to 'build' reputation in the same way you often do with IP reputation; rather, Gmail tends to tag domains with negative reputation only when they've observed unusually bad behavior associated with the domain. Sometimes this can happen due to domain forgery or spoofing. Sometimes it's because a sender is doing overly aggressive email marketing. Sometimes it's just because your domain shows up in someone else's bulk email. With full authentication, spoofing is unlikely; I also don't see any indication you're doing email marketing here, and your IP 88.198.5.136 doesn't appear to generate much more than a trickle of volume. So - perhaps the latter? Has anyone included a link to one of your blog posts in an email newsletter, for example? It would also be helpful to see full headers and content; it's great you're doing SPF/DKIM/DMARC, but they're not going to prevent all issues - there might be something elsewhere in your message that gives us a better idea where the problem is. Finally, keep in mind that anti-spam systems are highly dynamic, and results can change - sometimes in as little as a few minutes - based on the latest data feeding into the system. If you're lucky, you might see this resolve on its own fairly quickly.

krick 11y ago

I'm still wondering how does it happen that I never ever had as many problems with spam as I have with anti-spam systems and goddamn google in the first place. Actually, at this point it well might be only google. And it's hard to ignore when more and more emails in your contact book have "gmail" in them. Makes it seem that something as simple and fundamental as e-mail now belongs to google. Just ridiculous.

loupgarou21 11y ago

I never had an issue with gmail blocking my mail server, but I was getting flagged as spam by AOL every sixish month because my server wasn't sending them enough mail... The company I work for resells an antispam solution with outbound filtering, so I just relay through that now

dools 11y ago

I had a problem when I sent a message to about 20 associates and used BCC. Google apps for business then just marked me as spam. I would get lots of rejected email messages. The people I sent to were all people I knew who were attending an event. Just gmail being a dick about BCC. So now I send email through sendgrid from within gmail. Have you tried using an external smtp relay? It's more likely your IP address than your domain that is flagged.

cvs268 11y ago

Part of the gmail's spam filtering appears to be "crowd-sourced". People clicking on "Report Spam" or "Not Spam" on your emails in their gmail-inbox. How about asking a few of your friends to select your emails in their gmail spam folders and click "Not Spam". Hopefully that gets the ball rolling and the situation improves...

mortenlarsen 11y ago

I feel your pain. I have had similar problems over the last 15 years that I have been running my own mail server, but mostly with hotmail when they were the gmail of their time. (they would accept the mail and then silently drop it) I would maybe change the SPF record to fail instead of soft-fail (~all to -all). I have a @gmail.com account that I use for testing whenever I change something.

ciaoben 11y ago

Hi, I'm Nicolò and I work in the deliverability team of Qboxmail.com. We have run into similar problems in the past. The reason of the behavior of Gmail with your domain is not easy to understand, a theory could be that in the past someone has used it for sending spam, and even if it was not your server, you are paying for this. The solution we have used, it was bring Gmail to understand that your emails have been sent from a real user. You can simulate a conversation between your domain and other Gmail accounts. Send a first email, with a realistic text, to a Gmail account, if it goes in spam, mark as safe (remove it from spam), and reply to it, always with a realistic text. Continue to simulate a normal conversation, sending 4/5 emails between the 2 account. Re-do the operation from another Gmail account, but this time do the opposite: start from the Gmail one. It not bulletproof, but it has worked in the past for us.

leni536 11y ago
So the way to get off from spam blacklists is to send spam mails between two addresses? Looks reasonable.
- ciaoben 11y ago
  You send emails that Gmail looks wrongly like spam. If everything is clean, it is reasonable to think that it is matter of little fixes, and to "help" Gmail to understand that domain doesn't send spam. There is no way that this method would work id the domain it does actually send spam mails.
- giancarlostoro 11y ago
  Could always start legitimate conversations with a friend who marks you as safe. :)
- joering2 11y ago
  Sending about 250,000 emails per month for over a year now (all double optins), we found out that both gmail and yahoo has what we call "do you listen" rule. Basically even if user expressly add you to their contact list and pull you out of spam, etc, if you sending them just your email ignoring what they reply back to you, eventually you get back to spam. I think this is very clever and is based on simple human rule that most follow when conversating over email: when people reply to email the do not start a new email, they hit reply and previous email from their convo lands on the bottom of this new email. Once we started emailing new emails with adding content from addessee previous email to the bottom, our emails start getting back to addressee major mailbox. And this make perfect sense on how to detect a real conversation (gmail has your sent items so if incoming email contains something you wrote before, its an indication you have a real live conversation with someone)

Tepix 11y ago

What network are you hosted at? Perhaps your netblock is notorious?

digital-rubber 11y ago

Funny how when somebody else mentions the poor mail filtering at google few days ago, people assume it's that persons mistake. And now everybody reads to be understanding and trying to provide solutions. Must be the position of the moon or something.

fahadalie 11y ago

Please make sure that your domain or email address is not blacklisted by Spamhaus. You can check the status here: https://www.spamhaus.org/ https://www.spamhaus.org/ It is the blacklist removal center. I removed my email address from this list and since then, everything seems to be working perfect.

y0ghur7_xxx 11y ago

I know that on HN "me too" comments are not well seen, but I just have to this time, because I am really frustrated by this as well (as said before: https://news.ycombinator.com/item?id=9812157 https://news.ycombinator.com/item?id=9812157). This part sums up as I see it as well: I can only think this is intentional on Google’s part – they have a near monopoly; the vast majority of mail I send these days goes to Google – and if a small company is running their own mail server is too much of a hassle, then maybe they’d buy Google Apps. It’s bad, anti-competitive behavior on Google’s part. Shame on them if its true. I don’t know if it is, I can only guess, but they certainly have an incentive to make it difficult for the little guy. I’m just a geek that likes running my own servers. My pleas to Google’s impersonal forms fall on deaf ears, and I’m getting tired of telling everyone I e-mail to check their spam folders.

sneak 11y ago
I just tell people I email that their free email provider is fucking them out of their legitimate messages.

xedarius 11y ago

Of course it's deliberate, it's part of Google's ongoing business model of ring fencing the internet. People tell you not to run your own mail server as it's a nightmare to keep on top of all the security aspects, and yes that's a thing. The greater problem is getting your mail delivered to the vast majority of people with gmail/hotmail/yahooo mail.

teekert 11y ago

I have the same with my own server and Yahoo, after my first announcement of switching from Gmail to my own server Yahoo sent me a message that I was blocked forever. I have SPF set up but not DKIM. I also don't have a valid cert (I'd have to renew every year). Gmail never gave me any problems though. Yahoo apparently has procedures to deal with this but they are difficult to find. The annoying thing is that now I'm not even receiving any notification of the blocking anymore. The mail just disappears. Btw, mail from my Drupal system also always ended up in spam. But I kept removing the spam tag and now it is just goes into regular mail.

amelius 11y ago

I always wonder how companies like Mailchimp stay out of Google's blacklists.

pbhjpbhj 11y ago
Money?

pbhjpbhj 11y ago

I've had similar problems over the last year or two with Gmail and with Outlook/Live Mail. Our 2 person small time company sends a few hundred mails a month at most. We replay to someone on Gmail and get spam-binned. They whitelist us, we reply again and get spam binned. We send the mail via a major provider and it gets through. On Outlook a website mail form sending emails to a hotmail.co.uk address was getting blocked - the server has the same IP it's had for years, the form has been used for years, the recipient has whitelisted the email address. I forget what eventually fixed it, think it was addition of a reply-to address; quite ridiculous. In both cases they are long term domains with real ID info that hasn't changed, the domains have been on the same IP held by the same ISP for at least 3 years and owned by the same owner used for the same businesses for at least 10 years. Both domains are long-term registered in (Google|Microsoft) analytics. Yes I can see that such domains could be purchased by spammers and the prior owners may not change their ID info and the new holders may be able to purchase space on the old server and so keep the IP address (despite the established ISP having strong anti-spam policies) and may be able to then send out spam emails, but who would whitelist those emails??? IMO on either Outlook or Gmail if you whitelist something, even if it were spam from a known spammer, then they should let it through (sanitised if needs be). If they wanted to they could add a "99.999996% of others blocked this but you have whitelisted it, do you want to block emails from YourBestFriendWhoSendsSpam@theirISP.com in the future?". /rant

trusche 11y ago

I've had similar, unfathomable problems with Gmail delivery for a long time, until I added a PTR record for reverse IP lookup for the IPv6 version of my IP address. That did the trick, I haven't had delivery trouble since, even without DKIM (but with SPF).

kazinator 11y ago

> A few days ago, I attempted to e-mail a company regarding an online e-commerce order I had placed, from my personal address. What? How does e-mailing a company end up with a bounce from ... Google Groups? (See attached screenshot of the bounce.) Maybe the original outbound message had something funny in the To: or Cc: recipient lists.

z3t4 11y ago

Some years ago I ran a mail server on a dynamic IP. I basically contacted Google and had them white-list me. And at the same time, some small providers totally refused to accept mail from me because I used a dynamic IP :P Check-list before you attempt to get white-listed/ban-lifted on Gmail: Send bulk with "precedence: bulk" header. Use spf and dkim (optional). Always send from the same domain. Have your users "opt-in" to receive mail. Have a public e-mail policy. Basically, don't require users to enter their e-mail address.

cnst 11y ago

My 2c: I know it sucks when your mail doesn't get through, I've had the same issue with att.net, but gmail.com has never been a real problem for me. I have a lot of mail that gets forwarded to gmail, including from cron that goes to my own mailbox at gmail. I sometimes have to unmark it as spam, but not too often. My IPv4 doesn't even have a custom rDNS -- only provider-specific one -- nor have I bothered to implement DKIM, although I do have SPF and am also registered for Webmaster Tools (although I somehow doubt that really matters).

sashk 11y ago

I've had and still have similar issue like you do. I have my own smtp server, which sends some alerts to my google apps account. I've setup my Google Apps account to white -list all email arriving from my mail server, whitelisted source domain, added spf records, etc - but my mail still being blocked. First, Google told me that's because I use IMAP and I should stop using it - only webmail, or exchange on iOS devices - fine, still blocking. Then I was told, that I can't attempt connection to the second MX server if first fails - done, still have problems. Two or three month later, they confirmed issue on their end with firewall blocking DDoS by blocking whole /24 subnet, with no fix in sight. Remember, I am paid user and I went through rounds of bs with their support. I feel sorry for those, who have to send email to Google Apps/Gmail as part of their business. (I've posted this as a comment on original story, but decided to duplicate here, if someone will find this info helpful)

EmployedRussian 11y ago

Google just posted http://gmailblog.blogspot.com/2015/07/the-mail-you-want-not-spam-you-dont.html http://gmailblog.blogspot.com/2015/07/the-mail-you-want-not-... "but sometimes these wanted messages are mistakenly classified as spam. When this happens, you might have to wade through your spam folder to find that one important email (yuck!). We can help senders to do better, so today we’re launching the Gmail Postmaster Tools." https://gmail.com/postmaster/ https://gmail.com/postmaster/