11 ms·
Google App Engine Silently Stopped Sending Email 5 Weeks Ago
- markdown 11y agoNo change at https://status.cloud.google.com/ https://status.cloud.google.com/
- tedmiston 11y ago> I've set this thread to not be visible by anyone but yourself and Google. Or not.
- Klathmon 11y agoa few comments later he asked for it to be made public again.
- cyberpanther 11y agoI had a Google engineer tell me once that the App Engine email api was never meant to send a lot of mail. That is why the quota is so low. I would recommend using a third party service to do e-mail.
- Maxious 11y agoI've been getting bounces on Google Apps hosted gmail for calendar notifications coming from calendar-notification@google.com Email is hard.
- pmlnr 11y agoFor programmers, maybe. Running mail servers is a sysadmin task. (I mean no attack here, this is plain statement)
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- tshtf 11y agoRunning a mail server is simple, even for programmers. Running a mail server with reliable outbound delivery, and inbound spam filtering, is anything but simple.
- pmlnr 11y agoA reliable mail stack is configuring postfix+dovecot or cyrus, adding dspam, opendmarc, opendkim and a bunch of dns records. How is this any harder than anything else in computing?
- tshtf 11y agoDKIM + SPF + DMARC does not guarantee reliable delivery. Not even close.
- pmlnr 11y agoOf course not. However, in the past ~7 years, while running my own mail stack, I never had a delivery issue, though obviously this is small traffic. At work, we handle pretty large traffic for mail, with self-hosted mail servers as well - those are exim. No delivery issues either, although these are only the real, important, not even remotely spam-like, core business messages. Spam-like newsletters are done via sendgrid there.
- chris_wot 11y agoWhich is why you pay Google to do it for you via AppEngine!
- mikecb 11y agoSendgrid is a great alternative and not hard to set up.
- SuperKlaus 11y agoAlso: mailgun
- codenhagen 11y agoAlso: Mandrill.
- foxylad 11y agoYeah, but pricing is ten times Appengine's (actually more - recently we don't seem to be being charged for email at all). Amazon SES is the only service with a comparable cost that I've found. I'd love to hear of other alternatives at the 1c/100 level.
- daveguy 11y agoAt least a few reported that it was bouncing at the destination once they set up an appropriate handler. They are only "not being sent" when there is a url in the message and that is a big red flag on the receiving side (not much text but a url). It doesn't look like an App Engine problem. It looks like messages being identified as spam problem.
- markdown 11y agoNot the case for me. No bounces being reported. In any case, gmail and outlook don't bounce messages for spam; They put them in the spam bin. Nothing whatsoever changed in my code. I haven't touched it in a year, and all of sudden email stopped on the 1st of April. There's nothing in the changelogs to indicate there was a change that should cause this behaviour. It's absolutely an App Engine problem. https://cloud.google.com/appengine/docs/python/release-notes https://cloud.google.com/appengine/docs/python/release-notes
- daveguy 11y agoInteresting. I have been somewhat weary of the "machine learning for everything!" approach that google has taken recently -- i just don't think it's there yet. I wonder if a learning algorithm on the google side just started deciding outgoing spam and not giving correct feedback. It is very curious that no url works and with url doesn't. That should give straightforward repeatability (if they can simulate as your account). Have you tried sending to different target e-mail addresses during the test? I saw once it just worked and a bunch of stuff flew through. Also saw there was some question as to quotas. Did you get it clarified? Are you maybe above usage quotas with the regular e-mails so test e-mails are being stopped/delayed? Are test and production separate? It seems like if it were a google side issue there would be an uproar from the users.
- djsumdog 11y agoNo, gmail constantly drops messages without putting them in spam. That's actually a huge problem. Microsoft does the same. That's for e-mails that don't even contain any links!
- 11y ago
- fencepost 11y agoReading through the thread, it looks like in the last few days it's been narrowed down to "messages which contain the URL of the App Engine instance silently disappear with no bounces or errors."
- achant 11y agoAnd only sometimes. The issue resolved itself on my app two weeks ago. My guess is there is an outgoing spam filter run amok in some of the clusters.
- zitterbewegung 11y agoI think there is not a spam filter but this is a rule to prevent people from setting up phishing scams or other deceptive emails and they are using an appengine instance to host their scam.
- CydeWeys 11y agoThat seems like a very good guess to me. It's a tricky balancing act between allowing everything and locking things down to fight spammers.
- djsumdog 11y agoI wrote about this last year, but I feel it's still relevant. Spam filters, especially Google and Microsoft's (whose e-mail servers don't play by any of the standard rules nor do they send DMARC reports), are horrible over aggressive. http://penguindreams.org/blog/how-google-and-microsoft-made-email-unreliable/ http://penguindreams.org/blog/how-google-and-microsoft-made-... I have seen more phishing/ransomware e-mails come through mine (lots of malicious Javascript made to look like Excel files), so I can understand how great the risk is. The fact still remains though. E-mail is broken and is less reliable than an envelop dropped in a letterbox.
- tacon 11y ago>especially Google and Microsoft's (whose e-mail servers don't play by any of the standard rules nor do they send DMARC reports) I don't know about Microsoft, but I get a couple of DMARC reports per day from Google's email infrastructure. I also get reports from aol.com, yahoo.com, linkedin.com, comcast.com, fastmail.com, etc.
- deleted 11y ago[deleted]
- waleedka 11y agoMust be an isolated issue. I run two App Engine apps and send 30K emails a day and I haven't noticed any change in my email open/click stats, which implies that emails are being delivered as usual.
- Ensorceled 11y agoWell, that is one of the problems I kept hitting when I was on App Engine ... issues that "only" impact 10% of customers.
- markdown 11y agoAn isolated issue that's been allowed to continue for weeks.
- mattbillenstein 11y agoDon't use appengine for anything -- ever. It's a terrible platform/product and google would rather you move on to GCE anyway...
- enneff 11y ago> google would rather you move on to GCE anyway... That's not true. We continue to invest heavily in App Engine as a platform.
- groks 11y agoAppengine's whack pricing says: we'd like to kick you off appengine onto GCE but that would scare away all the enterprise users from GCE. If we announce regular Moore's Law following price cuts for GCE and nothing for appengine, well hey, we can't stop you leaving. We're totally not twisting you're arm! (aside: we are) "A commitment to Moore's Law: https://cloud.google.com/pricing/philosophy/ https://cloud.google.com/pricing/philosophy/ "
- JeremyBanks 11y agoWhat is this "Moore's Law"?
- cramsdale 11y agoI'm sorry, this simply does not make any sense. If you'd take the time to articulate a well-formed case, I will take the time to dig in. Thanks... -- Chris
- hartator 11y agoI think he is saying that GCE being less expensive than App Engine will fore people to migrate.
- boulos 11y agoThe new Tesla Model 3 is cheaper than the Model S. Is that Tesla encouraging people to migrate to the Model 3? They're not the same product, and in this case they serve different points in the space with different underlying requirements. App Engine Standard's cost structure on the backend just hasn't changed nearly as much as compute engine's lately where it's a much more direct: oh look Haswell released, more cores per host at similar dollars yields less $$/core. Disclosure: I work on Cloud (Compute Engine mostly) and care a lot about our prices.
- martindelemotte 11y agoI wonder if people at Google know how awful it is to report an Appengine bug. You have to battle for days to get your bug acknowledged. I've noticed that they've added more support people since the early days but it's still a pain as they seem to have an incentive to answer the tickets as fast as possible without doing any real investigation. The linked issue is a good example of this behavior.
- kordless 11y agoThey should be working on a migration tool for users of AppEngine to migrate to Google Container Engine.
- foxylad 11y agoWe wouldn't use it. To me, the whole point of Appengine is to abstract out system administration. GCE (like AWS) abstracts out the systems, but not the administration - you suddenly become responsible for all the scaling, fault-tolerance, and all the subsystems that Appengine manages for you.
- jdoliner 11y agoConfusingly enough Google Container Engine != GCE. GCE is Google Comput Engine, which doesn't abstract out much of the administration. Google Container Engine (or GKE) on the other hand does abstract out most of the administration via Kubernetes. We were recently attempting to use appengine for something but gave up due to how bad the support for Go is and migrated to GKE. It definitely isn't quite as abstracted as Appengine but it's darn close.
- deepsun 11y agoI believe you're talking about GCE (Compute Engine), while @kordless is talking about Container Engine. I've tried Containers with Google App Engine Flexible Environment (in beta now) and liked that it's much more customizable than standard GAE. Basically, you just need any Docker container listening on port 8080. But deploy time took 10-20 minutes, so I'm still preferring Standard. Hopefully they'll fix that before GA release.
- Animats 11y agoThis is what happens when you buy a service from an organization oriented toward ad-supported services. The mindset is that it doesn't matter if the user isn't being served.
- ajackfox 11y agoI'm pretty surprised by the long wait times between "updates" from support... that's pretty rough. I understand it can be hard to get a good signal to noise ratio with support, but when so many other people are reporting issues, perhaps it's time to take it a bit more seriously? This thread is a great example of why support is so hard.
- user_0001 11y agonot really. I wish I could get away with providing such crap support as the big players do. If my customers don't get a reply within a few hours, then all hell breaks loose. Even if, there request comes in at 3am / 0300 on a Sunday morning my time. To be fair, when I point this out to them they do apologise, didn't realise time difference, are in a different part of the world where Sunday is not a day off etc. I then ask what they expect from some billion dollar companies support wise and they say they are happy with a reply within a week. Asking them why a one man shop doing quite a bit less than billions of dollars is expected to provide so much more. Anyway, rant over. Support isn't hard. You hire enough competent people as your customers require. The end. Happy customers
- Godel_unicode 11y agoOh, is that all? Someone should let Google know.
- awinter-py 11y agoThe problem is that pesky humans can't decide what they want. Not sending mail is a bug but not receiving mail (i.e. spam filtering) is a feature? I can appreciate how this is confusing.
- markdown 11y agoNot sending is a bug when it's undocumented, when it happens without notification, when there's no error message, and when there's no outgoing-spam bin. The GAE docs say "do x and we'll send your mail". If we do x and mail isn't sent, then that is a problem.
- Godel_unicode 11y agoExcept as soon as you tell the spammer what is blocked and what isn't, they just add that to the playbook. I agree with you that some measure of notification that there's spam filtering would be good, but it's Google we're talking about. There's spam filtering.
- markdown 11y ago> I agree with you that some measure of notification that there's spam filtering would be good, but it's Google we're talking about. There's spam filtering. Nonsense! I had no idea this was a feature of GAE. I pay for the use of GAE, and so long as I send less than my quota of emails, it's not Google's business what the content of those emails are. There's no reason that there should be silent spam filtering on outgoing mail sent by my app. If they suspected that my app was sending spam, they should have notified me and disabled my app. That would have alerted me to the problem. I find it absolutely abhorrent that Goole would just send private communications sent to me by prospective clients into the trash where they can never be retrieved or restored, and to do so without telling me is beyond belief.
- drunken-serval 11y ago> it's not Google's business what the content of those emails are You are very wrong about this. Senders of email need to make sure they aren't being used for spam or they get blacklisted.
- tominous 11y agoMaybe I'm old-fashioned but if I was affected by this issue (and it had persisted for so long) I would set up an SMTP server and an appropriate MX record, then demonstrate using mail logs or port snooping that this was definitely not a receive-side issue. Of course the support organisation here (Google) should be well equipped to set up this kind of testing themselves and work with a customer to root cause the issue. In my experience though these issues are rarely taken seriously until there is no-one else left for support to blame.
- jpatokal 11y agoGoogle Cloud Support here. To correct the misleading title, this is not a generalized outage of any kind, but an issue with some applications sending mail that trigger spam/malware/etc filters. Since spammers generally do not comply with RFC 3514, filtering is unfortunately not an exact science, so false positives happen and we're working with the customers in question on a case-by-case basis. Also, App Engine is integrated with both Sendgrid and Mailgun, and we strongly recommend using these if you're planning on sending larger quantities of mail: https://cloud.google.com/appengine/docs/python/mail/sendgrid https://cloud.google.com/appengine/docs/python/mail/sendgrid https://cloud.google.com/appengine/docs/python/mail/mailgun https://cloud.google.com/appengine/docs/python/mail/mailgun
- markdown 11y ago5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did. Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would. Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had been delivering for years. It did this without notification and without triggering any errors. Approximately 35 quote requests that should have been delivered have disappeared into thin air. You've cost me money, and I'm disappointed to find out that you'd been notified of this issue weeks before it began affecting me.
- moondev 11y agoWhy do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area. At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.
- mryan 11y ago> It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area. I partially agree with this - dedicated email services have a higher deliverability rate than a random web server, especially a cloud-based server that might be using an external IP that was previously used to send spam. However, I can understand the parent poster being annoyed that their emails were being delivered one day, and not the next, without any changes on their end. > At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job. This is absolutely the wrong tool for the job. You should not be using a Gmail account to send out automated emails that you care about.
- cramsdale 11y agoHey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread. Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: cramsdale@google.com. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me. We’re here and we want to help. -- Chris (Lead PM for App Engine)
- je42 11y agoIsn't there an API to find out why a message was SPAM filtered ? Or are you afraid that "rogue" applications will use it to produce messages that are SPAM but yet not trigger the SPAM filter ?
- jcrites 11y agoThe recent post about Candy Japan's attempt to combat credit card fraud gives a good sense of what fighting fraud and abuse is like: https://news.ycombinator.com/item?id=11431881 https://news.ycombinator.com/item?id=11431881 As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked. Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment. All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).
- hartator 11y agoThis is a documented API endpoint: https://cloud.google.com/appengine/docs/python/mail/ https://cloud.google.com/appengine/docs/python/mail/ with quota.
- samskeller 11y agoHow could you loudly stop sending email?