11 ms·
Gmail and Google Drive Outage
- Mandatum 6y agoRight at 9am pacific too. Good reason for me to wrap up in APAC. Thanks Google!
- hnick 6y agoIt's nearly the end of the workday here in Australia. Very insensitive of them, they should be more timezone aware. Edit: I seem to be able to send from some accounts but not others. It looks like emails without attachments are fine, otherwise the drive error messes them up.
- codemac 6y agoPlease tell our SREs in the SYD (in darling harbour/pyrmont) office thank you from me.
- throwawaywrench 6y ago>It's nearly the end of the workday here in Australia. Very insensitive of them It's nearly the end of the workday on the east coast of Australia. Very insensitive of you. Sincerely, Western Australia The above is a joke just in case that wasn't clear.
- hnick 6y agoI guess I kicked an own goal there. At least it's not daylight savings yet so the QLDers can't be annoyed at me too!
- chippy 6y agoPacific as in Pacific Time (PST/PDT) ? It's not even 3am there currently.
- sushshshsh 6y agoThat would explain why my ETLs are failing at 3am in the morning !!!
- FridgeSeal 6y agoYour ETL processes write to google drive or are dependent on email? Earnest question, don't mean to sound derisive.
- shrikant 6y agoI'm assuming some kind of workflow which uses email notifications, and only goes to the next step if the email notification succeeds..?
- nimchimpsky 6y agosounds horrific
- pirocks 6y agoMy understanding is part of google cloud failed(Google Cloud Storage), causing many outages, including but not limited to google drive/email.
- sushshshsh 6y agoUser emails file to drive script connects to drive and parses list of files within script copies files into redshift and moves original file to different directory or deletes it alas what can be done? this is what the company gets for building like this ^.^
- jwally 6y agoIf it makes you feel any better; same thing happened to me. Ugly Frankenstein process to get our data out of a third party vendor who can only e-mail us reports failed half-way through its run, and I was about to spend hours digging through to see why until I saw this.
- OnWriting 6y agoGmail is practically unusable at the moment. It's been going on for a while too, so I'm interested in the post mortem for this. I'm in Sydney, Australia.
- hnick 6y agoDo you have a signature image or other attachment? It sends instantly without, fails with [on the desktop web client]. The google drive server they save our stuff on is giving a 503.
- OnWriting 6y agoMy email drafts can't even save. A popup saying "Oops, something went wrong. Recent changes may not have been saved." No images either!
- hnick 6y agoHmm odd. I had a draft from this morning (I took a photo on my phone, attached, then went to desktop later to type the rest up). It would not send and kept popping up that error you mentioned while typing. As soon as I deleted the attachment, I could send it. New emails with nothing attached work fine too in all 3 accounts.
- Yeri 6y agoYes -- remove the attachment (including in-line images) and it'll work
- Santosh83 6y agoPlain HTML GMail seems to be working as well as ever here in India...
- kuu 6y agoI wonder how much money this kind of outages can cost to all affected companies and institutions
- danparsonson 6y agoHopefully a lot less than they save by outsourcing this part of their infrastructure :-)
- kuu 6y agoYeah, I imagine so :D
- samdung 6y agoI'm so used to Gmail/Gsuite working that i assumed the problem was with my internet and restarted the router a couple of times.
- Havoc 6y agoYeah same w/ hotel internet. Especially because Teams status notifications seem funky today too
- laurynas-s 6y agoGmail works but I can't send any emails - so frustrating.
- tobtoh 6y agoYoutube is also affected. You can upload files but they won't process.
- synack 6y agoIncident says it's affecting anything that uses Cloud Storage, which I'm guessing is most things. https://status.cloud.google.com/incident/zall/20007 https://status.cloud.google.com/incident/zall/20007 Edit: New incident number https://status.cloud.google.com/incident/zall/20008 https://status.cloud.google.com/incident/zall/20008
- downvote-avoid 6y agoWell this happens if you rely only on 3rd party providers. My email server is still running without any issues and without any indecent for last 3 years (system reinstall). Surely no one can send me emails now that gmail is down as most of world is relying on single point of failure, but this is another story. Moral of this story is - always own your mission critical infrastructure.
- wuunderbar 6y ago> Moral of this story is - always own your mission critical infrastructure. Sure, if you have the time and money to own it properly. How far do you need to go to say you own it? Multi-regional servers located on properties that you own?
- b8ne 6y agoConfirmed it's an Object Storage issue. I've personally had dramas uploading APK's to the play store. Also noticed a bunch of big websites down. Must be a decent issue.
- naedish 6y agoI'm guessing this outage will allow GSuite customers to claim Service Credits under the SLA - does anyone have any experience with doing so? Google's documentation is lacking in detail[0]. [0] https://gsuite.google.com/intl/en/terms/sla.html https://gsuite.google.com/intl/en/terms/sla.html
- deleted 6y ago[deleted]
- colinbartlett 6y agoFollow up to that: Does anyone know if there are tools to help customers collect and submit data to their their vendors for SLA credits? I’d like to add something like that to StatusGator but I’m unsure if there’s a market.
- RachelF 6y agoGood luck with that - read GSuite's terms, they, Google, define what an outage is, not the customer.
- dannyw 6y agoThe linked terms say: > "Downtime" means, for a domain, if there is more than a five percent user error rate. Downtime is measured based on server side error rate.
- breakingcups 6y agoWhich is fantastic if the error causes the server side not to be able to log errors
- iagooar 6y agoIt depends on the specific country's laws - in Germany you can not write anything you want into a contract and call it valid. In this case, you could still sue Google if according to industry standard definition it actually was an outage.
- cezart 6y agoThe status from Google Cloud status page offers a bit more technical details of what happened: "We are experiencing an issue with Google Kubernetes Engine (GKE) clusters using node auto-provisioning becoming stuck during node version upgrades. Node auto-upgrades have been disabled temporarily." https://status.cloud.google.com/ https://status.cloud.google.com/
- piahoo 6y agoDoes it mean, that Gmail is hosted on GKE?
- 9nGQluzmnq3M 6y agoThat issue is unrelated. This is the correct one: https://status.cloud.google.com/incident/zall/20008 https://status.cloud.google.com/incident/zall/20008
- deleted 6y ago[deleted]
- bleepblorp 6y agoGmail accessible via IMAP at $undisclosed_location. SMTP send works, but sent messages reappear in the inbox rather than in sent mail. Haven't tested the Gmail web interface. I don't use it.
- londons_explore 6y agoSent messages appearing in the inbox has been a bug with Gmail since the dawn of time.
- maple3142 6y agoIs this only affecting G suite? I still can send and receive email nornally.
- jordanthoms 6y agoThere was an outage August 19th, 2019 - almost 1 year ago to the day. As I posted at the time: "Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time." My pet theory wasn't too popular but I'm going to stick with it :) 1- https://news.ycombinator.com/item?id=20740997 https://news.ycombinator.com/item?id=20740997
- plibither8 6y agoSee you in August 2021, good sir!
- piva00 6y agoI work on the educational part of the product my company develops and I can attest that school start is a stressful day with login attempts, assignments lookups and other setup activities for the school period. I wouldn't doubt that Google Classroom and other systems that use Google's SSO will be under strain from millions of students.
- mrkramer 6y agoMillions of people are searching simultaneously at google.com or youtube.com but servers are not crashing. Issue is not traffic overload but something else.
- klohto 6y agoThese are not the same products nor infrastructure
- mrkramer 6y agoBut I'm sure similar infrastructure architecture was applied to gmail.com as it was to google.com and youtube.com. And similar concepts of maintaining by sysadmins are practiced.
- londons_explore 6y ago
- tommykins 6y agoI have 5 tabs open with emails written ready to send at some point. Had issues saving drafts, too.
- blindm 6y agoIn fairness Gmail has a good uptime record, so it's not the end of the world
- bartman 6y agoWas greeted by this just as we started work. Observations so far: Reading and receiving emails via the Gmail web UI works, sending emails via the web UI doesn't work, sending emails via SMTP works.
- Yeri 6y agoIt's mostly related to attachment. Regular emails should be fine.
- nindalf 6y agoI'm not that interested in a dashboard, but I'd bee interested in reading the post-mortem. Since they have paying customers for these products, they might release that.
- octvcdrc 6y agoThey usually provide it on the same page after the incident (an example: https://www.google.com/appsstatus#hl=en&v=issue&ts=1593986399000&sid=1&iid=be4cdd50ed9bc6458ede98158a71cfd4 https://www.google.com/appsstatus#hl=en&v=issue&ts=159398639... )
- antihero 6y agoWe're unable to publish apps in the Play Store console/fastlane, too.