10 ms·
Google Services Experiencing Disruptions
- OnWriting 6y agoMy entire Meet call just got dropped and can no longer connect. Lots of 502's. The good news is that it happened on a Friday -- see you all next week!
- coderintherye 6y agoYeah that was what initially got me to check status. Checked twitter and saw others with the issue. Tried to do voice only call but Google Voice was down as well.
- alpb 6y agoIt's still Thursday in some places :)
- toomanybeersies 6y agoNot such good news if you were supposed to have a meeting in the morning with people in the UK. Due to timezones, it's had to be rescheduled to Tuesday morning.
- AnotherTechie 6y agoPoor UK people working Saturdays :(
- BrianOnHN 6y agoGoogle Analytics down Edit: Gmail Down
- johnward 6y agoYep analytics is experiencing issues for me.
- pirocks 6y agoI was writing a longish note in google keep, and now keep.google.com gives me 502s.
- ElijahLynn 6y agoSame here, it lost my note in Google Keep too, it was an important note too. The dashboard neglects to mention Google Keep experienced a disruption too.
- njovin 6y agoWe're getting 502 responses to all of our calls to all Google APIs. There doesn't seem to be a correlating incident on either dashboard: https://status.cloud.google.com/ https://status.cloud.google.com/ https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status
- erichurkman 6y agoSide note: Google'e status page, linked, breaks the back button. How the 'F can a simple status dashboard break something so simple? (In Google's own browser, too.)
- keerthiko 6y agoironically, if this is true for all chrome users, the back button works just fine on firefox + windows.
- Austin_Conlon 6y agoBack button disrupted for Safari on iOS.
- glxxyz 6y agoWorks in Chrome & Firefox for me on macOS Catalina but not in Safari.
- crazygringo 6y agoThe culprit is that linking to: https://www.google.com/appsstatus Then loads: https://www.google.com/appsstatus#hl=en&v=status If you hit back, it goes to the URL without hash, but then instantly goes to the one with the hash. Pretty annoying. (If you paste the 2nd URL directly, you will not have problems.)
- cgb223 6y agoIronically, this increases their own page’s pagerank
- gowld 6y agoHow?
- cgb223 6y agoPage views are one of the variables of the page rank algorithm If you keep going to the site over and over, then potentially it counts as new views (albeit not unique ones) That said Google probably exempts their own sites from page ranks since they you know, own the search engine
- Nuzzerino 6y agoMy Google Voice call was dropped about 5 minutes ago.
- adminprof 6y agoDo these status dashboards have any more credibility left? Always showing green when services are down.
- njovin 6y agoA few months ago when all the auth services stopped working it took them a while to update the pages too.
- clopez 6y agoOf course. Green means the service is working for me. It is Red when the service is not working for me, and clearly neither for nobody else. Orange means it doesn't work for me, but you might have better luck.
- njovin 6y agoSeveral services haven't been working for me for 20 minutes and the dashboard hasn't been updated. They're effectively useless.
- Polylactic_acid 6y agoI imagine its insanely difficult to write an automated status dashboard since a problem can cause the service to not work while not having everything entirely broken. The apis tested by the automated system could all still be working. Probably the only way would be to have a constantly running vm testing everything via the web ui.
- lrem 6y agoIt's not that difficult, just a piece of work. Google has teams dedicated to just that kind of work and internally has the automated dashboards that update real-time. The confusion here is that the page linked is not it. This is some static HTML or such that is updated when a human has triaged the issue, determined and aggregated scope and got an OK from another human (or more, depending on the situation).
- ikiris 6y ago
- coderintherye 6y agoOf course both dashboards show all green: https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status https://status.cloud.google.com/ https://status.cloud.google.com/
- ncr100 6y agoI am experiencing intermittent issues with GMail not loading new conversations in a timely fashion, and showing "something went wrong" notification messages.
- mmaunder 6y agoThere are definitely issues. GDocs functionality is intermittent. e.g. the ability to move a doc into a new folder or even see what folder it's in using the little folder icon at the top while editing has disappeared. I've suggested to our team they copy/paste locally anything they're currently editing.
- cgb223 6y agoWhat good is a dashboard if it doesn’t show actual status
- reustle 6y agoAre there any large community run status dashboards? Seems like it is something that we can't trust companies to be honest about (due to SLAs).
- booleandilemma 6y agoJust check Hacker News, of course :)
- gruez 6y agohttps://downdetector.ca/ https://downdetector.ca/
- Areading314 6y ago
- adamqureshi 6y agogmail down
- ab-dm 6y ago2/3 of my Google mail accounts are going up and down like a yoyo. Poor OSX doesn't know how to deal with it
- dkulchenko 6y agoWow, this is substantial. All my mail sync is down, Analytics isn't loading, Sheets and Docs are all stuck in "Reconnecting". I wonder what could cause this.
- martius 6y agohttps://twitter.com/uhoelzle/status/1309313556328841216?s=19 https://twitter.com/uhoelzle/status/1309313556328841216?s=19 "a pool of servers that route traffic to application backends crashed"
- hpjules 6y agoI'm also getting intermittent 502s in Gmail
- ataylor32 6y agoGoogle Calendar has been spotty for me and Hangouts isn't working at all.
- xyzzy_plugh 6y agoMy meet call dropped & hangouts chat is failing to connect. Maybe time to switch to Jitsi!
- neap24 6y agoFirebase api calls don't seem to be working either.
- jeanvalmarc 6y agoYeach, we're seeing `Authentication Expired` for our firebase auth calls. Firebase static hosting still looks up.
- walto2 6y agoGoogle Drive down. 502
- arrty88 6y agoFWIW, the GMail imap interface in Apple Mail is still working alright for me. Can confirm the gmail web interface is completely borked.
- johnnyb9 6y agoYouTube TV not working on my phone.
- integrate-this 6y agoOn the bright side, their DNS servers didn't go down, otherwise I would have spent about an hour arguing with my ISP.
- lostdog 6y agoIt's always good to memorize at least one IP address! Very similar to memorizing at least one phone number in this day and age ;)
- enlyth 6y agoGoogle image search is completely down for me
- mayank 6y agoSearch is still up and snappy! I may be biased as an ex-search Xoogler, but Search's resilience is a testament to how battle-hardened a stack it is, considering GDocs and Gmail appear to be hard down.
- catmanjan 6y agoIsn't search more or less static? Docs and mail change instantly and users expect instant feedback
- serioussecurity 6y agoSearch handles probably an order of magnitude more changes per second than GMail+Docs combined
- aeyes 6y agoBut if the index doesn't update you won't notice.
- derefr 6y agoYeah, search isn’t an OLTP system.
- DarthGhandi 6y agoOfftopic, but it blows my mind seeing <1 min old StackOverflow comments indexed by Google search, the sheer amount of work going on behind the scenes to make that happen.
- Polylactic_acid 6y agoI'm pretty sure this only happens when the site is actively pushing updates to google via their webmaster tools. I think your sites importance would also impact how fast google indexes it.
- throwbacktictac 6y agoSorry, it was me guys. I was just clone the quiche repo and I accidentally brought down the whole system.
- quacksmack 6y agoSide note: Hacker News is an IT guys best friend, came here for confirmation after staff started complaining about gsuite being down and the disruption announcement is top spot. Much appreciated
- thomas 6y agotwitter too is good for this, especially international
- sircastor 6y agoI just had problems playing Google Music. I assumed it had something to do with service derating because of the move to YouTube music.
- deleted 6y ago[deleted]
- pkaler 6y agoGoogle search is up. Google Mail, Calendar, Docs seem to be down. (This is probably a testament to who pays the bills.)
- nonbirithm 6y agoIt makes me wonder if it has anything to do with GCP. Pure speculation, but if it's in the cloud then they probably wouldn't hesitate to dogfood their own platform. But then it would be interesting since search still works. Maybe search was too high-value to risk having a hard dependency on whatever it is that is causing the outage.
- grumblez 6y agoThere was interruption between GCP's monitoring & services on GCP. I also have my own Prometheus/Grafana stack in there and it kept chugging along fine, even though their DB graphs show severe dips in tx/sec in the time period in question.
- dodobirdlord 6y agoGoogle has an internal cloud that they talk about sometimes. Borg, bigtable, spanner, etc. They run all their services on their internal cloud, including their public cloud, with the exception of Compute Engine, which is its own thing.
- jsolson 6y agoCompute Engine isn't really a separate thing so much as it's another thing built on the same infra. All the way down to the individual VMs, it runs on Google's production infrastructure and is scheduled by Borg (onto dedicated machine pools, although that sort of dedicated resource pooling is not unique to GCE). Once you get into VM networking things get a bit more exotic, although all of that still runs atop the same software defined network infrastructure as everything else (but, much like with Borg, it uses more features than many workloads at Google). (I work on Compute Engine's virtualization infrastructure)
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- ISL 6y agoChat coming back up. Hulu was down, appears to be back up? Edit: YouTube livestream is up, despite downdetector claim that it is down. Docs, inside and outside of GSuite appears to be up. 1836 Pacific: Downdetector shows improvement across Google services. 1840 Pacific: Downdetector shows marked improvement across Google services. They don't have units, but ~80% improvement in many services over peak down-ness. 1852 Pacific: Continued improvement. 1854 going for a walk. Looks like things are generally headed for normality.
- cwilkes 6y agoHah I read those as years and was wondering if there was an invasion from the Pacific in 1854.
- quacksmack 6y agoLooks like it might be fixed now?
- Polylactic_acid 6y agoI can access gmail just fine right now.
- aussieguy1234 6y agoI've seen Google Login fail on multiple services including Trello
- acruns 6y agoI had issues signing into YouTubeTV.
- daguava 6y agoIt's really funny reading this and seeing the lower-ranked comments saying "Time to switch!" Yeah, you won't. Stop being hyperbolic. You're amazingly served by google and you'll forget this 20 minutes after service is restored. It makes me wonder, what duration of outage is necessary for people to actually switch?
- leesalminen 6y agoWe switched from Rackspace’s cloud to AWS due to terrible failure modes with their managed MySQL product. We’d have a db cluster go down and wait hours for them to fix several times a year for several years. Switched to Aurora and haven’t looked back.
- vmception 6y agoI tried Zoho for my last startup, since gsuite nuked the free tier. That lasted 2 weeks. Being locked out of or requiring special considerations for the first calendar invite, or video conference, or collaborative document was a complete unnecessary pain. “But you can make a google account with your external email” barf. Its sad because Zoho is trying so hard too. Imagine planning and working on all those equivalent products. lol
- viraptor 6y agoI'm not sure I understand. What do you mean by locked out and by special considerations?
- _-___________-_ 6y agoPresumably they were still trying to collaborate with people inside Google's walled garden of documents, calendar events and videoconferences.
- vmception 6y agoWhich is everyone, or at least a very predictable friction.
- deleted 6y ago[deleted]
- llacb47 6y agoI didn't notice any issues at all. Weird
- deleted 6y ago[deleted]
- grumblez 6y agoJudging by the tx/sec count on a database I have hosted in us-west2, looks like their network backplane went up & down a few times between 5:58pm and 6:21pm tonight
- grumblez 6y agoActually, looks like it was something between their monitoring system & the DB. Per my prometheus stats, DB kept chugging along the whole time.
- dqv 6y agoSorry, I was doing some organizing on my gmail inbox with 100,000 unread messages.
- vonseel 6y agoNest app is not working either.
- dunham 6y agoWe just had a hiccup on that too. Seems to be back now.
- hughes 6y agoNot sure why Maps shows no disruption. It didn't work at all when I was driving home earlier.
- outside1234 6y agoI was using Google Reader and suddenly it vanished? Is that the disruption?
- bobsoap 6y agoClever!
- rkalla 6y agoTesla, Cox and Google in the last 48hrs =/
- sillysaurusx 6y agoOne downside of training on TPUs is that by default they save models to GCE buckets. When buckets go down, there’s no way to save or load your models. It’s kind of funny: my model is training right now; I see it training right now; but it can’t save, so the progress will be wiped out if the TPU preempts or malfunctions. Luckily they’ll fix the issue within 24h, which is when TPUs preempt. :)
- mlthoughts2018 6y agoThe other downside of TPUs is that they are ridiculously overpriced for all workloads, they are basically GCP snake oil, and on a cost benefit basis there is not a single use case where it’s rational to choose TPUs, not even over other options hosted in GCP.
- logicslave12 6y agoTPUs are more expensive per model train then regular GPUs? I thought the point was they were way faster, which makes up for the cost. Data scientists using gpus usually can do the math on costs...
- mlthoughts2018 6y agoEven accounting for speed, TPUs are worse on a cost benefit basis. People (some of whom are data scientists) pay to use them because they are the shiny new toy and there’s entertainment in blogging about it or writing a walkthrough, not because they economically solve any problem. I manage an enterprise machine learning team and we do tons of stuff on GCP. I’m dead serious: there is not a single use case where it makes sense to choose TPUs unless the main value you’re seeking is just the entertainment factor of using Google’s new thing.
- google234123 6y agoWow, you really like dealing with absolutes...
- NotHereNotThere 6y agoIncidentally, this affected Google's game streaming service Stadia, which is an interesting timing with Amazon's Luna announcement ;)
- tus88 6y agoNobody gets fired for buying AWS.
- young_hopper 6y agoGoogle Voice was down for me earlier, as well as a couple unresponsive Compute engines (despite not showing problematic on this list).
- GarrisonPrime 6y agoTinfoil theory: This was a test of their kill-switch.
- AnotherTechie 6y agoWhat does this button do? >It tanks our stock Why do we even have that button??? >lol idk
- zen_mike_eu 6y agoWasn't gmail down about 1month ago? Jeez, my boss looks at me weird now for switching to gSuite 4months ago
- ElijahLynn 6y agoWas taking important notes using Google Keep yesterday. Then, all of the sudden I couldn't take notes anymore. Couldn't type, it wouldn't accept input. Refreshed and got a 502. Then eventually could take notes again. And then I eventually ended up in a state where my new additions were DELETED. And now I guess I am not using Google Keep anymore, which is sucky but probably good for me in the long run.
- M4v3R 6y agoMaybe it's just me, but I try to never do any serious work in the browser, or to never write longer blocks of text in a browser app. And it's precisely for this reason - few applications implement measures for either the server or my internet going offline that work 100% the time.
- ElijahLynn 6y agoGoogle Keep experienced a disruption yesterday too, which the dashboard does NOT show.
- rataata_jr 6y agoAll the diversity hires are working out it seems.