19 ms·

Google outage – resolved

• NicoJuicy 6y ago
  Google employees can't login to update their status board on https://www.google.com/appsstatus https://www.google.com/appsstatus ?
• bpmayer 6y ago
  Just use alternatives: * DuckDuckGo instead of Google Search * Bitchute / LBRY / Odyssee instead of YouTube *Nextcloud instead of Google Drive
• mfi 6y ago
  Google has now updated the status page, every single service has an outrage: https://www.google.com/appsstatus https://www.google.com/appsstatus
• PersonalHitman 6y ago
  How coud all of googles websites be down this long tho. You would at least think google being the company they are would have some sort of like backup serevers or backup plan type stuff. Idk
• bberkgaut 6y ago
  It would be so interesting to see the postmortem...
• mtmsr 6y ago
  The moment you realize that the only way you notice Google is down is through HN. Cheers all you selfhosted-FOSS-alternatives! Time to bump up those Patreon contributions...
• __henil 6y ago
  Strangely everything seems to be little faster in incognito mode
• tcpekin 6y ago
  I happened to use Google Takeout to download everything I had last weekend (~200 Gb)... at first I thought my account was banned and that was the only solace I had, but this definitely is killing my productivity.
• Havoc 6y ago
  Is google's status page actually linked to anything? Or is it ahem airgapped?
• PersonalHitman 6y ago
  Hey can someone explain to me how all google apps are down cause idk a lot and ik that at least a kid ddosing these websites is not a possiblity so what kind of attacks are going down here? And how come google doesnt have like backup servers to run there websites on or is that not a thing?
  • deleted 6y ago
    [deleted]
  • wut42 6y ago
    Woah
• joseluisq 6y ago
  Look the status page still works https://google.com/appsstatus#hl=da&v=status https://google.com/appsstatus#hl=da&v=status but it says everything is broke.
• mesutpiskin 6y ago
  just authentication services does not work in Turkey.
• ricardo81 6y ago
  Back up for me, almost. Some warnings on gmail but the inbox is loading up.
• Geeflow 6y ago
  "Sign in with Google" also fails. So this outage is taking quite a few otherwise unrelated third party apps with it.
• Intermernet 6y ago
  Thank God! Google Cloud Print is still working! I'd be destroyed if that ever stopped working!
• m-i-l 6y ago
  The Google outage is one of the top BBC headlines now, so it has become mainstream news: https://www.bbc.co.uk/news/technology-55299779 https://www.bbc.co.uk/news/technology-55299779 "The outage started shortly before noon UK time, with Google sites returning server errors when visited. Users around the world reported problems with Gmail, Google Drive, the Android Play Store, Maps, and more. ... Despite the widespread outage, Google's service dashboard for its services reported no errors."
  • benjaminwai 6y ago
    "Google has been contacted for comment, but one spokesperson said they were unable to access their email."
  • sn41 6y ago
    It has become regional news in local regional newspapers in South India. It's that big.
• known 6y ago
  https://xkcd.com/2347/ https://xkcd.com/2347/
• rpadovani 6y ago
  This morning also Microsoft O365 had problems with authentication
• deleted 6y ago
  [deleted]
• miki123211 6y ago
  Everything seems to be back to normal, for me at least.
• justkez 6y ago
  Unless I'm misunderstanding something, they are going to have an immense SLA claim issue. 99.9% SLA on Workspace services, so any business paying for Google for Business (now known as Workspace) is going to have a credit claim (assuming the outage is longer than 43m 49s which feels like it will be). Edit: As I comment it looks like things are coming back! Timing or what...
  • deleted 6y ago
    [deleted]
  • oneeyedpigeon 6y ago
    Looks like the outage was very close to that duration. Over what time period is that?
  • robin_reala 6y ago
    57 minutes for full Gmail restoration according to https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid=52f0b876cce03b08d13dc753c55fc3b2 https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...
  • andruby 6y ago
    0.1% of a year is 8h45m. Or did you mean 99.99% SLA?
    • monospaced 6y ago
      99.9% but calculated on a monthly basis
• tango12 6y ago
  Bummed to have not received an email about what's happening and why from the Google team. Has anyone on a "platinum"/"enterprise" google workspace plan received any relevant communication about what's happening and ETA on uptime?
• jerbearito 6y ago
  I think it's back. Youtube is loading.
• thret 6y ago
  Is it due to the SolarWinds Orion exploit? https://cyber.dhs.gov/ed/21-01/ https://cyber.dhs.gov/ed/21-01/
• elcomet 6y ago
  It's back
• mehta_rohan 6y ago
  moment of empathy for engineers
• miej 6y ago
  i wonder what % of the hn crowd cant work if google auth is down
• rad_gruchalski 6y ago
  Calendar is also down.
• IceWreck 6y ago
  Whoose laughing at self-hosters now huh ?
• pachico 6y ago
  Yup, suffering it right now.
• manquer 6y ago
  Yeah hangouts/meet as well
• niklasmtj 6y ago
  Same here in Germany. Gmail says that my account can not be found. Plus Google's support page is also down
• asdojasdosadsa 6y ago
  Can verify - Finland
• nashtheman 6y ago
  Yeah, drive and youtube seem to be down as well
• _1tan 6y ago
  Drive too. Status board still green though.
• Jumziey 6y ago
  Exciting!
• 0x54MUR41 6y ago
  Yes, I can confirm this. I got an error when accessing Google Docs, Google Meet, and Gmail.
• clusmore 6y ago
  Australia, same problem here.
• domoritz 6y ago
  Yep, drive and gmail give me 500 errors.
• noutella 6y ago
  Alors, Search queries taking up to 3 seconds here in France.
• opdahl 6y ago
  Down in Norway too.
• madjam002 6y ago
  Google Home not working in UK
• degosuke 6y ago
  Anyone else has this in the technical details? Numeric Code: 7444
  • janpot 6y ago
    7498
• dkjaudyeqooe 6y ago
  Pretty much anything that requires login is broken.
• maxekman 6y ago
  Meet is also down. From Sweden.
• Avalaxy 6y ago
  The status page has proven itself useless again. According to that page, everything is working perfectly fine.
  • kiechu 6y ago
    What if it is a static website of corporate propaganda?
    • username90 6y ago
      It works 99.9% of the time, good enough eh?
      • RupertEisenhart 6y ago
        better than a stopped clock
  • daliusd 6y ago
    It works fine. You can't login only.
• tunnuz 6y ago
  Same (Austria).
• proto-n 6y ago
  Hungary - gmail loads, can't send email (error #2014)
• lwhi 6y ago
  Google home down too!
• pvsukale3 6y ago
  Google meet is also down
• ivan1783 6y ago
  Can confirm from Germany, youtube is broke.
• tosh 6y ago
  Just checked https://www.google.com/appsstatus https://www.google.com/appsstatus all green, which does not reflect reality for me (e.g. Gmail is down) edit: shows how incredibly difficult introspection is
  • TimMeade 6y ago
    Solid red now. Everything down
  • faeyanpiraat 6y ago
    HN is my goto status page when these things happen, never failed to provide up-to-date reliable information.
    • jansan 6y ago
      Google should write an AI application that checks on HN if their systems are working and display the result live on their status page. That would be more reliable than the current system they have in place (which is obviously not working at all).
      • jaimex2 6y ago
        It would probably just end up banning Googles own account and leave them with no human escalation option.
      • gostsamo 6y ago
        won't help when the ai service itself is down.
    • polote 6y ago
      Seems like everyone going on HN now. The site is so slow
      • davnicwil 6y ago
        Interesting cascade effect of sites going down! I wonder where people will go next to check if HN is down? Also wondering if this is perhaps the fastest upvoted HN post ever? 8 mins -> ~350 votes, 15 mins -> ~750 votes. I wonder if @dang could chime in with some stats? Update: looks like it hit 1000 upvotes in ~25 mins! Update: 1500 in ~40 mins Update: 2000 in ~1 hour 20 mins (used the HN API for the timestamp)
• rochacon 6y ago
  Google Meet and YouTube also 500 in Brazil
• surajs 6y ago
  can confirmt this
• akhiluk 6y ago
  YouTube shows a "something's broken" error. Similar errors when trying to access Gmail. Tried on multiple browsers and connections, got the same result. The search engine works fine, as of now.
  • tonfa 6y ago
    Youtube works when not logged in though.
    • akhiluk 6y ago
      You are right, the YT homepage opens up in private tabs. It seems strange, I wonder might be causing this.
• pharmhax 6y ago
  Same here, can't access gmail (currently on a trip in Poland).
• sugestercrm 6y ago
  Poland - the same
• d99kris 6y ago
  GMail IMAP is also down.
• slmyldz 6y ago
  can verify - Turkey
• decafbad 6y ago
  Yep. Gmail redirects to Gsuite Status Control and shows everything as fine there.
• meheleventyone 6y ago
  Dead in Iceland.
• henearkr 6y ago
  In Japan Youtube is down too.
  • henearkr 6y ago
    Everything is back and working fine now, here.
• jerbearito 6y ago
  I can access youtube if I'm not logged in (Incognito)
  • vineyardmike 6y ago
    Thanks for the tip! Several people said its related to logins/auth so this helps getting access to videos/info. maybe this is their fingerprinting/cookies/tracking server side logic has some bug that is failing?
• XOKP 6y ago
  Can't access my files on Google Drive, shit.
• himujjal 6y ago
  Good. Now, at least a few people would search Google alternatives in DDG
  • Veen 6y ago
    Search is still working. It’s Drive, Meet, and other stuff that millions of people depend on for work, so maybe stow your glee for a moment.
    • himujjal 6y ago
      Google still is the best when it comes to search. But its also important to know the alternatives. You will be surprised how many people dont know Google alternatives. lol
• HelloNurse 6y ago
  I'm able to access GMail and Maps but not Youtube or Drive.
• nicolapcweek94 6y ago
  Gmail, Chat and Meet down from Italy as well, was in a Meet call and got disconnected, only getting 500s across all three now
• behindai 6y ago
  Sweden. The same problem. Authorization doesn't work. Gmail doesn't work.
• avibhu 6y ago
  Can verify - India. Gmail, Drive, Docs, Youtube not working.
  • rk06 6y ago
    as another user mentioned, they work in incognito. looks like there is a problem with google auth which is causing the outage
• milankragujevic 6y ago
  All down for me, Serbia. YouTube works for me when logged out (i.e. in Incognito).
• dhbradshaw 6y ago
  Wow, I'm glad there are a few things that I do that don't require Google auth.
• zubairov 6y ago
  Gmail, Calendar, Google Accounts down in Germany
  • csunbird 6y ago
    Same, my work mail does not work. I was supposed to be in a meeting right now.
  • Kelteseth 6y ago
    That's why my Thunderbird constantly popped up to enter my Google Mail credentials…
• cabbagepatch 6y ago
  Google Cloud Platform is down in US.
• Foivos 6y ago
  same here in Norway.
• niico 6y ago
  Also down in Paraguay
• tango12 6y ago
  Google login also seems to be affected, meaning all services using google account login are inaccessible if you aren't logged in.
• skinnymuch 6y ago
  Yeah weird it’s all green. I just noticed things down as well. Came to HN to see how quickly people posted. This is already #1 at 6:55 am EST
  • nobleach 6y ago
    Yeah, that status page is really frustrating. Didn't anyone consider building it to REALLY check if services were working? not just returning a 200 Status... but actually DOING things?
• AbacusAvenger 6y ago
  The Nest endpoints are returning 500 as well.
• farbrortumm 6y ago
  yeah, same in the UK
• tibbydudeza 6y ago
  Was wondering why youtube can't show videos from the search page , it even reverted my language settings. South Africa
• have_faith 6y ago
  Gmail said my account was "temporarily" unavailable... had a moment considering if it wasn't temporary. Good reminder to remove my reliance on gmail especially.
  • tomglynch 6y ago
    Same for me - and then I tried logging in to gmail in an incognito window and got "Couldn't find your Google account" which really scared me.
    • JorgeGT 6y ago
      Time to stop postponing that large Google Takeout for me!
      • alpineidyll3 6y ago
        Clicking the stupid download links is so tedious...
  • tux3 6y ago
    Thunderbird prompting me to login, and getting "Google does not recognize this email address" as a reply was a nice adrenaline spike, until I checked the status on HN!
    • lioeters 6y ago
      This happened to me with a work email. For a moment I thought, "Well, I had a good run at this company.." My next thought was, "Ha, Google must be down - better check HackerNews."
    • BugWatch 6y ago
      I am still experiencing this problem on all my synced-to-Thunderbird Gmail accounts, so it either hasn't been completely fixed yet, or there's another ongoing issue.
  • imperfectcats 6y ago
• Dobby2020 6y ago
  I confirm, these are all Russian hackers.
• n_time 6y ago
  The console for Google Cloud Platform is also down.
• a012 6y ago
  Youtube is up for me, but CANNOT login to Gmail
• pantulis 6y ago
  Confirmed from Spain. Google Workspace seems to be down.
• Betelgeuse90 6y ago
  YouTube shows a monkey for me in Israel
• kokotkohaha 6y ago
  without adblock it works
• griffiths 6y ago
  What do you think, how much money are they losing right now every second?
  • skinnymuch 6y ago
    I was thinking the same thing. Also this funny bit that’s relevant: https://entertainment.theonion.com/google-shuts-down-gmail-for-two-hours-to-show-its-immen-1819595180 https://entertainment.theonion.com/google-shuts-down-gmail-f... “Google Shuts Down Gmail For Two Hours To Show Its Immense Power”
  • vnxli 6y ago
    Do tech companies post that kind of data? When I worked in the automotive industry, the Detroit Big 3 American auto makers had figured out their cost per minute of downtime. They would even charge that to their suppliers and anyone who shut down one of their plants. It was something like $10,000 per minute since they’re able to roll a complete vehicle off the assembly line about every minute
• zupa-hu 6y ago
  Hungary - gmail down, youtube down if logged in, search works with 5s response time
• kokotkohaha 6y ago
  In inkognito window (without Adblock) it works for me!
• notyourplayer 6y ago
  Maybe it’s time to go outside?
  • henearkr 6y ago
    This outage is an idea from Calico then? (just kidding)
  • jsemrau 6y ago
    What is this bright thing in the sky?
  • tiborsaas 6y ago
    Let's not get ahead of ourselves :)
  • paulmbw 6y ago
    back inside we go guys:(
  • muyuu 6y ago
    let's not make panicked drastic decisions just yet
• hemmert 6y ago
  Google Reader and Google Wave also seem to be affected.
  • madjam002 6y ago
    This made me chuckle :)
  • V-2 6y ago
    My FOMA on Google+ intensifies...
  • fouronnes3 6y ago
    Google Buzz as well
    • scrollaway 6y ago
      I can't even load Google+.
      • zapdrive 6y ago
        Same with Orkut.com, can't even open it.
        • deleted 6y ago
          [deleted]
    • timvisee 6y ago
      I'm experiencing issues with some other services https://killedbygoogle.com/ https://killedbygoogle.com/
      • hemmert 6y ago
        A very nice collection! (...and growing...)
  • nurettin 6y ago
• Sebb767 6y ago
  210 points in 5 minutes. 500 in 10. Seems everyone see's the status page and is like "hey, time to check hn" ;-) EDIT: You can permanently refresh and see the score increase by 2-3 points each second. Wow. EDIT 2: 1800 after one hour, but a few points were probably lost due to HN downtime. Being quick to post is important, guys;-)
  • bryanrasmussen 6y ago
    I noticed Gmail and of course the google doc I was working on were unavailable. I didn't come check HN until I said to my wife "Google's down" when she complained her email didn't work and she said "check the news to see if Google's really down"
  • Dayshine 6y ago
    I feel like HN's server aren't very happy with everybody doing this though. Stop bullying the servers :(
    • dkarp 6y ago
      Yep, HN getting it's very own HN hug of death from google
      • shoshino 6y ago
        Haha, the irony!
    • Sebb767 6y ago
      This is probably like an user flood wave, crushing all platforms in its wake. Give it 10 minutes and Reddit will be down as well :)
    • rakoo 6y ago
      This wouldn't be a problem if HN migrated to a blockchain written in go for performance
      • 6y ago
• anmolparashar 6y ago
  Can verify (India)
  • _navaneethan 6y ago
    Working in Incognito mode. Not when you logged In.
• deleted 6y ago
  [deleted]
• nihil75 6y ago
  That company is on a downward spiral...
• pachico 6y ago
  Same in Spain
• mr-developer 6y ago
  Gmail and YouTube are down in India as well.
• Waterluvian 6y ago
  1. Phone Gmail won't load. 2. Turn off wifi and use cell network. Still won't load. 3. Check hacker news.
  • elymar 6y ago
    Yeah I couldn’t get the hours of operation for a store near me on Google maps. Thought it was wifi at first.
• simlan 6y ago
  Maps is affected as well
• Ginkgophyta 6y ago
  youtube works if you don't log in. Otherwise, gmail, cloud, youtube with accounts are all down.
• sd5869 6y ago
  This is google's way of saying do not work from home
• real_ben_michel 6y ago
  Seems to be down in a bunch of places https://istheservicedown.co.uk/status/gmail https://istheservicedown.co.uk/status/gmail
• jlgaddis 6y ago
  Yep. Launched YouTube app on the Roku and was prompted to sign in. Opened browser on PC, entered "activation code" from YouTube app, after entering my Google account username on the login page, it presented me with the reassuring "Couldn't find your Google Account" message. Tried logging in to Gmail directly with the same effect. Thanks to the Twitters, I realized that Google hadn't canceled me, specifically (apparently they decided to temporarily cancel everyone!). FWIW, I typically get directed to their Chicago datacenter(s). (Note: 7 A.M. on the U.S. east coast on a Monday morning; at least they have impeccable timing!)
• metahost 6y ago
  Can confirm from India.
• deforciant 6y ago
  GCP console is not working too https://console.cloud.google.com/ https://console.cloud.google.com/, however at least the services are still running :) phew
  • factsaresacred 6y ago
    It's a mixed bag. Cloud storage is down but at least cloud logging is up to stream all the errors.
    • nobleach 6y ago
      Yup, Ops Genie is BLOWING UP with alerts... but can I get into GCP to see what's going on? Nope.
• ryankrage77 6y ago
  UK here, YouTube and Gmail are innacesible, search is working fine. Workspace Status Dashboard is showing all green.
• vnxli 6y ago
  My Gmail app on iOS keeps displaying a “No Connection” error even though my phone is connected to the Internet
• tekudiv 6y ago
  YouTube is working in India. But can not log in to any Google Accounts. GMAIL and Drive not accessible via web. Drive sync is working though. Appstatus still showing green.
• _spoonman 6y ago
  Google classroom is also down
• blntechie 6y ago
  Transitioned to Google Workspace from Microsoft Office 365 last week and it’s a nice welcome.
• sureshssarda 6y ago
  GCP and Firebase not working too.
• pupdogg 6y ago
  Confirmed: all mentioned above down midwest!
• Legogris 6y ago
  Google Cloud is also down. Can not access either Cloud Console or any of our GKE clusters. Stackdriver, Cloud Auth also down.
• Intermernet 6y ago
  Can confirm Incognito Mode and Private Tabs are still working on YT. Seems to be something auth based.
• lnsp 6y ago
  Seems like some core service crashed since all of these sites are unavailable. However, google.com seems to be fine.
  • tomglynch 6y ago
    I think it's just Accounts is down, so anything where you are logged in isn't working.
• nimitzeoauto 6y ago
  Youtube down in India, Docs Down, Mail working
• ffpip 6y ago
  HN is the fastest way to know about outages. Youtube & Gmail are down for me.
• max_hammer 6y ago
  Can verify - India
• nimitzeoauto 6y ago
  Youtube, docs down in India, Mail working
• username90 6y ago
  I feel like Google is having more outages recently. I worked there for 4 years and even in such a short time you really noticed the shift from engineering focus to business focus. But maybe it is just a coincidence.
  • rataata_jr 6y ago
    Or all the diversity hires.
  • woofie11 6y ago
    I interviewed for Google many years ago and recently. The expectations fell through the floor. I've see plots of number of employees too. I'm not sure how Google will dig itself out of that one.
    • thu2111 6y ago
      They made you interview again even after having worked there before? That's odd. It used to be that they didn't do that.
      • woofie11 6y ago
        I never worked for Google. They wanted to hire me after the first set of interviews, but I took a different opportunity at the time. Interview process was an intense, and the interviewers were sharp. I came out even more impressed with Google. This was way back in the day -- early '00s. I would totally be excited to work for that Google. It's just that, well, there are lots of awesome things to do, and I had (what seemed like at the time) a more interesting option. I sometimes had regrets and sometimes not. Second process was maybe 2-3 years ago. It didn't get to a full onsite, since after a few conversations with the team, it was clear there wasn't a fit. The old arrogance was still there, but without the same sharpness or cleverness. I spoke to a team working on a new product (under NDA) in a field I had a lot of experience in, and: 1) There were computer scientists without a clear understanding of the target market domain. 2) They believed they were the best-and-brightest, and didn't need to consult experts in the field. 3) There was a lot of hype and salesmanship. I like to be surrounded by smart people, and it felt like I'd be the sharpest guy in that room. I was a better computer scientist, AND better domain expert in the field the product was in than anyone on that team. That's not a position I want to find myself in. And I have a job which pays a lot less than Google, but I'm surrounded by smart people I learn a heck of a lot from, where I'm working on meaningful things, and having fun. I'm also continuing to build my personal brand. Now a few years later, the product they were working on never shipped, so if I worked for Google, I would have likely been on a failed project. On the other hand, my mortgage would likely be paid off.
• Turbots 6y ago
  Youtube is working if you're not logged in It's their authentication system I think
• kilon 6y ago
  youtube greece is down returns "oops" error but i can still playback a 2 hour video that I was already watching before it went down. Trying to open any other video or channel returns this error.
• rataata_jr 6y ago
  Down in India
• adriancooney 6y ago
  Google Home and Chromecast also rendered useless. _Hey Google, how are you?_ leads to some very passive aggressive replies.
  • lxgr 6y ago
    In my house, it leads to a hilarious echo of multiple devices responding simultaneously with an error message. Having only one device respond seems to be cloud-mediated too.
  • AntiImperialist 6y ago
    "Hey Google!" "Something went wrong. Try again in a few seconds."
• mmillin 6y ago
  Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.
  • mrcus 6y ago
    Nope, especially considering the implications of this, with the amount of people working remotely. Google Meet, Classroom, etc. are down. This is probably literally costing billions every minute just in loss of productivity.
    • ants_a 6y ago
      Total world economic output is ~$150M / minute, so billions every minute is off by few orders of magnitude.
      • rocho 6y ago
        That figure seems way too low, what are your sources on it?
        • HPsquared 6y ago
          World GDP is $80 trillion per year.
        • rakoo 6y ago
          World GDP was ~$90B last year (https://databank.worldbank.org/data/download/GDP.pdf https://databank.worldbank.org/data/download/GDP.pdf), which averages to ~$150M/minute
          • vultour 6y ago
            That's trillion not billion
            • 6y ago
• alarmingfox 6y ago
  Only thing working for me is the Google search engine
• cerw 6y ago
  Google Login down in Australia
• tibbydudeza 6y ago
  It seems 2020 is not done yet :).
• fny 6y ago
  Can I just say I'm remarkably relieved I have an excuse to kick back for the next hour and do nothing?
• risyachka 6y ago
  Google play web also down along with others.
• Aeolos 6y ago
  Can confirm, gmail, gdrive and youtube are down.
• ingen0s 6y ago
  same just lost access to all services including Firebase here in Toronto Canada
• glenstein 6y ago
  Also Google Voice. I ctrl+F'd for that one, and it's one thing I haven't found a way to easily degooglefy from. Also down.
  • lxgr 6y ago
    I wonder if this means Google Fi is affected too? And I also wonder how many Google engineers might not even be getting paged right now as a result.
• thegeomaster 6y ago
  It looks like the auth service is broken. Everything works when logged out. Attempting to log in gives "Account not found". EDIT: Looks like they logged everyone out while they fix the auth issue, presumably so people can use YouTube and other stuff that doesn't necessarily need login?
• shimonabi 6y ago
  And just on the day that electoral college certifies the election. Great timing!
• omadalosso 6y ago
  Down in Brazil
• olso 6y ago
  "Couldn't find your Google Account"
• jbverschoor 6y ago
  Also in europe
• lucidlogic 6y ago
  Down in South Africa
• ciccionamente 6y ago
  It's also weird the page title with "Error 500 (Server Error)!!1" - Why !!1 at the end of it?
  • wffurr 6y ago
    Someone forgot to hold down thr shift key when typing the third mark? Secret status code to differentiate types or server errors? A joke?
  • joeraut 6y ago
    It's probably a running joke internally---anecdotally, I've came across it a few times on different Google properties over the years.
  • Andrex 6y ago
    Internet jargon; when a commenter is overly excited, they tend to overuse exclamation points. In the rush to add many exclamation points, the shift key might not be held down for all of them. So you end up with things like "OMG!!!!111"
• javindo 6y ago
  The fact it seems to be related to login reminds me of the Tom Scott doomsday fictional lecture which I would link to but you know...
  • partyguy 6y ago
    Oh yeah that talk was crazy!!
    • rgrau 6y ago
      It's been ages (literally) since I felt so helplessly out of a conversation not being able to check a reference instantaneously. Weird feeling...
  • graposaymaname 6y ago
    https://www.youtube.com/watch?v=y4GB_NDU43Q https://www.youtube.com/watch?v=y4GB_NDU43Q Works fine once you log out.
    • ytch 6y ago
      opening it with Incognito mode works too
  • xendergrunge 6y ago
    Nice. very subtle reference to our dependability on google products. you can still link the YT lecture from a private session.
• jakozaur 6y ago
  All down, but all green. New norm, let's have status page, but make it green all the time :-). Surprisingly consistent practice.
• geomark 6y ago
  Gmail and Drive down in Thailand.
• Galanwe 6y ago
  Everything down in HK
• av8avenger 6y ago
  Switzerland's down too. It works if you access it without session though.
• adychandra 6y ago
  Down in india As well, Gsuite, GCP IAM, Youtube, Calendar, Drive et al.
• wappa 6y ago
  Not working for me either, Canada
• TheWellKnownEIP 6y ago
  Can confirm from the Netherlands, Hangouts also down.
• hpeinar 6y ago
  Seems to also affect Google Cloud console to some extent. Dashboard data is not loading at the moment. Not sure what services exactly are affected there. Google Cloud SQL seems to be up and running so far.
• jsnell 6y ago
  Seems to work when you're not signed in. Won't help with gmail and docs, but maybe a workaround for YouTube and search.
• AgentK20 6y ago
  And somehow Pokemon Go is down, too. They use GCP for hosting, but I'm not seeing other outages on GCP. Maybe they have deeper integration with Google's Cloud than publicly disclosed?
  • jrockway 6y ago
    Discord was sort of down (wouldn't load), but I could log in again a few minutes later and everything was OK. My company's product is hosted on GCP and there are no problems that I can detect without administrative access (which I can't obtain because it's behind a Google login); services are returning correct results.
  • Benjamin_Dobell 6y ago
    Discord is GCP and down also.
  • tajchert 6y ago
    Google Maps for sure, maybe some GCP as well (as console is down, Discord and Playstation Network as well are having issues - which seems like some GCP service)
  • ytch 6y ago
    It works if login with facebook or Apple ID, so GCP itself is fine, but I guess admin cannot login to management
• MamaJumba 6y ago
  Interesting that I have been watching a long video (over an hour long) and the buffer is still loading while YouTube in general is down for me
• sschueller 6y ago
  Google is also rejecting SMTP ingress traffic. (delivery temporarily suspended: lost connection with aspmx.l.google.com[2a00:1450:400c:c04::1b] while sending DATA command)
  • Andrex 6y ago
    That is Really Bad™. People are going to miss critical emails with no recovery.
    • r1ch 6y ago
      These are temporary deferrals, a properly configured outbound mailserver should retry for days.
    • rasguanabana 6y ago
      Fortunately email was designed back in a day when mail servers could fail. Don’t worry - proper SMTP servers will retry patiently.
    • bartvk 6y ago
      Emails will be delayed but sending SMTP servers have a retry mechanism up to a few days.
• dannyw 6y ago
  Looks like if google services are down for more than 45 minutes, Workspace customers gets 3 days added.
• jacquesm 6y ago
  Google meet/hangouts, appliances, the lot.
• kongo23 6y ago
  Same in Ukraine
• shoshino 6y ago
  Google Pay contactless transaction not working in UK news.google.com returning 500 finance.google.com returning 502
  • opal9 6y ago
    oof i didn't even think of google pay
• pragmaticpandy 6y ago
  Jarring reminder of how addicted I am to YouTube... getting childishly irritated that I can't continue to binge lol
  • thunderrabbit 6y ago
    Use duckduckgo to search for videos and you can watch most of them right on the results page.
• Mtntk 6y ago
  Maps are partially affected, page review returns 500, now even location pages doesn't open
• durnygbur 6y ago
  Migrated out od G infrastructure long time ago. The G Search working fine, G Maps down (continental EU). HN very laggy.
• dreamer7 6y ago
  I was kicked out of a Google Meet call just now
• speedgoose 6y ago
  Looks like some Google Cloud customers are offline too.
• darkof 6y ago
  Same situation here at Liman 1.
• Mistri 6y ago
  It's super interesting that all Google services that I've tried are down _except_ for Google Search. What would isolate Search from the rest of Google's products such that it wouldn't be affected by a mass outage like this?
  • dreamer7 6y ago
    It is something related to authentication. Even YouTube works without being logged in
    • easytiger 6y ago
      When logging into any of my gmail personal/free and corporate paid for accounts i get "Couldn't find your google account" in red writing
  • fishbacon 6y ago
    It looks like it is related to authentication, landing pages seem to be working, so going incognito you can still use youtube. I would guess the search page does not actually use your credentials for anything since fingerprinting is separate from login?
    • Mistri 6y ago
      I would've thought Google remembers what you search while logged in and sells it off. Maybe they do it in a different way - never really thought about that. I'm logged into Google when I search but my profile fails to load at the top right.
      • fishbacon 6y ago
        They probably do, but it is not essential to the functionality so it can work even if auth fails. Fingerprinting happens separately because then they can identify you even if you are not using your own computer or are in incognito. Login is just a convenient way for you to tell them who you are.
  • 6y ago
• netcrash 6y ago
  UAE - down
• dilsmatchanov 6y ago
  incognito youtube is working btw
• pupdogg 6y ago
  I’m seeing poor internet performance overall, even accessing HN...too many relying on Google DNS maybe? Even ISPs?
  • lxgr 6y ago
    Probably just people checking non-Google sites whether it's everyone or just them. 8.8.8.8 at least seems to be working, as does Google DNS for domains hosted on domains.google.com.
• xchaotic 6y ago
  youtube / gmail down in Central Europe too
• tartrate 6y ago
  This is a good time to think about what would happen if you lost access to your Google account.
• jacquesm 6y ago
  So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
  • deleted 6y ago
    [deleted]
  • coldtea 6y ago
    >So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Much less chance of that happening than my local backups getting borked...
    • jacquesm 6y ago
      But much higher than the chances of both of them getting borked.
  • Turbots 6y ago
    What if your datacenter explode? Will you lose nothing? Is everything backed up/replicated to the second/third one? Because I think that's a bigger possibility than Google not coming back.
  • bonoboTP 6y ago
    I guess a lot of people are fine with the risk. Everybody uses it, so if, like, Gmail loses all the emails, we are then in such a state that the consequences will be more bearable and socially normal. Most people are fine with accepting that whatever future thing will happen to most people will also happen to them. Because then the consequences will also be normal. If the apocalypse comes, it comes for almost all of us and that's consolation enough.
    • mizzao 6y ago
• neals 6y ago
  How's things like Stadia? Or is anybody using more exotic Google stuff? Like, how are their selfdriving cars doing?
  • dkarp 6y ago
    Stadia is down for me
  • gooses 6y ago
    Stadia is down.
• donohoe 6y ago
  Appears to be only affecting pages that have an authenticated session; Youtube works in a private window, but not in window with cookies.
• deleted 6y ago
  [deleted]
• fode 6y ago
  All service down in Sénégal as well. I first noticed Youtube, then everything else followed.
• nedsma 6y ago
  The worst outage I've ever seen with Gmail. I can't log in, getting "Couldn't find your Google account" message. Also, with the already logged in account, I can't browse anything but whatever is in cache. I'm definitely worried.
• dark_ph0enix 6y ago
  Same in the UK.
• chrisked 6y ago
  Most of google services are offline.
• petercooper 6y ago
  You say "outage", Google will probably say "sunsetting" or "deprecated."
• rayrag 6y ago
  killedbygoogle.com is still working. ;)
• walrus01 6y ago
  regular text search is working for me right now, but gmail and maps are totally broken. google drive/docs also totally non functional.
• ionwake 6y ago
  At this point Amazon and Google should just have a static page with green buttons for their status page. They NEVER show red, or oddly even go down themselves during outages. ( Last time amazon went down springs to mind).
• caarlos0 6y ago
  GCP seems to be down as well.
• mattisxy 6y ago
  Some stuff works if you are unauthenticated.
• pupdogg 6y ago
  I’m seeing poor internet performance overall, even accessing HN...too many relying on Google DNS maybe? Even ISPs?
  • petemir 6y ago
    Well, HN's auto-hug of death. Also, probably most sites can't load either analytics nor ads...
• fode 6y ago
  All Google services down in Sénégal as well.
• Benjamin_Dobell 6y ago
  Google Cloud outage? Looks like Discord is down too. Might be a coincidence.
• speeder 6y ago
  Google is scaring the shit out of us, we were already trying to move out of Google for some past bizarre outages, and now before they went down, they outright claimed business account didn't exist anymore, our business drive and all our IMAP e-mail clients got kicked out.
• magdyks 6y ago
  same here from Egy
• paganel 6y ago
  Gmail, Docs and YT down for me too in Romania.
• r4fterman 6y ago
  Google page is right: servers are up and running - but not reachable. Could be an infrastructure attack (DNS?) or someone just broke the cable :)
• nevi-me 6y ago
  YouTube Music failed in an interesting way. My music has been playing, but I started hearing ads (I'm on Premium), which prompted me to check what was going on.
  • rafaelflores 6y ago
    I am still listening to youtube bot on discord... Maybe it will crash soon.
  • dgb23 6y ago
    That supports the claim about it being an auth issue.
• mickuehl 6y ago
  Google Workplace (EU) is down and also Google Cloud (login at least)
• jamesblonde 6y ago
  Somebody threw a Spanner in the works is my guess. Global-scale database underpinning everything (or maybe the giant that is colossus is having an off-day).
• henearkr 6y ago
  As a consequence, many websites depending on Google services are stuck too. Like the nytimes.com. Ah btw HN is being crushed by the load lol.
• h43k3r 6y ago
  Everything is affected. Its Gaia(Google Auth) outage, most probably. Disclaimer - Googler here whose workplace chat is not working.
  • LandR 6y ago
    I can't login to anything via Google Auth this morning. It reports error that my Google Account can't be found. I thought my account had been deleted!
  • gfitz 6y ago
    The nature of this downtime is quite severe. How are Googlers resolving this if their internal communications are down?
    • BossingAround 6y ago
      There are other ways to deliver packets... https://en.wikipedia.org/wiki/IP_over_Avian_Carriers https://en.wikipedia.org/wiki/IP_over_Avian_Carriers
    • Gigablah 6y ago
      I recall that their SREs use IRC?
      • q3k 6y ago
        Yep. At least some of the teams do.
        • geogra4 6y ago
          not hangouts? allo? duo?
          • q3k 6y ago
            I don't think Allo and Duo are products designed for use in teams/organizations, or marketing themselves as such.
• timack 6y ago
  I cant even access my Google+ at the moment!
• buro9 6y ago
  At times like this I am thankful of running a Google takeout periodically. Not because I use it as a live failover for Google services, but because these outages are good reminders of the mortality of online services.
  • gzer0 6y ago
    How long did your takeout take? It’s been 3 weeks and mine either says not ready yet, or when it is it will say unable to download and force me to start again...
    • buro9 6y ago
      When I request a full takeout it usually takes 2-4 weeks to generate. This time around it was missing a .tgz which had error'd. A subsequent full takeout request took a couple of days. The .tgz files I find can only be reliably extracted on a Linux... the Gmail export for me is 20GB which in the last few exports has been a single file in the last .tgz
• bwb 6y ago
  We've got about 3,000+ people a minute hitting Downforeveryoneorjustme.com, pretty crazy.
• js4ever 6y ago
  This topic is so hot it's crashing HN! Super long server response time and I get this error quite oftenly: "We're having some trouble serving your request. Sorry!"
• shimonabi 6y ago
  The Youtube maintenance monkey is wearing a MAGA hat.
• acorna920 6y ago
  Down also from Italy
• shashankjain16 6y ago
  HN also seems to be loading slow with sudden inflow of traffic. Would be funny if HN goes down because of Google outage
• xabush 6y ago
  Youtube, Gmail and Drive pages open in incognito mode but can't sign in. The issue seems with the account manager.
• beshrkayali 6y ago
  Fastmail is still up!
• mrkickling 6y ago
  It is possible to reach youtube in incognito mode, so it seems to be a google accounts issue.
  • tr1ll10nb1ll 6y ago
    exactly right
• lucb1e 6y ago
  But why is HN so slow? Pages take like 30+ seconds to load for me (German vantage point, other sites are fine). Does it timeout on some Google dependency or try to use a Google submarine cable or something? Or is everyone just posting to HN about it?
  • RupertEisenhart 6y ago
    they arn't prepared for such big news, you are part of the involuntary DDOS
  • deleted 6y ago
    [deleted]
  • emirhan 6y ago
    I guess it's the demand of everyone checking HN to see if Google is down or not.
  • roboyoshi 6y ago
    I think that is simply because all HN readers using some google service jump over here to check, because google's own statuspage is useless.
  • kfrzcode 6y ago
    If memory serves, HN is hosted on a relatively lightweight server configuration.
  • rk06 6y ago
    It is called HN hug of death (or Reddit hug of death). the traffic due to google outage is much more than usual
• mcintyre1994 6y ago
  Discord also seems to have gone down for me. Also (probably unrelated?) HN is really slow compared to normal right now.
• jamie_ 6y ago
  All G Suite down for me, says "Couldn't find your Google Account". Broke youtube mid-stream.
• kschiffer 6y ago
  Guys, chill out! We're DOS'ing HN already!
• adarshaj 6y ago
  It's up in Japan
  • henearkr 6y ago
    Down for me. Have you tried Youtube while being logged in?
• mendyberger 6y ago
  Incognito seems to be working
• Jamie9912 6y ago
  HN down?
• a012 6y ago
  Even HN is being hugged by death of Google
• pablogancharov 6y ago
  Down in Uruguay too
• mendyberger 6y ago
  Looks like Hacker News is having trouble as well
• darkcha0s 6y ago
  Hackernews is also having some kind of stability issues, either everyone is on here complaining about the google outages, or somehow they share infra.
• panda4126 6y ago
  GSuite all down. Confirmed in Japan.
• kokotkohaha 6y ago
  Down in Slovakia
• sdan 6y ago
  HN is also down because everyone is checking whether Google is down
  • remus 6y ago
    Turns out hacker news has an indirect dependency on google!
• punnerud 6y ago
  Seems like the auth solution that is down. Every service work in incognito/private. Shared drives in Docs/Drive does not work. Probably connected to the auth service?
• fattybob 6y ago
  yup, all down in Vietnam too, this page took some time to appear too, so i suspect a lot of issues in the pipe (mostly via China i gather)
• deleted 6y ago
  [deleted]
• hankchinaski 6y ago
  is it me or I noticed that google search rarely goes down? is it somewhat in an isolated system or a fundamentally different architecture?
• dhruvarora013 6y ago
  Amazing how many products are clearly getting affected due to this. The top 18 products on downdetector.com are all Google-owned or related
• sciurus 6y ago
  This affects GCP as well as Google's consumer services. I'm here because I was paged about low traffic for a service. Turns out to just be that I can't access any metrics at all from Google Cloud Monitoring (i.e. Stackdriver).
• lambalungi 6y ago
  https://www.chrisranjana.com https://www.chrisranjana.com why
• max_ 6y ago
  My service accounts for the play store are all returning err 500.
• AdamGibbins 6y ago
  Word from inside google is their auth is down, so anything that doesn't require auth should work fine. Log out of youtube, works fine.
• xyst 6y ago
  Even my personal e-mail (using google workspace) is not accessible Did some asshole forget to renew a certificate or run the cron job to renew?
• purpleidea 6y ago
  Everything down in Canada. At least everything that seems to require auth. Gmail, Gsuite, Youtube and more.
• deleted 6y ago
  [deleted]
• XCSme 6y ago
  Blogger is also down, even for public site access so it's not only auth related.
• deleted 6y ago
  [deleted]
• davidhq 6y ago
  http://zetaseek.com http://zetaseek.com -- install locally and search your data in private, connect to friends and search theirs.
• deleted 6y ago
  [deleted]
• tleb_ 6y ago
  I hope it lasts a bit, not just five minutes. We might finally question our dependency on Google services.
• mozinator 6y ago
  edit: added details edit: redacted my phone number edit: big mistake to add phone number edit: I think illic is right, probably not me edit: removed details
  • dannyw 6y ago
    It’s probably not you.
    • contravariant 6y ago
      Although since it seems to be back up now I suppose there is one way we could find out for sure.
  • rubatuga 6y ago
    What are IAM permissions?
    • kristopolous 6y ago
      Just permissions. The "IAM" can be safely dropped. It's exactly what you think it'd be: restrictions and privileges. "IAM" is basically the name for a specific model of doing it. Unless something really crazy happened, this user is unlikely to be correct. Accounts are supposed to be firewalled/sandboxed in a way that you can't contagion across to someone else's let alone systemwide. It's possible (some sweeping script on a powerful connection that smashes just the right things or some exploit to break the sandboxing), just probably not likely - especially unintentionally. But crazier things have happened https://books.google.com/books?id=rRp7DkTegMEC&newbks=0&printsec=frontcover&pg=PA127 https://books.google.com/books?id=rRp7DkTegMEC&newbks=0&prin...
    • bgilroy26 6y ago
      It's what Amazon calls your cloud login account Identity and Access Management (IAM)
  • lalwanivikas 6y ago
    What exactly did you do that makes you so confident?
• lutoma 6y ago
  Seems to be auth related. Login does not work, and Youtube is also throwing Internal Server Errors, but only if I'm signed in.
• dsachin 6y ago
  So called 2020...
• joseluisq 6y ago
  Curious that minutes ago I was listening music on YT. BTW not all services are down.
  • das_keyboard 6y ago
    Me too. But it started playing ads because apparently I wasn't logged in anymore... So it works but without "using" your account.
• LoveMortuus 6y ago
  youtube-dl is still able to download videos, which is quite interesting ^^
• olcor 6y ago
  Apparently Google "can't find" my account. I was so scared for a while, then I found that all of my family members have the same issue. Then checked HN and breathed a sigh of relief :)
• rhysd 6y ago
  Seems that Google has no budget to pay their GCP instances
• GoogleUser123 6y ago
  Cool way to end 2020.... Pandemic and now no more Google. Thank you God :)
• itoocode 6y ago
  Looks like the authentication services are having problem. I have account not found error.
• Raphty 6y ago
  YouTube work for me in Firefox just fine. But I have it configured with some of the recommendations from Privacytools.io
  • rplnt 6y ago
    That's interesting, Youtube never works for me in Firefox just fine. Video playback gets stuck somewhat regularly, requiring FF restart. But yeah, the page loads for me as well :)
• zeospec 6y ago
  I hate the fact that the status page is not reflecting anything related to it.
• ZoomZoomZoom 6y ago
  I'm very interested in HN' crowd back-of-the-envelope estimations on how much this outage costs the world (when we know how long it lasted).
• carlhjerpe 6y ago
  Seems like services does indeed work in incognito. https://news.ycombinator.com/item?id=20132520 https://news.ycombinator.com/item?id=20132520 Maybe this beast kicked the bucket.
• olsn 6y ago
  Allright, what did Gilfoyle do this time? ...
• alvis 6y ago
  HN outage is soon to be the next one. The HN server is overwhelmed by the traffic due to Google’s outrage. :/
• alexhayes 6y ago
  I feel kinda bad - I'm pretty sure I broke Google as I was uploading quite a few large files via Google Drive and it was chugging pretty hard. And now because of that it seems HN is also hurting. I'm sorry internet.
• rwx------ 6y ago
  same here - Mars
• netdur 6y ago
  I think they are out of hard drive space.
• soumyadeb 6y ago
  HN is loading very slowly too for me. Related to Google outage?
  • xendergrunge 6y ago
    Sortof, HN is slow because after the outage, a LOT of people are on HN. The servers seems overloaded.
• aritmo 6y ago
  All services are still down, https://www.google.com/appsstatus#hl=en&v=status https://www.google.com/appsstatus#hl=en&v=status Crisis averted, was not locked out. Gotta take out all the data though when it gets back up.
• palmeida 6y ago
  Seems the issue is related to the google authentication system since if logged out you can visit youtube without error
• DebtDeflation 6y ago
  Just got a text from my kids' school saying GMail is down and to use the school's direct email. Immediate reaction was "Google isn't down you idiots, the problem is on your end", go to check GMail, yep it's down.
  • fouronnes3 6y ago
    yeah this is like saying it must be a compiler bug
    • yreg 6y ago
      I had a few of these this year, e.g. thinking I have local connectivity issues when it turned out the entire backbone is in a bad shape[0]. Or that time when 80% of the apps on my iPhone crashed on launch[1] — must be an issue with the device, right? [0] Level 3 Global Outage - https://news.ycombinator.com/item?id=24322861 https://news.ycombinator.com/item?id=24322861 [1] Facebook iOS SDK Remotely Crashing Spotify, TikTok, Pinterest, Winno and More - https://news.ycombinator.com/item?id=23097459 https://news.ycombinator.com/item?id=23097459
• sgt 6y ago
  Luckily nobody is using Google services for anything critical. Or what, are you?
• vineyardmike 6y ago
  Help! My Waymo taxi won't open the doors without me logging into the app. Its driving around in circles on route 500 and won't stop. /s - for now ;)
  • xdavidliu 6y ago
    how's the converted offshore oil platform treating you?
  • derwiki 6y ago
    I know you’re joking, but I am curious if Waymo’s fleet was affected. Shouldn’t be right? But I’m also surprised every time I fresh login to Gmail and YouTube shows up in the intermediary redirect chain.
    • deleted 6y ago
      [deleted]
• ricc 6y ago
  Looks like the problem is with Accounts. Most Google apps work when not logged in, i.e. can view YouTube videos.
• arvidkahl 6y ago
  This puts into sharp relief how outages and downtime can hit ANYONE. Cheers to all the small SaaS businesses out there keeping their services up and running without much of a hick-up all year round.
• ulzeraj 6y ago
  Gcloud/GKE command line utils are also having problems. Was deploying some test applications and kubectl started complaining about gcloud auth helper throwing not zero errors. Trying to launch cloud shell from the website and nothing happened. The web application which is a site that does not rely on any external API is running fine.
• Uberphallus 6y ago
  Drive and Gmail seem to be back for me (12:31 GMT)
• madmax108 6y ago
  Atlassian's #HugOps page[1] collates messages of support for Devops as a public embedded Google Map and it really encapsulates the current feeling! https://imgur.com/a/3RurTYf https://imgur.com/a/3RurTYf Don't forget to #HugOps all the people who've been woken up on a monday morning with this! Hope this gets resolved soon :) [1] https://www.atlassian.com/software/statuspage/hugops https://www.atlassian.com/software/statuspage/hugops
• shashwat986 6y ago
  It's up again!
• kofblade 6y ago
  Here in Brazil, YouTube has just up!
• known 6y ago
  https://viewdns.info/ismysitedown/?domain=google.com https://viewdns.info/ismysitedown/?domain=google.com points to 172.217.1.46
• pg_bot 6y ago
  Gmail and Youtube are back online for me. Hangouts is beach balling. There is a warning at the top of my screen though. "Gmail is temporarily unable to access your Contacts. You may experience issues while this persists."
• deleted 6y ago
  [deleted]
• thret 6y ago
  Is Google Auth down because of the SolarWinds Orion exploit? https://cyber.dhs.gov/ed/21-01/ https://cyber.dhs.gov/ed/21-01/
  • deleted 6y ago
    [deleted]
• tasqa 6y ago
  Looking at the other comments, it seems highly likely zanzibar[1] is down. Their auth service. [1] PDF: https://research.google/pubs/pub48190.pdf https://research.google/pubs/pub48190.pdf
  • felixr 6y ago
    zanzibar is an authorization service, and we are looking auth as in authentication here So no, not zanzibar
• deleted 6y ago
  [deleted]
• neilshevlin 6y ago
  Seems to back up. Most services working outside of incognito
• dannyr 6y ago
  Even Google Sign In service is down. Sites like Airbnb, Duolingo, etc that uses Google Sign In can't log in their users.
• maxpert 6y ago
  Whataya know Youtube didn’t work and I came to HN to check what happened.
• ibraheemmmoosa 6y ago
  Hmm. My magical spell works.
• bkantt 6y ago
  Seems to be back up (Europe/NL).
• Foivos 6y ago
  At least Google DNS (8.8.8.8) still works. So, the people who use it, still have access to the rest of the Internet.
• rd07 6y ago
  Seems like they are up again
• hkai 6y ago
  I wonder why services like Uber (which rely on Google Maps for everything are still working, with things like routing being completely okay).
• leonidasv 6y ago
  We use Google Cloud and Firebase at work and our application and servers are still up and running. Can't say the same about the Cloud Admin Console, tho.
• ttn 6y ago
  I did not realize how dependent I was to google/google services until 30 minutes ago.
• jakemacale 6y ago
  Firebase is down too.
  • jakemacale 6y ago
    Firebase projects are working but can't go to dashboard.
• PersonalHitman 6y ago
  Hmmm google is back... wonder if this was an attack or if this was an internal problem with something?
• carlosr2 6y ago
  ads seem to be working though
• asdojasdosadsa 6y ago
  Seems to work now
• contravariant 6y ago
  The problems seem to have been (mostly) resolved over here, at least the authentication seems to work again.
• soygul 6y ago
  HN is faster and more accurate than status pages of major service providers themselves. Special thanks to people working at those service providers and posting here!
• tibbydudeza 6y ago
  It is back again.
• CephalopodMD 6y ago
  I wonder how on earth googlers are supposed to fix this when they can't log in to make changes...
• kristopolous 6y ago
  These rare events help make extremely visible how much of our general basic infrastructure we've farmed out to a small handful of companies, centralizing something that was supposed to avoid the problems of centralization.
• johanam 6y ago
  seems to be working again, for me. except gmail says "Gmail is temporarily unable to access your Contacts. You may experience issues while this persists."
• hathym 6y ago
  The cool thing is that newpipe is working while youtube is down.
  • memorysafety 6y ago
    Hey, are you updating newpipe from F-Droid? I do, and it stopped working for me weeks ago.
• deleted 6y ago
  [deleted]
• isaacimagine 6y ago
  Google is back up
• blntechie 6y ago
  All services back up for me.
• bgdam 6y ago
  I really hope this is just a temporary issue and no accounts are going to be lost. If that happens, way too many people are going to feel the pain of thinking Google makes backups unnecessary - including me.
• deleted 6y ago
  [deleted]
• kokotkohaha 6y ago
  Gmail service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.
• curiousllama 6y ago
  Pour one out for the SREs...
• oknoorap 6y ago
  It's time to decentralized Gmail, Youtube, and Google Docs, maybe their Search Engine too.
  • sn41 6y ago
    That's what it was 15 years back. I was slightly concerned when Google Mail and Chat were integrated. Look where we are now. Right from "The battle of the red cliffs", we will never learn the disadvantage of lashing too many boats together.
    • oknoorap 6y ago
      But our tech right now is far more advanced than 15 years ago. We have IPFS, Blockchain, dat protocol today. I think it's possible to kill the giants. Even, Tim Berners-Lee want to decentralized the web: https://techcrunch.com/2018/10/09/tim-berners-lee-is-on-a-mission-to-decentralize-the-web/ https://techcrunch.com/2018/10/09/tim-berners-lee-is-on-a-mi...
      • diggan 6y ago
        Even? Tim Berners-Lee tried since day 0 to make the web decentralized. One of the initial requirements in the 1989 proposal for a global hypertext system included "Non-Centralisation" where "new system must allow existing systems to be linked together without requiring any central control or coordination". See https://www.w3.org/History/1989/proposal.html https://www.w3.org/History/1989/proposal.html for the rest. While it went OK-ish for the internet, we massively screwed it up for the web.
        • thu2111 6y ago
          No we didn't. The web is very nicely decentralised, especially with the general malaise and decline of Facebook. What isn't decentralised is web apps but TBL never intended the web to be used for full blown desktop app replacements in the first place so no surprise it doesn't meet its design goals for that.
• mkl95 6y ago
  How much do you think Google can learn from this incident's postmortem? Writing down all the internal details probably amounts to a small book.
• aabbcc1241 6y ago
  Time to mention the alternative, a decentralized internet, e.g. https://zeronet.io https://zeronet.io
• sidcool 6y ago
  It's restored now.
• ransom1538 6y ago
  "all green, which does not reflect reality for me (e.g. Gmail is down)" This isn't true. Which make it more complicated as we try to figure out what systems are down -- since their status board is usually faked. gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed: - Internal Error
• hackernews1134 6y ago
  4:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their services or I am encountering underutilized servers since it is not fixed for everyone. It is nice to experience (even if it is short lived) the snappiness of Google services if they weren't so multi-tenented.
  • rubatuga 6y ago
    Oh man, you're right. Bloated gmail loaded instantly. What's going on? It's loading almost 2x to 3x faster.
    • deleted 6y ago
      [deleted]
    • ithkuil 6y ago
      Isn't this a good indication that the performance problem if gmail may not be related to the "bloat" of the frontend itself?
      • yjftsjthsd-h 6y ago
        It might suggest that the frontend isn't the only issue, at least - and maybe this explains why it's usually so slow, if the frontend can be fast on a fast enough backend. On the other hand, the speed of the "basic HTML" version implies that the frontend can be the issue.
        • savanaly 6y ago
          Entirely possible as well that the "basic HTML" uses different API service in the background that are snappier for comparative lack of users.
    • 6y ago
• itvision 6y ago
  First the two Gmail accounts I had in Thunderbird logged me out and asked to relogin. Then Google said my accounts did NOT exist which made me feel very uneasy. Banned? Lost all my data? OMG. Then I got error "502 That's all we know" after entering my passwords. Finally I realized it couldn't be just me, so I opened HN and confirmed my suspicions. It's all up and running now but I was really scared. I'm really curious what happened because it surely looks like Google has a single point of failure in their authentication mechanism which is just horribly wrong. The company has over two billion users - a situation/configuration like this just shouldn't be even theoretically possible.
• deleted 6y ago
  [deleted]
• notorandit 6y ago
  Going back to normal.
• mkl95 6y ago
  How does this incident affect IoT devices? I'd expect many of them to be using Firebase just like HN.
  • JorgeGT 6y ago
    People were reporting not being able to turn their lights on: https://twitter.com/joemfbrown/status/1338452107419148290 https://twitter.com/joemfbrown/status/1338452107419148290
• ransom1538 6y ago
  GOOGLE UPDATE YOUR GCP STATUS PAGE. Stop making us run around trying to figure out what systems are down. Running: timeout 60 gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed: - Internal Error
• mstipetic 6y ago
  Now we'll have a bit of outrage, talking about how dangerous it is to have the whole world impacted by one company's processes and forget about it in 2 days
• pdkl95 6y ago
  As many comments in this thread have mentioned, the crash seems to happen when when logged in; youtube and other services seem to work when don't send the session cookie. This suggests something very fundamental related to user account/sessions is failing... Did Maria Christensen make a mistake when adding "return true;"? (This is a joke referencing Tom Scott's 2014 parable[1] about the danger of designing a system with a single point of failure. Tom's tells the fictional tale of a high level employee at Google adding "return true;" to the global "handleLogin()" function.) [1] https://www.youtube.com/watch?v=y4GB_NDU43Q https://www.youtube.com/watch?v=y4GB_NDU43Q (you might need to open this link in a private window...)
• l33tman 6y ago
  Services are not restored. Some came up again, some not. My Gsuite business mail is still completely down, while youtube started working again. I'm pretty sure there will be some internal conferences at Google after this to make sure infrastructure problems can't propagate across the entire company and world at this rate even in the event of a sysop fatfingering a console...
  • woofie11 6y ago
    ... staffed by MBAs. :)
  • taf2 6y ago
    If you are still down try restarting app... this worked for me
• hartator 6y ago
  Google Search seemed to be also down: https://serpapi.com/status https://serpapi.com/status
• jamespetercook 6y ago
  A good explainer video here: https://www.youtube.com/watch?v=5p8wTOr8AbU https://www.youtube.com/watch?v=5p8wTOr8AbU
• Jumziey 6y ago
  It's impressive how fast they are at fixing outages, barely a coffee break required. I'm almost starting to believe in this DevOps thing...
  • keithnoizu 6y ago
    Still down for me. Must not be a "vast majority of users" bit of a premature "green" on that.
• exhaze 6y ago
  Seems like the incident lasted from 04:00 AM PST to 04:30 AM PST, at least for BigQuery (sorry, image timestamps are in JST, since I'm in Japan): https://imgur.com/a/kiHED6v https://imgur.com/a/kiHED6v
  • sweezyjeezy 6y ago
    Definitely started before the hour.
• PascLeRasc 6y ago
  [deleted]
  • ryankrage77 6y ago
    > I wonder if it relies on some cloud service /s, right? right?!
• varispeed 6y ago
  Guys, Orkut also seems to be down!
• robertlagrant 6y ago
  Maps could be the scariest one! All those drivers wondering how to get where they're going.
• kabes 6y ago
  When I tried to login it said: 'this account does not exist'. So my first thought was some algorithm made a mistake and my account got deleted for no reason. I already imagined the only solution now was to write a medium post and hope it gets some traction on hackernews and google support steps in. Thinking to myself I was an idiot for knowing all this and still thinking it wouldn't happen to me. And even though it turns out to be an outage, it gave me a bad enough feeling to start using a domain name I own for my email.
  • burnthrow 6y ago
    Those hosting with their own domains on Google mail had an identical experience, if it makes you feel differently.
    • archi42 6y ago
      The advantage of a custom domain name is: As long as you can update DNS records, you can have your mail easily hosted elsewhere. Obviously not relevant for this kind of outage, but in the scenario outlined by GP - Google randomly kills you off, and there is nothing you can do - this is at least an emergency strategy.
    • kabes 6y ago
      Yep, but losing access to my emails isn't that big of a deal, losing access to my email address is much worse. Especially since it's also my login on a lot of other services.
• totaldude87 6y ago
  Has google.com ever suffered such outages in recent past?
• optimalsolver 6y ago
  >drive not working Something something someone else's computer.
• satysin 6y ago
  Waiting for the post-mortem to see it was all because someone forgot to renew an SSL cert that nobody knew was a single point of failure for literally everything. :D
• unixhero 6y ago
  The appstatus page from Google is a good indicator for me of which products are currently live from Google. Uhm, more of a sidenote :)
• jerrygreenest 6y ago
  Not just google, hackernews was also displaying: "We're having some trouble serving your request. Sorry!", has it something to do with google architecture? Btw, that's funny you listed all the google services exactly in the order I found them unavailable: first youtube, then gmail, then docs
• ffpip 6y ago
  2000 points in an hour on HN. 120K upvotes on down detector. Trending worldwide with 500K tweets.
• hbt 6y ago
  FYI if you have cronjobs using Google apis, disable them or delay them. a bot is flagging ips as abusive. it should clear up later.
• woj-tek 6y ago
  Not a fan or even their user but kudos to Google for admitting the failure. Amazon (at least AWS) customers would have probably been shown a green dashboard and told to review their code.
• nunodonato 6y ago
  So happy that I didnt even notice this, because I stopped using Google's services :)
  • derwiki 6y ago
    Kudos! What do you use instead of YouTube? That’s the main product keeping me on the franchise
    • nunodonato 6y ago
      You mean as a consumer or as a creator?
      • derwiki 6y ago
        Consumer
• eric_khun 6y ago
  If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_calculation https://en.wikipedia.org/wiki/High_availability#Percentage_c...
  • api 6y ago
    SLAs are largely bullshit.
    • kaushikt 6y ago
      why?
      • gsich 6y ago
        In this example: you get free days. Which depending on your business might be worthless if you have suffered more monetary loss due to the downtime than the free days are worth.
        • SifJar 6y ago
          But still better than nothing. And for some (most?) people/businesses, probably worth more than any resulting monetary loss
        • Cthulhu_ 6y ago
          Exactly; downtime doesn't cost a cloud service much. At worst it causes reputation damage, with possibly large companies deciding to go for a competitor, losing a contract worth tens or hundreds of millions.
      • fullstop 6y ago
        Not OP, but how do you measure them? Let's say, for example, you can send and receive email, but attaching files does not work. Is the service up or down? What if the majority of your users can access the service, but one of your BGP peers is not routing properly and some of your users are unable to access?
• va1en0k 6y ago
  looking forward to reading the postmortem
• 1hakr 6y ago
  So to easily fix such down time, i have alerts setup for website, server and API using https://simpleops.io/ https://simpleops.io/ After that I realised how frequently you can get down time
• kaushikt 6y ago
  Calculating cost of downtime ==> World economic output is - $150mn/minute Current world population is 7.8bn. Google has 4.8bn users. That's approx 61% of the population. Let's assume about 50% of the users were impacted = 2.4bn (which is 30% of the population). So, the loss could have been about $50mn/minute. This is without taking SLA into consideration. There will be losses incurred on that too, wouldn't it?
• deleted 6y ago
  [deleted]
• beyondzero 6y ago
  Maybe google had to quickly remove solarwinds from their network. :)
• brandmeyer 6y ago
  Singalong: She broke the whoooole world, with her change, he broke the whole wide world, with his change, they broke the whole world with their chaaaaange! I'm sure its stressful right now. But someday, these engineers will look back and retell the stories about how it happened and the lessons they learned. Hopefully with a laugh.
• nojvek 6y ago
  Don’t blame people, blame systems and processes. I assume Google has a blameless process too. If an engineer can bring down huge swaths of google that’s not a human problem. As an eng your you should heavily invest in a sane test -> deploy -> monitoring process, and reward reliability. Give people a bonus when things didn’t break, not only when there is a superhero that fixes broken things. Then you’re rewarding fragile systems that need superheroes.
• xtracto 6y ago
  On reddit thread there are some really good jokes about this [1] related to their asinine interview questions. Like: > Did they try to fix them by inverting a binary tree? >> Yeah maybe implementing a quick LRU cache on the nearest whiteboard will help them out here >> Did they try checking what shape their manhole cover is? >> Dev ops was too busy out counting all the street lights in the United States [1] https://www.reddit.com/r/programming/comments/kcwqij/every_single_google_service_is_currently_out/gft1x2j/ https://www.reddit.com/r/programming/comments/kcwqij/every_s...
  • mav3rick 6y ago
    None of these are asked in G interviews. The commenters are asinine.
    • xtracto 6y ago
      No? https://twitter.com/mxcl/status/608682016205344768?lang=en https://twitter.com/mxcl/status/608682016205344768?lang=en
      • mav3rick 6y ago
        Okay ? No one's discounting his achievements. What's wrong with knowing stuff about a binary tree ? Is ignorance a badge of honor now ?
        • rococode 6y ago
          Yeah, nothing wrong with not having memorized how to do that - it's certainly not often useful in practice. But if you're given the definitions and a couple minutes to figure it out (which, in an interview, you are), and you actually can't come up with the 10 lines of code to do it, maybe that question is working as intended after all...
          • reallydontask 6y ago
            > But if you're given the definitions and a couple minutes to figure it out (which, in an interview, you are), and you actually can't come up with the 10 lines of code to do it, maybe that question is working as intended after all... or maybe the person struggles to work things out in an extremely stressful situation I speak as somebody that finds interviews Extremely stressful and if working there is regularly my interview levels of stress then please do reject me.
• rollschild 6y ago
  Anyone experiencing lost emails? I'm not sure if it's actually me messing things up or because of the Gmail outage, I couldn't find an extremely important email and I've tried archive, trash, deleted, spam, sent boxes..
• mgiampapa 6y ago
  Gmail is currently returning "550 5.1.1 The email account that you tried to reach does not exist" for many valid gmail addresses.
• mszone 6y ago
  This is a great and interesting thread. But I believe nobody discussed here the opposite and postivie effects of this kind of golbal outages. There are funny calculations for example of having less spam mail open-clciks or things like that. So think in such massive otuage of millions of users and auatomated process what means in term to generate less CPU, RAM, I/O and as consecuence less Power/Cooling >> less CO2 and pollution... It's huge! and at the Earth's weather impacts is like charge trillions in advance that we''ll have to paid in a future not only with money... (Economy Degrowth)