10 ms·

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a

• mrcus 6y ago
  Nope, especially considering the implications of this, with the amount of people working remotely. Google Meet, Classroom, etc. are down. This is probably literally costing billions every minute just in loss of productivity.
  • ants_a 6y ago
    Total world economic output is ~$150M / minute, so billions every minute is off by few orders of magnitude.
    • rocho 6y ago
      That figure seems way too low, what are your sources on it?
      • HPsquared 6y ago
        World GDP is $80 trillion per year.
      • rakoo 6y ago
        World GDP was ~$90B last year (https://databank.worldbank.org/data/download/GDP.pdf https://databank.worldbank.org/data/download/GDP.pdf), which averages to ~$150M/minute
        • vultour 6y ago
          That's trillion not billion
          • jrh206 6y ago
            https://en.wikipedia.org/wiki/Billion https://en.wikipedia.org/wiki/Billion A billion is a number with two distinct definitions: - 1,000,000,000, i.e. one thousand million, or 10^9, as defined on the short scale. This is now the meaning in both British and American English. - 1,000,000,000,000, i.e. one million million, or 10^12, as defined on the long scale. This is one thousand times larger than the short scale billion, and equivalent to the short scale trillion. This is the historical meaning in English and the current use in many non-English-speaking countries where billion and trillion 10^18 maintain their long scale definitions. Nevertheless almost everyone uses 1B = 10^9 for technical discussions
            • ikt 6y ago
              This is a financial discussion though so: https://www.worldometers.info/gdp/gdp-by-country/ https://www.worldometers.info/gdp/gdp-by-country/ World's GDP is $80,934,771,028,340 (nominal, 2017). https://www.wolframalpha.com/input/?i=%2480%2C934%2C771%2C028%2C340 https://www.wolframalpha.com/input/?i=%2480%2C934%2C771%2C02... $80.93477102834 trillion Nobody would argue world GDP is anything billion, that's crazy.
              • selectodude 6y ago
                https://fr.wikipedia.org/wiki/Liste_des_pays_par_PIB_nominal https://fr.wikipedia.org/wiki/Liste_des_pays_par_PIB_nominal In France, they use milliard and billion.
          • rakoo 6y ago
            Sorry, language mistake. The result is the same: GDP is ~$150M/minute
          • kerng 6y ago
            That depends where in the world you are!
      • ascar 6y ago
        Simple math says: World GDP (via Google) $80,934,771,028,340 Minutes per year 365 * 24 * 60 = 525,600 Divide and you get 153,985,485
    • tzs 6y ago
      You are assuming that a minute of disruption can not cause more than a minute's loss of productivity. I don't think that assumption is justified. Consider an exactly one minute outage that affects multiple things I use for work. First, I may not immediately recognize that the outage is actually with some single service provider. If several things are out I'm probably going to suspect it is something on my end, or maybe with my ISP. I might spend several minutes thoroughly checking that possibility out, before noticing that whatever it was seems to have been resolved. Second, even if I immediately recognize it for what it is and immediately notice when it ends it might take me several minutes to get back to where I was. Not everything is designed to automatically and transparently recover from disruptions, and so I might have had things in progress when the outage stuck that will need manual cleanup and restarting.
      • ants_a 6y ago
        I'm also assuming most of the world doesn't grind to a halt when gmail is down. Crops keep growing and factories keep running.
        • itsrajju 6y ago
          Even software engineers who are in a state of flow keep working :)
      • deleted 6y ago
        [deleted]
    • yashap 6y ago
      Indeed. Also, Google’s revenue is about $300K per minute. The value they provide is likely higher than that, but as you said, being able to send an email an hour later than you hoped it’s fine in most cases. Also, Google Search was fine, and that’s their highest impact product. I’d guess actual losses to the world economy were more on the order of about $100K per minute, or about 1/3 of Google’s revenue. MAYBE a few hundred thousand per minute, though that seems unlikely with Search being unaffected, and everything else coming back. Certainly a far cry from billions per minute :)
  • optimalsolver 6y ago
    I was unable to watch the Mogwai - Autorock music video. :-(
  • PeterStuer 6y ago
    Besides the exaggerated figure, I always find these claims bizarre. Sure, there was some momentary loss, but aggregated over a month this will not even register.
  • demosito666 6y ago
    I never understood this type of calculation as it implies that time is directly converted into money. However, I struggle to come up with an example for this. Even the most trivial labor cases like producing paperclips don't seem to be directly converting time into profit: even you will make 10k units instead of 100k this hour, you don't sell them immediately. They bring revenue to the firm via a long chain of convoluted contracts (both legal and "transactional") which are very loosely coupled to the immediate output. Nothing is operating at minute margins unless it's explicitly priced on a minutely basis, like a cloud service. Even if a worked on a conveyor belt can't produce paperclips without looking at Google Docs sheet all the time, this will be absorbed by the buffers down the line. And only if the worker will fail to meet her monthly target due to this, loss of revenue might occur. But in this case the service has to be down for weeks. In case of more complex conversions of time into money, like in the most of intellectual work, this is even less obvious that short downtimes will cause any measurable harm.
• reddotX 6y ago
  yeah, not working in Europe
• leonidasv 6y ago
  Makes sense, at work we have an application running on Google Cloud and everything seems to be working. So the outage is probably not at network or infrastructure level.
• ojosilva 6y ago
  If this does not improve soon, we're looking at one of the most significant outages in recent internet history, at least from the number of people impacted.
• tarruda 6y ago
  > I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
  • ignoramous 6y ago
    Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).
    • robertlagrant 6y ago
      Who automates the automators? :)
    • papito 6y ago
      Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.
      • ignoramous 6y ago
        Yes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.
        • qz2 6y ago
          Apart from the ones that they haven't worked out yet :)
          • 6y ago
• ilikehurdles 6y ago
  Coincidentally, Google Authenticator was finally just updated on iOS after many years without update.
• beyondcompute 6y ago
  I am not sure why are they allowing it. Meaning why aren’t services completely isolated? Isn’t it obvious that in an intertwined environment those things are bound to happen (as in “question of when, not if”)? I understand, in smaller companies that are limited in resources (access to good developers and pressure to get product to market as soon as possible) we have single points of failure all over the place. But “the smartest developers on the planet”? What is it if not short-sighted disregard for risk management theories and practices? I mean, Calendar and Youtube, say, should be completely separate services hosted in different places, their teams should not even talk to each other. Yes, they can use same software components, frameworks and technologies. Standardization is very welcome. But decentralization should be an imperative. Edit: again downvotes started! Thanks to everyone “supporting freedom of expression” :)
• ikiris 6y ago
  Went to reply, then saw the username. My guess was lb layer
• Corrado 6y ago
  In a previous lifetime I removed an "unused" TLS certificate. It turns out that it was a production cert that was being used to secure a whole state's worth of computers. In my defence, the cert was not labeled properly, nor was it used properly, and there was no documentation. It took us 2 days to create a new cert and apply it to our software and deliver it to the customer. Those were 2 days I'll never get back. However, when I was finished the process was documented and the cert was labeled, so I guess its a win.
• Diederich 6y ago
  Several others have shared their 'I broke things' experiences, and so I feel compelled to weigh in. Many years ago, I was directly responsible for causing a substantial percentage of all credit/debit/EBT authorizations from every WalMart store world-wide to time out, and this went on for several days straight. On the ground, this kind of timeout was basically a long delay at the register. Back then, most authorizations would take four or five seconds. The timeout would add more than 15 seconds to that. In other words, I gave many tens of millions of people a pretty bad checkout experience. This stat (authorization time) was and remains something WalMart focuses quite heavily on, in real time and historically, so it was known right away that something was wrong. Yet it took us (Network Engineering) days to figure it out. The root cause summary: I had written a program to scan (parallelized) all of the store networks for network devices. Some of the addresses scanned were broadcast and network addresses, which caused a massive amplification of return traffic which flooded the satellite networks. Info about why it took so long to discover is below. Back in the 1990s, when this happened, all of the stores were connected to the home office via two way Hughes satellite links. This was a relatively bandwidth limited resource that was managed very carefully for obvious reasons. I had just started and co-created the Network Management team with one other engineer. Basically prior to my arrival, there had been little systematic management of the network and network devices. I realized that there was nothing like a robust inventory of either the networks or the routers and hubs (not switches!) that made up those networks. We did have some notion of store numbers and what network ranges were assigned to them, but that was inaccurate in many cases. Given that there were tens of thousands of networks ranges in question, I wrote a program creatively called 'psychoping' that would ICMP scan all of those network ranges with adjustable parallelism. I ran it against the test store networks, talked it over with the senior engineers, and was cleared for takeoff. Thing is, I didn't start it right away; some other things came up that I had to deal with. I ended up started it over a week after review. Why didn't this get caught right away? Well, when timeouts started to skyrocket across the network, many engineers started working on the problem. None of the normal, typical problems were applicable. More troubling, none of the existing monitoring programs looked for ICMP at all, which is what I was using exclusively. So of course they immediately plugged a sniffer into the network and did data captures to see what was actually going on. And nothing unusual showed up, except a lot of drops. We're talking > 20 years ago, so know that "sniffing" wasn't the trivial thing it is now. Network Engineering had a few extremely expensive Data General hardware sniffers. And to these expensive sniffers, the traffic I was generating was invisible. Two things: the program I wrote to generate the traffic had a small bug and was generating very slightly invalid packets. I don't remember the details, but it had something to do with the IP header. These packets were correct enough to route through all of the relevant networks, but incorrect enough for the Data General sniffer to not see them. So...there was a lot of 'intense' discussions between Network Engineering and all of the relevant vendors. (Hughes, ACC for the routers, Synoptics and ODS for the hubs) In the end, a different kind of sniffer was brought in, which was able to see the packets I was generating. I had helpfully put my userid and desk phone number in the packet data, just in case someone needed to track raw packets back to me. Though the impact was great, and it scared me to death, there were absolutely no negative consequences. WalMart Information Systems was, in the late 1990s, a very healthy organization.
• robotnikman 6y ago
  I've been in that situation before at one of my previous jobs, where some important IT infrastructure when down for the whole company. Nowhere as big of a scale as this, but it was easily one of the most stressful moments of my life