7 ms·

Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time.

stronglikedan 7y ago
Says something that they know the what, where, when, and why, but the still don't (or won't) prevent it.
- bena 7y ago
  It's a matter of cost. It's a temporary spike of traffic far outside of average use. In order to handle it, they'd need a lot of resources that just aren't justifiable for the rest of the year. You seem similar things when places sell highly desired items in online stores.
  - vorpalhex 7y ago
    They're a bleeding cloud provider. I think they can temporarily scale up a bit.
    - bena 7y ago
      Temporarily build a data center? Temporarily shut down other people's services to favor their own?
      - lbatx 7y ago
        I guess you're not familiar with how cloud services work. There's (generally) plenty of excess capacity.
        bena 7y ago
        It's the (generally) that's the issue here, now isn't it?
        vorpalhex 7y ago
        I didn't see any notifications that GCP was out of capacity. Did I miss one?
        londons_explore 7y ago
        Gcp capacity and capacity for Google services is fairly isolated. They never coexist on the same racks
        lbatx 7y ago
        Doubtful. I put that in since absolute statements are always dangerous. I'd say at any given time, there's an excess of 20-50%. There's virtually no chance they were lacking capacity.
        londons_explore 7y ago
        Within Google, there is plenty of wasted capacity, but getting capacity for a service to scale up is a fiendishly difficult task. They even employ hundreds of people whose main role is to try to allocate all the various types of resources to teams. "Oh - you want 1700 GB of bigtable in Reliability zone 3 in Atlanta? I'm afraid your department doesn't own any there - you can trade with the ads department who aren't using theirs, if you give them 2Tbits of network bandwidth between Peru and Brazil? That trade will only be till the end of the quarter though, because then they need it back."
pugworthy 7y ago
Sounds pretty anecdotal. Can you back up this claim?
- jordanthoms 7y ago
  We run a app making millions of API calls to Google every day, so they show up in our monitoring, even if they don't make the status page. It's a pattern I've noticed from getting paged at 3am (NZ time) this time of year going back the last 4 years, I don't have hard data at hand for it though. It's not the same thing every year either - this time OAuth got hit hard, previously we've mostly seen slowdowns and higher error rates on the Drive APIs.
khc 7y ago
I thought that's because this is the time when interns' code get deployed

stronglikedan 7y ago

Says something that they know the what, where, when, and why, but the still don't (or won't) prevent it.

bena 7y ago
It's a matter of cost. It's a temporary spike of traffic far outside of average use. In order to handle it, they'd need a lot of resources that just aren't justifiable for the rest of the year. You seem similar things when places sell highly desired items in online stores.
- vorpalhex 7y ago
  They're a bleeding cloud provider. I think they can temporarily scale up a bit.
  - bena 7y ago
    Temporarily build a data center? Temporarily shut down other people's services to favor their own?
    - lbatx 7y ago
      I guess you're not familiar with how cloud services work. There's (generally) plenty of excess capacity.
      - bena 7y ago
        It's the (generally) that's the issue here, now isn't it?
        vorpalhex 7y ago
        I didn't see any notifications that GCP was out of capacity. Did I miss one?
        londons_explore 7y ago
        Gcp capacity and capacity for Google services is fairly isolated. They never coexist on the same racks
        lbatx 7y ago
        Doubtful. I put that in since absolute statements are always dangerous. I'd say at any given time, there's an excess of 20-50%. There's virtually no chance they were lacking capacity.
      - londons_explore 7y ago
        Within Google, there is plenty of wasted capacity, but getting capacity for a service to scale up is a fiendishly difficult task. They even employ hundreds of people whose main role is to try to allocate all the various types of resources to teams. "Oh - you want 1700 GB of bigtable in Reliability zone 3 in Atlanta? I'm afraid your department doesn't own any there - you can trade with the ads department who aren't using theirs, if you give them 2Tbits of network bandwidth between Peru and Brazil? That trade will only be till the end of the quarter though, because then they need it back."

pugworthy 7y ago

Sounds pretty anecdotal. Can you back up this claim?

jordanthoms 7y ago
We run a app making millions of API calls to Google every day, so they show up in our monitoring, even if they don't make the status page. It's a pattern I've noticed from getting paged at 3am (NZ time) this time of year going back the last 4 years, I don't have hard data at hand for it though. It's not the same thing every year either - this time OAuth got hit hard, previously we've mostly seen slowdowns and higher error rates on the Drive APIs.

khc 7y ago

I thought that's because this is the time when interns' code get deployed