3 ms·

Interested to hear how many "hands on deck" this gets vs a Google Cloud or Google Workspace outage. Don't fuck with the money!

stingraycharles 4y ago
I like to think that at a certain point, more “hands” doesn’t help solve the issue faster (maybe even counter productive), so that in both circumstances the outages get the same amount of hands — the maximum.
hacym 4y ago
Does GCP not make Google money?
- leoh 4y ago
  You wouldn’t think so based on the quality of their support team
- coder543 4y ago
  No. GCP consistently loses money. One of many sources: https://www.theregister.com/2022/07/27/alphabet_q2_2022/ https://www.theregister.com/2022/07/27/alphabet_q2_2022/ > Google Cloud brought in $6.3 billion of revenue, a 35 percent year on year increase compared to Q2 2021’s $4.6 billion revenue. But the $858 million loss was 45 percent higher than the $591 million deficit recorded last year. You might ask: "how is that possible, with those same margin-intensive prices that AWS and Azure use?" I have no clue. It makes no sense, given the prices they charge.
  - sidibe 4y ago
    What is or isn't cloud is different at every company on top line but especially bottom line. For example how far down the stack are engineers still considered cloud when they are supporting both cloud and internal products? Investors are fine losing money on growing cloud and also love to see ad margins up? Then everyone is cloud.
    - coder543 4y ago
      I don't think salaries are the driving factor behind GCP's balance sheet, but that's just my opinion. I obviously can't wave a magic wand and find out for sure either, but my own back of the napkin math can't see that reasonably working out. They brought in billions of additional revenue, but their losses also increased.
  - zaphirplane 4y ago
    Similar reasons to a company with 30 tps 8am - 6pm has a 3 h/a Kafka running on a k8s cluster, multi master MySQL cluster, all the trimmings Built to scale that isn’t here today ? Resume Padding ? Data centers are expensive and slow unto build
- why_only_15 4y ago
  GCP revenue losses from being down are approximately lower bounded by (downtime in seconds * revenue per second). Their costs don't decrease from being down.
- xyst 4y ago
  Labor costs to keep all of these features/servers/infra running is expensive. Google is probably charging less due to competition from AWS/Azure/Oracle. Similar to how Amazon.com is subsidized by AWS. GCP is subsidized by the ad biz
more_corn 4y ago
P0, page everybody. But their incident response planning and training is on point so there’s one incident commander and everyone doing individual tasks reports to them. I suspect not more than a dozen cooks are in the kitchen as it were. Any more and people would be getting in each other’s way. I seem to recall that ads runs on one of the largest distributed MySQL instances on the planet. That’s probably the most complicated component at play here. Then again everything has to have active, secondary and tertiary so maybe it’s something upstream like dns dos protection or cdn. Not a Google employee, just guessing.

stingraycharles 4y ago

I like to think that at a certain point, more “hands” doesn’t help solve the issue faster (maybe even counter productive), so that in both circumstances the outages get the same amount of hands — the maximum.

hacym 4y ago

Does GCP not make Google money?

leoh 4y ago
You wouldn’t think so based on the quality of their support team
coder543 4y ago
No. GCP consistently loses money. One of many sources: https://www.theregister.com/2022/07/27/alphabet_q2_2022/ https://www.theregister.com/2022/07/27/alphabet_q2_2022/ > Google Cloud brought in $6.3 billion of revenue, a 35 percent year on year increase compared to Q2 2021’s $4.6 billion revenue. But the $858 million loss was 45 percent higher than the $591 million deficit recorded last year. You might ask: "how is that possible, with those same margin-intensive prices that AWS and Azure use?" I have no clue. It makes no sense, given the prices they charge.
- sidibe 4y ago
  What is or isn't cloud is different at every company on top line but especially bottom line. For example how far down the stack are engineers still considered cloud when they are supporting both cloud and internal products? Investors are fine losing money on growing cloud and also love to see ad margins up? Then everyone is cloud.
  - coder543 4y ago
    I don't think salaries are the driving factor behind GCP's balance sheet, but that's just my opinion. I obviously can't wave a magic wand and find out for sure either, but my own back of the napkin math can't see that reasonably working out. They brought in billions of additional revenue, but their losses also increased.
- zaphirplane 4y ago
  Similar reasons to a company with 30 tps 8am - 6pm has a 3 h/a Kafka running on a k8s cluster, multi master MySQL cluster, all the trimmings Built to scale that isn’t here today ? Resume Padding ? Data centers are expensive and slow unto build
why_only_15 4y ago
GCP revenue losses from being down are approximately lower bounded by (downtime in seconds * revenue per second). Their costs don't decrease from being down.
xyst 4y ago
Labor costs to keep all of these features/servers/infra running is expensive. Google is probably charging less due to competition from AWS/Azure/Oracle. Similar to how Amazon.com is subsidized by AWS. GCP is subsidized by the ad biz

more_corn 4y ago

P0, page everybody. But their incident response planning and training is on point so there’s one incident commander and everyone doing individual tasks reports to them. I suspect not more than a dozen cooks are in the kitchen as it were. Any more and people would be getting in each other’s way. I seem to recall that ads runs on one of the largest distributed MySQL instances on the planet. That’s probably the most complicated component at play here. Then again everything has to have active, secondary and tertiary so maybe it’s something upstream like dns dos protection or cdn. Not a Google employee, just guessing.