5 ms·

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the

As bad as this article makes AWS sound, it's actually the reason you should go with AWS over say Azure or GCP; when AWS goes down, its owners actually feel the pain with you, Microsoft and Google run their own stuff elsewhere...

nodesocket 5y ago
Source? I'd be surprised if Google does not dog food. Though, I guess when GCP had their recent global load balancer outage but neither Gmail or Google search went down maybe not.
- shadowgovt 5y ago
  Most of Google's stuff runs on Borg, which predates GCP (and is the fabric GCP runs on top of).
  - nodesocket 5y ago
    Borg is the predecessor to Kubernetes (sort of) right? They could have switched to Kubernetes and run on Google Kubernetes Engine (GKE).
    - shadowgovt 5y ago
      I'm a bit out of the loop, but the vibe for years was "It isn't broke, so we're not going to rewrite it to run on something else." Actually, it's a bit more than that. Some half-decade ago, Google Cloud got stung by a coordinated attack over the holidays where attackers used stolen credit cards to build a net of GCE instances and do an attack on Tor via endpoint control. Cited by SRE in the postmortem was the relative immaturity of the cloud monitoring, logging, and "break glass" tools that SREs were accustom to in Borg... Essentially, Cloud didn't have the maturity of framework that Borg did and they felt the extra layer of abstraction complicated understanding and stopping the abuse of the service. This report had a chilling effect internally. Whereas management had previously been encouraging people to migrate to Cloud as quickly as possible, after this incident software engineering teams and the SREs that supported them were able to push back with "Can we trust it to be as maintainable as what we already have?" and put cloud on the defensive to prove that hypothesis.
tw04 5y ago
> Microsoft and Google run their own stuff elsewhere... That is simply not true. https://www.zdnet.com/article/microsoft-moves-closer-to-running-all-of-its-own-services-on-azure/#:%7E:text=Microsoft%20committed%20a%20decade%20ago,t%20actually%20hosted%20on%20Azure.&text=%22Azure%20is%20the%20cloud%20platform,cloud%20services%2C%20including%20Microsoft%20Teams https://www.zdnet.com/article/microsoft-moves-closer-to-runn...
- deleted 5y ago
  [deleted]
- aenis 5y ago
  true for google though, with their borg-thingy being separate from gcp last time i asked them
  - mghfreud 5y ago
    Gcp runs on top of borg. (Gcp failure does not mean borg failure though)
zeusk 5y ago
Well you definitely don't understand Azure or GCP then. Office 365, Teams, Dynamics, xcloud, xbox live - they all run on Azure.
pyrophane 5y ago
No, not true at all. What do you think Google and Microsoft run their cloud services on?
- admax88qqq 5y ago
  Google does not run the bulk of their services on GCP. Unlike AWS, GCP was not a productization of their existing infrastructure, but rather a separate cloud product developed fairly independently. I'm sure that's changing with time. YouTube also has its own infrastructure independent of GCP and the rest of Google
  - marcan_42 5y ago
    GCP is not separate from Google's core infrastructure; rather, it's built on top of it. That means that while you can certainly have GCP specific outages, this kind of core infra "everything is down" situation is almost guaranteed to hit everything, GCP and not included. A lot of GCP sub-products are productionizations of existing Google tech; e.g. BigQuery is a public version of Dremel, an internal database/query engine they'd been using internally for a while. I'm pretty sure YouTube hasn't had their own infra in quite a while. When I was there ~8 years ago I think it was all integrated already. Certainly database, video processing, storage, CDN were all on core Google infra, and I'm sure the frontends were too though I don't remember looking into that explicitly.
    - ec109685 5y ago
      There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis. That’s the big difference between the two clouds level of dog fooding.
      - Jensson 5y ago
        > There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis. Doing something on top of GCP rather than the normal way at Google was a huge pain. Borg tutorials and documentation were just far superior, I could get a thing running on borg in an hour from not knowing anything about borg, I spent a week trying to get something running internally on GCP but still couldn't get it right (our team wanted to see if we could run things on GCP so I was tasked with testing it, I couldn't find anyone who knew how to do it so we just gave up after I didn't make any real progress). That was the worst documented thing I've ever worked with. And even worse the internal GCP pages were probably running in california and probably weren't tested from Europe, so the page took like 2 seconds between mouse click and it responded to anything. That was years ago though and I no longer work there, but at least back then the work to make using GCP internally seamless wasn't done. Maybe it is simpler if you run everything in it and don't need it play well with borg, but there is a reason why it isn't popular internally. And likely you wont find many engineers who left Google who recommend you will use it, since they probably didn't test it and if they did it probably was a bad experience (unless they worked on GCP).
reasonabl_human 5y ago
This is simply not true. Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure. Also, having consumed both cloud service offerings, customer support within Azure was far more responsive. Source: used to work as a SWE on a flagship Azure service
- avh02 5y ago
  > Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure so it's even more embarrassing how terrible the software is. trolling aside - if this is true, the state of e.g: MS Teams is a travesty. Implementation of replies to messages implemented in 2021!? So many bugs, etc. it seriously damages productivity. And don't get me started on sharepoint.
  - avh02 5y ago
    Since it seems I need to back up my statement: https://news.ycombinator.com/item?id=29492884 https://news.ycombinator.com/item?id=29492884 is one example... Is that teams' fault? No, but they'd be able to call 911 if it wasn't installed. I have a running list of teams bugs/flaws/inferiorities, it's currently about 30 items, will probably publish it sometime soon
johncena33 5y ago
HN is a very pro-Amazon place. Regardless of what Amazon does, at least one of the top 3 comments is always justifying Amazon's actions.
- vineyardmike 5y ago
  That’s not the impression I got from the last 24 hours, nor the last few years. 1. Scale is hard and downtime is hard, HNers either recognize the struggle or appreciate their lack of experience. When AWS fails many armchair architects come out to suggest solutions but many more techies just sympathize with the Amazonians. 2. Technically, Amazon has built something impressive. It might not be what you want, or what others have, but AWS is impressive in scale and scope and even reliability. Many people share credit where due. 3. One can criticize the treatment of warehouse and delivery workers that Amazon is known for, but this has little baring on the tech workers there nor AWS generally. So AWS stories tend to be free from the social critique the company as a whole receives.
  - joe_chip 5y ago
    [flagged]
cush 5y ago
> Microsoft and Google run their own stuff elsewhere... Where on Earth did you learn that?

nodesocket 5y ago

Source? I'd be surprised if Google does not dog food. Though, I guess when GCP had their recent global load balancer outage but neither Gmail or Google search went down maybe not.

shadowgovt 5y ago
Most of Google's stuff runs on Borg, which predates GCP (and is the fabric GCP runs on top of).
- nodesocket 5y ago
  Borg is the predecessor to Kubernetes (sort of) right? They could have switched to Kubernetes and run on Google Kubernetes Engine (GKE).
  - shadowgovt 5y ago
    I'm a bit out of the loop, but the vibe for years was "It isn't broke, so we're not going to rewrite it to run on something else." Actually, it's a bit more than that. Some half-decade ago, Google Cloud got stung by a coordinated attack over the holidays where attackers used stolen credit cards to build a net of GCE instances and do an attack on Tor via endpoint control. Cited by SRE in the postmortem was the relative immaturity of the cloud monitoring, logging, and "break glass" tools that SREs were accustom to in Borg... Essentially, Cloud didn't have the maturity of framework that Borg did and they felt the extra layer of abstraction complicated understanding and stopping the abuse of the service. This report had a chilling effect internally. Whereas management had previously been encouraging people to migrate to Cloud as quickly as possible, after this incident software engineering teams and the SREs that supported them were able to push back with "Can we trust it to be as maintainable as what we already have?" and put cloud on the defensive to prove that hypothesis.

tw04 5y ago

> Microsoft and Google run their own stuff elsewhere... That is simply not true. https://www.zdnet.com/article/microsoft-moves-closer-to-running-all-of-its-own-services-on-azure/#:%7E:text=Microsoft%20committed%20a%20decade%20ago,t%20actually%20hosted%20on%20Azure.&text=%22Azure%20is%20the%20cloud%20platform,cloud%20services%2C%20including%20Microsoft%20Teams https://www.zdnet.com/article/microsoft-moves-closer-to-runn...

deleted 5y ago
[deleted]
aenis 5y ago
true for google though, with their borg-thingy being separate from gcp last time i asked them
- mghfreud 5y ago
  Gcp runs on top of borg. (Gcp failure does not mean borg failure though)

zeusk 5y ago

Well you definitely don't understand Azure or GCP then. Office 365, Teams, Dynamics, xcloud, xbox live - they all run on Azure.

pyrophane 5y ago

No, not true at all. What do you think Google and Microsoft run their cloud services on?

admax88qqq 5y ago
Google does not run the bulk of their services on GCP. Unlike AWS, GCP was not a productization of their existing infrastructure, but rather a separate cloud product developed fairly independently. I'm sure that's changing with time. YouTube also has its own infrastructure independent of GCP and the rest of Google
- marcan_42 5y ago
  GCP is not separate from Google's core infrastructure; rather, it's built on top of it. That means that while you can certainly have GCP specific outages, this kind of core infra "everything is down" situation is almost guaranteed to hit everything, GCP and not included. A lot of GCP sub-products are productionizations of existing Google tech; e.g. BigQuery is a public version of Dremel, an internal database/query engine they'd been using internally for a while. I'm pretty sure YouTube hasn't had their own infra in quite a while. When I was there ~8 years ago I think it was all integrated already. Certainly database, video processing, storage, CDN were all on core Google infra, and I'm sure the frontends were too though I don't remember looking into that explicitly.
  - ec109685 5y ago
    There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis. That’s the big difference between the two clouds level of dog fooding.
    - Jensson 5y ago
      > There aren’t 100k Googlers developing on top off GCP services to get their job done on a daily basis. Doing something on top of GCP rather than the normal way at Google was a huge pain. Borg tutorials and documentation were just far superior, I could get a thing running on borg in an hour from not knowing anything about borg, I spent a week trying to get something running internally on GCP but still couldn't get it right (our team wanted to see if we could run things on GCP so I was tasked with testing it, I couldn't find anyone who knew how to do it so we just gave up after I didn't make any real progress). That was the worst documented thing I've ever worked with. And even worse the internal GCP pages were probably running in california and probably weren't tested from Europe, so the page took like 2 seconds between mouse click and it responded to anything. That was years ago though and I no longer work there, but at least back then the work to make using GCP internally seamless wasn't done. Maybe it is simpler if you run everything in it and don't need it play well with borg, but there is a reason why it isn't popular internally. And likely you wont find many engineers who left Google who recommend you will use it, since they probably didn't test it and if they did it probably was a bad experience (unless they worked on GCP).

reasonabl_human 5y ago

This is simply not true. Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure. Also, having consumed both cloud service offerings, customer support within Azure was far more responsive. Source: used to work as a SWE on a flagship Azure service

avh02 5y ago
> Microsoft dog foods tons of its own software and all internal cloud services I know of run on azure so it's even more embarrassing how terrible the software is. trolling aside - if this is true, the state of e.g: MS Teams is a travesty. Implementation of replies to messages implemented in 2021!? So many bugs, etc. it seriously damages productivity. And don't get me started on sharepoint.
- avh02 5y ago
  Since it seems I need to back up my statement: https://news.ycombinator.com/item?id=29492884 https://news.ycombinator.com/item?id=29492884 is one example... Is that teams' fault? No, but they'd be able to call 911 if it wasn't installed. I have a running list of teams bugs/flaws/inferiorities, it's currently about 30 items, will probably publish it sometime soon

johncena33 5y ago

HN is a very pro-Amazon place. Regardless of what Amazon does, at least one of the top 3 comments is always justifying Amazon's actions.

vineyardmike 5y ago
That’s not the impression I got from the last 24 hours, nor the last few years. 1. Scale is hard and downtime is hard, HNers either recognize the struggle or appreciate their lack of experience. When AWS fails many armchair architects come out to suggest solutions but many more techies just sympathize with the Amazonians. 2. Technically, Amazon has built something impressive. It might not be what you want, or what others have, but AWS is impressive in scale and scope and even reliability. Many people share credit where due. 3. One can criticize the treatment of warehouse and delivery workers that Amazon is known for, but this has little baring on the tech workers there nor AWS generally. So AWS stories tend to be free from the social critique the company as a whole receives.
- joe_chip 5y ago
  [flagged]

cush 5y ago

> Microsoft and Google run their own stuff elsewhere... Where on Earth did you learn that?