3 ms·

Most companies are not Netflix and I’m not sure I understand why we are discussing AWS? That post is more a statement on how errors which can be handled at the

Most companies are not Netflix and I’m not sure I understand why we are discussing AWS? That post is more a statement on how errors which can be handled at the app layer can have catastrophic effects on lower level components. You cannot assume end customers are running thing at the scale or with the fault tolerance of google, or Netflix.

joshuamorton 7y ago
Certainly, but you're making the assumption that Google does assume that. Its completely possible to have different alerting setups for different binaries, and consider CPU throttling to not be a page-worthy issue for GFEs, but to be page-worthy for GCP jobs.
jldugger 7y ago
Most companies are not Netflix but all cloud customers can learn from their public design discussions. The only reason I mentioned AWS is that they are Netflix a high profile AWS customer, and their lessons in cloud architecture apply pretty cleanly to GCP. You cannot assume an SLA of 100 percent, even if it works out that way on shorter time scales. It's really no different than running your own datacenter, so I don't know where this 'monitoring will prevent catastrophy' fight is coming from. > You cannot assume end customers are running thing at the scale or with the fault tolerance of google, or Netflix. Correct, but there's a gradient between 'we have 10 copies of the service in 10 different countries and use Akamai GTM in case of outage' and Dave's one-off-VM. One-off VMs are fine if you know what you're getting into, and I use that setup for my personal, lowstakes & zero revenue website. But if you are a paying cloud customer, it makes sense to pay attention to availability zones regardless of scale. And sure, there might be a market somewhere for a more durable VM setup. At a past non-profit job we provided customers the illusion of a single HA VM using Ganeti (http://www.ganeti.org/ http://www.ganeti.org/). But it's not clear to me that the segment is viable -- customers at the low and top end don't need the HA.

joshuamorton 7y ago

Certainly, but you're making the assumption that Google does assume that. Its completely possible to have different alerting setups for different binaries, and consider CPU throttling to not be a page-worthy issue for GFEs, but to be page-worthy for GCP jobs.

jldugger 7y ago

Most companies are not Netflix but all cloud customers can learn from their public design discussions. The only reason I mentioned AWS is that they are Netflix a high profile AWS customer, and their lessons in cloud architecture apply pretty cleanly to GCP. You cannot assume an SLA of 100 percent, even if it works out that way on shorter time scales. It's really no different than running your own datacenter, so I don't know where this 'monitoring will prevent catastrophy' fight is coming from. > You cannot assume end customers are running thing at the scale or with the fault tolerance of google, or Netflix. Correct, but there's a gradient between 'we have 10 copies of the service in 10 different countries and use Akamai GTM in case of outage' and Dave's one-off-VM. One-off VMs are fine if you know what you're getting into, and I use that setup for my personal, lowstakes & zero revenue website. But if you are a paying cloud customer, it makes sense to pay attention to availability zones regardless of scale. And sure, there might be a market somewhere for a more durable VM setup. At a past non-profit job we provided customers the illusion of a single HA VM using Ganeti (http://www.ganeti.org/ http://www.ganeti.org/). But it's not clear to me that the segment is viable -- customers at the low and top end don't need the HA.