4 ms·

And any one of those could become an existential risk, could take down your entire company. Do you trust your people to never slip up? To make sure there's rea

And any one of those could become an existential risk, could take down your entire company. Do you trust your people to never slip up? To make sure there's really the reliability & security guarantees that your company depend on? Are there competent technical hands available at all times to handle issues as they come up? This adds significantly to the org chart, requires constant caring to be added, to keep it steered and staffed. I'm a huge fan of self hosting, of buying your own hardware. Especially for very technical entities, that have lots of overlap & intimacy with computers. But your company needs to have reason to believe it can do the job not just well, but to a degree where you're sure you won't lose the entire business. Needs to believe they really are on top of this huge domain! I see very clearly why folks go cloud, and I wish the world was in a better position, had more to say, to show why buying pizza boxes is gonna not just save you money buy be an enduring acceptable safe choice. Much safer to not wade in here, shutting the hell up is free, and there's plenty that can go wrong, but I do think Kubernetes presents the first maybe acceptable opportunity for broadly acceptable DIY computing, delivering key three things: an essential integrativeness of concerns allowing it to be a comprehensive cross-cutting platform for your business's compute, delivers strongly on the security and dependability you absolutely must have in a well known way, and avoids the historical trap of each company ending up married to their own specific boondoggle infrastructure that's been hand cobbled together by whomever had such and such task at the time. There's a stable practice here, one that many companies and practitioners are honing; there's much difference between Kube clusters sure, but these intracacies and elaborations support the creation of something that is mostly alike, if you look from one company's clusters to the next.

luckylion 1y ago
Would you trust cloud providers to never slip up though? I don't think the split is "perfect but pricey" vs "less expensive but shaky". It feels similar to the autonomous driving question, where safety of autonomous driving is often compared to perfection instead of human drivers. Would UniSuper engineers have trashed their entire infrastructure the way Google Cloud did? Granted, that's not a common thing to happen, but they got _very_ lucky that day that they had a backup elsewhere and didn't fully commit to trusting Google.
- paulryanrogers 1y ago
  There is an element of "never fired for buying IBM" involved. If AWS screws up then theoretically they've a bigger bank roll for any settlement or judgement.
- jauntywundrkind 1y ago
  Nothing's perfect? But theres a lot of orders of magnitude difference in investment in what most companies can and will do versus what a hyperscaler will! And while your company is a single petri dish with your own specific problems, these folks have to see a lot of shit. Sometimes yes it all goes bad even there anyways! The resiliency fails! But given how much they've thought about it & how much they've hedged against 1 in a million 1 in a billion cases that they see semi regularly that your company might never notice until it's too late, I think there's a ton of reasons to trust these generally very well exercises systems with pretty sizable engineering teams & deployments behind them. More so than what most folks can bring to bear. These matters are existential threats for your business. And they are existential threats for big providers, for HyperScalers, even more so. Google very publicly owned up to UniSuper, found out why it happened (a blank form on an internal tool lead to the account having a 1 year time limit with auto-deletion), and publicly announced their three fold plan to make sure at least this specific horribly threat would never happen again: > We deprecated the internal tool that triggered this sequence of events… We scrubbed the system database and manually reviewed all GCVE Private Clouds… We corrected the system behavior that sets GCVE Private Clouds for deletion for such deployment workflows. https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident/ https://cloud.google.com/blog/products/infrastructure/detail... The mere fact that this was such a notable event should be itself notable. In general, the hyperscalers have been stupendously reliable. And absurdly secure. They have small armies dedicated insuring both. I look forward to this new era where finally we might have computing of our own good enough & broadly known & trusted enough to make doing our own - even with small teams - viable!! But heavens yes, keep off-site backups!!!
  - luckylion 1y ago
    The scale becomes not quite as impressive if you consider that those thousands or tens of thousands of engineers also maintain hundreds more services in vastly more complex scenarios because they need to cover many more use cases than yours. Most companies don't need to run everything AWS has to offer, so they wouldn't need to handle all of that complexity. Individual AZs have significantly worse uptime than any DC I've ever worked with. They are very reliable, if you are multi-region - and you hire a small army of consultants.

luckylion 1y ago

Would you trust cloud providers to never slip up though? I don't think the split is "perfect but pricey" vs "less expensive but shaky". It feels similar to the autonomous driving question, where safety of autonomous driving is often compared to perfection instead of human drivers. Would UniSuper engineers have trashed their entire infrastructure the way Google Cloud did? Granted, that's not a common thing to happen, but they got _very_ lucky that day that they had a backup elsewhere and didn't fully commit to trusting Google.

paulryanrogers 1y ago
There is an element of "never fired for buying IBM" involved. If AWS screws up then theoretically they've a bigger bank roll for any settlement or judgement.
jauntywundrkind 1y ago
Nothing's perfect? But theres a lot of orders of magnitude difference in investment in what most companies can and will do versus what a hyperscaler will! And while your company is a single petri dish with your own specific problems, these folks have to see a lot of shit. Sometimes yes it all goes bad even there anyways! The resiliency fails! But given how much they've thought about it & how much they've hedged against 1 in a million 1 in a billion cases that they see semi regularly that your company might never notice until it's too late, I think there's a ton of reasons to trust these generally very well exercises systems with pretty sizable engineering teams & deployments behind them. More so than what most folks can bring to bear. These matters are existential threats for your business. And they are existential threats for big providers, for HyperScalers, even more so. Google very publicly owned up to UniSuper, found out why it happened (a blank form on an internal tool lead to the account having a 1 year time limit with auto-deletion), and publicly announced their three fold plan to make sure at least this specific horribly threat would never happen again: > We deprecated the internal tool that triggered this sequence of events… We scrubbed the system database and manually reviewed all GCVE Private Clouds… We corrected the system behavior that sets GCVE Private Clouds for deletion for such deployment workflows. https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident/ https://cloud.google.com/blog/products/infrastructure/detail... The mere fact that this was such a notable event should be itself notable. In general, the hyperscalers have been stupendously reliable. And absurdly secure. They have small armies dedicated insuring both. I look forward to this new era where finally we might have computing of our own good enough & broadly known & trusted enough to make doing our own - even with small teams - viable!! But heavens yes, keep off-site backups!!!
- luckylion 1y ago
  The scale becomes not quite as impressive if you consider that those thousands or tens of thousands of engineers also maintain hundreds more services in vastly more complex scenarios because they need to cover many more use cases than yours. Most companies don't need to run everything AWS has to offer, so they wouldn't need to handle all of that complexity. Individual AZs have significantly worse uptime than any DC I've ever worked with. They are very reliable, if you are multi-region - and you hire a small army of consultants.