3 ms·

My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the f

My last startup migrated from Verizon Terremark after the healthcare.gov fiasco several years ago. We also suffered from that massive outage and that was the final straw in migrating to AWS. At AWS, we built a few layers of redundant infrastructure with mulit-AZ availability within a region and then global availability across multiple regions. All this was done at roughly half the cost of the traditional hosting, even when including the additional person-hours required to maintain it on our end. Keeping our infra simple helped that work, and it's literally been years since an outage caused by any AWS issues, even though there have been several large AWS events.

zymhan 5y ago
Indeed, if you only deploy resources in us-east1, or any other single region, you're risking the occasional downtime. I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.
- hinkley 5y ago
  Honestly, I have an app in production that isn't completely hardened against single zone outages. There was pressure to turn off some redundancy in our caching infra, and not every backend service we call is free of tenant affinity, so we could well lose at least 1/3rd of our customers in a single AZ failure in the wrong region, or have huge latency issues for all of our tenants based on high cache miss rates. Having written this, I'm going to ping our SME on the cache replication and remind him that since the last time he benchmarked it, we've upgraded to a newer generation of EC2 instances that has lower latency, and could he please run those numbers again.
- fatnoah 5y ago
  As I discovered many years ago when our infra was only in US-EAST1, failures were also easier to explain since many, many other companies would be offline as well. It made it more of an "Internet problem" than our own company's problem. For whatever reason, customers were far more likely to accept those kinds of outages.
hinkley 5y ago
Every time one of these conversations happen I end up thinking to myself that Oxide Computing needs three more competitors and a big pile of money. AWS maintains a fiction of turnkey infrastructure, and the reality of building your own is so starkly different that I haven't seen an IT group for some time that could successfully push back on these sorts of discussions. Building your own datacenter is still too much like maintaining a muscle car, fiddly bits and grease under your fingernails all the time, meanwhile the world has moved on, and we now have several options in soccer mom EVs that can challenge a classic Corvette in the quarter mile, and obliterate its 0-60-0 time. There is no Hyundai for the operations people, and there should be. I don't know the physics of shipping such a thing, but I think we really do need to be able to buy a populated and pre-wired rack and slot it into the data center. Literally slot it in. If you've ever been curious about maritime shipping, you know that they have a system for securing containers to cranes, trailers, each other, and I don't see a reason you couldn't steal that same design for mounting a server rack to the floor. Other than the pins would need to be removable (eg, a bolt that screws into a threaded hole in the floor) so you don't trip on them. In a word, we need to make physical servers fungible. There are any number of things that we need to do to get there, but I think we can. Honestly I'm surprised we haven't heard more of this sort of talk from Dell, especially after they bought VMWare. This just seems like a huge failure of imagination. Or maybe it's simply a revolution lacking a poster child. At this rate that 'child' has already been born, and we are just waiting to see who it is.
- jeremyjh 5y ago
  I don't think putting the hardware into the rack is really the sticking point; what people like about the cloud is that it abstracts all kinds of details away for them and provides a cohesive system to manage it. AWS, Azure and Google are actually selling something like what you are talking about now [1], where for whatever legal/legacy/performance reason you need it on-prem but still want to pay AWS 5x the cost just to give you the same management interface, and they have some kind of pod they slap into your data-center. What does it tell you that there is a market for this, where essentially what you are buying from them is a management and control plane, when other companies like BMC have been selling that as a standalone product for decades (and for the most part failing to live up to their customer's actual expectations)? [1] https://www.bizety.com/2020/06/28/aws-outposts-google-anthos-gke-and-azure-on-prem-overview/ https://www.bizety.com/2020/06/28/aws-outposts-google-anthos... edit: I actually think a big pull of the cloud is also about shutting down archaic internal IT organizations that have been slowing people down so that it takes weeks and weeks to launch a simple new webservice. Better to give your programmers a cloud account and let them get shit done.

zymhan 5y ago

Indeed, if you only deploy resources in us-east1, or any other single region, you're risking the occasional downtime. I'd wager that will still give you more uptime than a physically-hosted solution for the same cost.

hinkley 5y ago
Honestly, I have an app in production that isn't completely hardened against single zone outages. There was pressure to turn off some redundancy in our caching infra, and not every backend service we call is free of tenant affinity, so we could well lose at least 1/3rd of our customers in a single AZ failure in the wrong region, or have huge latency issues for all of our tenants based on high cache miss rates. Having written this, I'm going to ping our SME on the cache replication and remind him that since the last time he benchmarked it, we've upgraded to a newer generation of EC2 instances that has lower latency, and could he please run those numbers again.
fatnoah 5y ago
As I discovered many years ago when our infra was only in US-EAST1, failures were also easier to explain since many, many other companies would be offline as well. It made it more of an "Internet problem" than our own company's problem. For whatever reason, customers were far more likely to accept those kinds of outages.

hinkley 5y ago

Every time one of these conversations happen I end up thinking to myself that Oxide Computing needs three more competitors and a big pile of money. AWS maintains a fiction of turnkey infrastructure, and the reality of building your own is so starkly different that I haven't seen an IT group for some time that could successfully push back on these sorts of discussions. Building your own datacenter is still too much like maintaining a muscle car, fiddly bits and grease under your fingernails all the time, meanwhile the world has moved on, and we now have several options in soccer mom EVs that can challenge a classic Corvette in the quarter mile, and obliterate its 0-60-0 time. There is no Hyundai for the operations people, and there should be. I don't know the physics of shipping such a thing, but I think we really do need to be able to buy a populated and pre-wired rack and slot it into the data center. Literally slot it in. If you've ever been curious about maritime shipping, you know that they have a system for securing containers to cranes, trailers, each other, and I don't see a reason you couldn't steal that same design for mounting a server rack to the floor. Other than the pins would need to be removable (eg, a bolt that screws into a threaded hole in the floor) so you don't trip on them. In a word, we need to make physical servers fungible. There are any number of things that we need to do to get there, but I think we can. Honestly I'm surprised we haven't heard more of this sort of talk from Dell, especially after they bought VMWare. This just seems like a huge failure of imagination. Or maybe it's simply a revolution lacking a poster child. At this rate that 'child' has already been born, and we are just waiting to see who it is.

jeremyjh 5y ago
I don't think putting the hardware into the rack is really the sticking point; what people like about the cloud is that it abstracts all kinds of details away for them and provides a cohesive system to manage it. AWS, Azure and Google are actually selling something like what you are talking about now [1], where for whatever legal/legacy/performance reason you need it on-prem but still want to pay AWS 5x the cost just to give you the same management interface, and they have some kind of pod they slap into your data-center. What does it tell you that there is a market for this, where essentially what you are buying from them is a management and control plane, when other companies like BMC have been selling that as a standalone product for decades (and for the most part failing to live up to their customer's actual expectations)? [1] https://www.bizety.com/2020/06/28/aws-outposts-google-anthos-gke-and-azure-on-prem-overview/ https://www.bizety.com/2020/06/28/aws-outposts-google-anthos... edit: I actually think a big pull of the cloud is also about shutting down archaic internal IT organizations that have been slowing people down so that it takes weeks and weeks to launch a simple new webservice. Better to give your programmers a cloud account and let them get shit done.