4 ms·

The idea for this comparison came when I read in article that 40% of internet downtime is due to hardware issues (not sure if this is true). The experiment aims

The idea for this comparison came when I read in article that 40% of internet downtime is due to hardware issues (not sure if this is true). The experiment aims to compare the uptime between a website using a reliable cloud provider and one managed in-house due to compliance or other reasons. I understand your point about the variety of configurations that exist beyond simple cloud vs. on-premises. However, my focus is specifically on comparing traditional on-premises data center management against using established cloud services. Here's the repo of the project if that helps you believe in what I say https://github.com/mateomaza/cloudXground https://github.com/mateomaza/cloudXground. I don't even have the knowledge to do something wrong with the info tbf, I mean just look at my GitHub.

LinuxBender 2y ago
One more factor to consider in your project is whether or not the cloud or non-cloud providers are doing live kernel patching. Most cloud providers are incentivized to do live kernel patching to avoid interrupting their customers, thus not rebooting the hardware. Full reboots at the hardware layer is where most borderline hardware issues are detected during power-on-self-test. Some on-prem companies are doing live patching now to avoid having to swap hardware as often.
- mateo-maza 2y ago
  Wow this is extremely good info, I'm going to investigate on this for the comparison. Thanks for the tip, your comment was just too helpful!
solardev 2y ago
FWIW, it seems to me that a test like this would more likely compare network architecture & topology (and plain dumb luck) rather than hardware failure rates. Since you're not controlling for other variables, and your sample size is likely to be tiny, it's probably just looking at the results of failover handling (or lack thereof) and regional network issues. A cloud, after all, is just a set of someone else's data centers, abstracted and managed for large-scale resale. In the cloud you can have load balancers and reverse proxies and CDNs and database clones and robust DNS setups all handling failovers across multiple data centers. You can do the same thing across several smaller data centers that you rent or colo in. You can also do the same thing across several bare-metal Raspberry Pis you install at friends' houses around the world. But none of those will really be a good test of hardware failure rates, because they're subject to so many other things that can go wrong: buggy code, a mistake in config, different kernel versions or distro patches, firewall settings, small ISP outages, cut wires... you won't really be able to tell what is hardware-related vs software vs environmental vs user error. It's not even really a good test of overall reliability because it's not an apples-to-apples comparison between host setups. It's entirely possible for a few homebrewed PCs set up by an experienced admin to be faster and more reliable than an unpatched EC2-micro setup in a single data center/availability zone. It's also possible for free cloud host to be faster and more reliable than some enterprise's botched $50k rollout that they didn't architect well. High uptime isn't (just) the result of reliable hardware, but good failover and route management. Hardware will fail no matter what, but good hosts -- whether it's a 100-person team at a big cloud or a single wizened beardadmin -- anticipate that, plan for it, code for it, replicate your data, and regularly test the whole stack (and charge accordingly, or at least use economies of scale to benefit smaller customers with the lessons learned from bigger ones). In a simple network test like yours, the geographic proximity (in network hops) between the source and destination are likely to have a far bigger impact (by reducing network variability, caching and firewall/anti-bot issues) than any actual hardware failures on either side. It's pretty rare for even cheap off-the-shelf hardware to fail out of the blue, and you're unlikely to catch it happening in real time unless the whole website goes down and the sysadmin just doesn't care enough to fix it, or it regularly breaks by design (like Low-Tech Magazine's solar server). On the other hand, a well-configured, redundant host probably has hardware failures all day long, every day, but you won't even notice because your traffic just seamlessly routes around it. What you're doing is like visiting a corner store and a big grocery store every day, looking for the same can of soda. Most days both will have it in stock, but some days the big grocery store might be out. But it doesn't really tell you much about either store's behind-the-scenes logistics. Maybe the big grocer has a vertically integrated international freight network that brings in 10,000 cans every week, but they're often out because of their 50,000 customers. Meanwhile the corner store guy just buys a few cases from Costco once a year, but you happen to be their only customer out of 50 or so who likes that soda, so they never have to restock. It doesn't really tell you anything except that the testing methodology isn't sufficient for measuring that particular failure mode.

LinuxBender 2y ago

One more factor to consider in your project is whether or not the cloud or non-cloud providers are doing live kernel patching. Most cloud providers are incentivized to do live kernel patching to avoid interrupting their customers, thus not rebooting the hardware. Full reboots at the hardware layer is where most borderline hardware issues are detected during power-on-self-test. Some on-prem companies are doing live patching now to avoid having to swap hardware as often.

mateo-maza 2y ago
Wow this is extremely good info, I'm going to investigate on this for the comparison. Thanks for the tip, your comment was just too helpful!

solardev 2y ago

FWIW, it seems to me that a test like this would more likely compare network architecture & topology (and plain dumb luck) rather than hardware failure rates. Since you're not controlling for other variables, and your sample size is likely to be tiny, it's probably just looking at the results of failover handling (or lack thereof) and regional network issues. A cloud, after all, is just a set of someone else's data centers, abstracted and managed for large-scale resale. In the cloud you can have load balancers and reverse proxies and CDNs and database clones and robust DNS setups all handling failovers across multiple data centers. You can do the same thing across several smaller data centers that you rent or colo in. You can also do the same thing across several bare-metal Raspberry Pis you install at friends' houses around the world. But none of those will really be a good test of hardware failure rates, because they're subject to so many other things that can go wrong: buggy code, a mistake in config, different kernel versions or distro patches, firewall settings, small ISP outages, cut wires... you won't really be able to tell what is hardware-related vs software vs environmental vs user error. It's not even really a good test of overall reliability because it's not an apples-to-apples comparison between host setups. It's entirely possible for a few homebrewed PCs set up by an experienced admin to be faster and more reliable than an unpatched EC2-micro setup in a single data center/availability zone. It's also possible for free cloud host to be faster and more reliable than some enterprise's botched $50k rollout that they didn't architect well. High uptime isn't (just) the result of reliable hardware, but good failover and route management. Hardware will fail no matter what, but good hosts -- whether it's a 100-person team at a big cloud or a single wizened beardadmin -- anticipate that, plan for it, code for it, replicate your data, and regularly test the whole stack (and charge accordingly, or at least use economies of scale to benefit smaller customers with the lessons learned from bigger ones). In a simple network test like yours, the geographic proximity (in network hops) between the source and destination are likely to have a far bigger impact (by reducing network variability, caching and firewall/anti-bot issues) than any actual hardware failures on either side. It's pretty rare for even cheap off-the-shelf hardware to fail out of the blue, and you're unlikely to catch it happening in real time unless the whole website goes down and the sysadmin just doesn't care enough to fix it, or it regularly breaks by design (like Low-Tech Magazine's solar server). On the other hand, a well-configured, redundant host probably has hardware failures all day long, every day, but you won't even notice because your traffic just seamlessly routes around it. What you're doing is like visiting a corner store and a big grocery store every day, looking for the same can of soda. Most days both will have it in stock, but some days the big grocery store might be out. But it doesn't really tell you much about either store's behind-the-scenes logistics. Maybe the big grocer has a vertically integrated international freight network that brings in 10,000 cans every week, but they're often out because of their 50,000 customers. Meanwhile the corner store guy just buys a few cases from Costco once a year, but you happen to be their only customer out of 50 or so who likes that soda, so they never have to restock. It doesn't really tell you anything except that the testing methodology isn't sufficient for measuring that particular failure mode.