4 ms·

What is your setup to where you are isolated from "other person making a mistake"? Even if you're a box in a colocated datacenter you're still able to get knock

by rtpg 4y ago

What is your setup to where you are isolated from "other person making a mistake"? Even if you're a box in a colocated datacenter you're still able to get knocked off the net from some maintenance on the surrounding pipes. Hell, hosting your own box doesn't cause Comcast DNS issues to not knock off a bunch of people either. I do think there is some holistic overview of hosting stuff on the internet, where you could label each extra actor that can break things, mitigation strategies, and costs of such. Someone better than me would be able to place relative risk (and I think in that model laying out various provider uptimes/issues would be great!) and offer a smart way of dealing with the buy vs. build question on this.

mwcampbell 4y ago
Fair point. Still, based on my anecdotal experience using leased dedicated servers, mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline.
- rtpg 4y ago
  It does feel like more "hosted" environments are trying to do more fancy stuff inside the network, so have more failure cases. Or perhaps services that do a lot of things, even if you end up just using simple server components. I still have a fun memory of half of IBM Cloud's servers falling over, meaning that our production app was luckily still up but our staging server fell over. I could get to their website, but their login stuff was all messed up. I believe that one was also a "routing stuff got messed up" issue....
- gtirloni 4y ago
  > mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline Not in my experience. Things break all the time, the difference is nobody notices because either the colocating ISP is too small or we are.
toast0 4y ago
If you need fault tolerance/isolation, you want to have a second box in a different colo (preferably in a different city; a different coast/continent if it's important). If you can live with dns round robin between the two, then you can easily host the DNS with multiple providers and avoid SPOF (could maybe host it on the two boxes you already have, too). You're still at risk of domain registry/registrar failures, and failures of their tld nameservers (very rare for well run tlds) and the root servers (not sure if they ever had a widespread failure). And of course, simultaneous failure of both locations isn't impossible, just less likely. On Comcast DNS failures... Most of the recent ones I've heard of manifested as users on Comcast can't resolve X, but were really X had bad DNSSEC records and Comcast DNS refused to return records that weren't signed properly. It's easy to avoid that by not using DNSSEC. In the general case of working despite bad ISP dns, you can't do much (anything?) for web browsers, but if you build apps, you can hard code fallback IPs for when DNS doesn't work... But you need to have IPs that stick around for the lifetime of your app downloads.

by rtpg 4y ago

mwcampbell 4y ago

Fair point. Still, based on my anecdotal experience using leased dedicated servers, mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline.

rtpg 4y ago
It does feel like more "hosted" environments are trying to do more fancy stuff inside the network, so have more failure cases. Or perhaps services that do a lot of things, even if you end up just using simple server components. I still have a fun memory of half of IBM Cloud's servers falling over, meaning that our production app was luckily still up but our staging server fell over. I could get to their website, but their login stuff was all messed up. I believe that one was also a "routing stuff got messed up" issue....
gtirloni 4y ago
> mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline Not in my experience. Things break all the time, the difference is nobody notices because either the colocating ISP is too small or we are.

toast0 4y ago

If you need fault tolerance/isolation, you want to have a second box in a different colo (preferably in a different city; a different coast/continent if it's important). If you can live with dns round robin between the two, then you can easily host the DNS with multiple providers and avoid SPOF (could maybe host it on the two boxes you already have, too). You're still at risk of domain registry/registrar failures, and failures of their tld nameservers (very rare for well run tlds) and the root servers (not sure if they ever had a widespread failure). And of course, simultaneous failure of both locations isn't impossible, just less likely. On Comcast DNS failures... Most of the recent ones I've heard of manifested as users on Comcast can't resolve X, but were really X had bad DNSSEC records and Comcast DNS refused to return records that weren't signed properly. It's easy to avoid that by not using DNSSEC. In the general case of working despite bad ISP dns, you can't do much (anything?) for web browsers, but if you build apps, you can hard code fallback IPs for when DNS doesn't work... But you need to have IPs that stick around for the lifetime of your app downloads.