4 ms·

At times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy

At times like this and the big Fastly outage roughly a year ago, choosing to host on a simple, independent bare-metal box doesn't seem like such a bad strategy (as long as one has backups for disaster recovery, of course). Sure, other things can cause downtime in that kind of infrastructure, but at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong.

shric 4y ago
Are there places to host an independent bare-metal box where the internet provider for that box is more reliable than cloudflare?
- Semaphor 4y ago
  We’ve been on Hetzner for several years now. So far the only outages we had were from us moving servers (yeah, we don’t have high availability or load balancing, just a single beefy dedicated server). So, yes?
  - jonatron 4y ago
    Last company I worked for, we had many Hetzner servers. We had many drive failures and CPU fan failures. It's fine if you can deal with a relatively high chance of hardware failure.
- js4ever 4y ago
  According to my monitoring, yes and by a large difference
- gfosco 4y ago
  Any local data center.
- Santosh83 4y ago
  Perhaps not, but those who want to avoid Cloudflare for technical or idealogical reasons won't realistically expect identical performance from smaller alternatives. Same as using Linux. People use it knowing fully well it may not support the latest & greatest consumer gadgets like Windows, but unless people use alternatives despite minor downsides, we shouldn't be distressed when we eventually reach a point of global near-monopoly.
  - guerrilla 4y ago
    > People use it knowing fully well it may not support the latest & greatest consumer gadgets like Windows, Like what exactly?
- HelloNurse 4y ago
  Your box running your web server is far less complicated than using a CDN and worrying about countless additional points of failure. Network problems are only a minor risk.
  - ranguna 4y ago
    And where will I host my box? I'm my apartment? My Internet goes down at least twice a year and my electricity goes down even more, specially in the winter. So no, this is not more reliable than cloudflare.
    - HelloNurse 4y ago
      In a discussion about using a CDN, it's implicit that it represents an addition to "professional" hosting with servers in a well managed data center that has, at least, redundant high-bandwidth network connections, not to a domestic network connection. Note that your home network could be good enough for a personal web site that nobody pays you to respect a SLA on.
      - ranguna 4y ago
        Soooo... Cloudflare?
        mwcampbell 4y ago
        No, we're talking about a colocation provider, or a leased dedicated server provider. I went with OVHcloud US for my latest deployment. HN is at m5hosting.com.
        ranguna 4y ago
        OVH had some server fires that caused some amount of user downtime. I'm not really sure how that's gonna help. Unless you have fallback with multi cloud deployments.
    - zahllos 4y ago
- jwr 4y ago
  I have been running my business on Hetzner bare-metal servers for the last 7 years. During that time there were several brief network outages, on the order of minutes. I think one network outage was 30 minutes. Other than that, no problems. Given the price and performance difference between bare-metal and everything else, I am puzzled as to why small businesses that do not need scalability do not go with bare metal. And given the speeds of todays hardware, if you are not doing something stupid and you have a B2B SaaS, it's really difficult to need "scalability" beyond several bare-metal servers. To be clear, I do not consider my bare-metal boxes "reliable", I have a multi-server setup managed by ansible, with a distributed database, and I can take a single-node failure without problems. I also have a staging setup that can be converted to production quickly, and a terraform setup that can quickly spin up a Digital Ocean cluster if needed.
buro9 4y ago
I'm on Linode. Linode is down because Cloudflare is down. Can't login to their control panel, etc. You'd need to go fully independent and roll your own, with zero dependencies, to really make this work.
- stephenr 4y ago
  Linode control panel being down doesn't mean that the servers they host are down. For *most* web facing apps/sites, a site hosted on e.g. Linode like this, but not using Cloudflare, would be unaffected by such an outage.
  - Operyl 4y ago
    I guess it depends. If you scale up and down via the API and can’t access the API .. you have a pretty good chance of a down scenario if you had a traffic spike you can’t scale for.
- feross 4y ago
  Does Linode really use Cloudflare? They were bought by Akamai earlier this year.
  - buro9 4y ago
    Feel free to try https://login.linode.com/login https://login.linode.com/login Whilst the incident is happening you'll see the Cloudflare 522 page.
    - archon810 4y ago
      Interestingly enough, I'm already logged in, and the homepage as well as the rest of the Linode dashboard are operational. It seems only the login page is down.
  - fomine3 4y ago
    Google's Firebase uses Fastly even after acquisition. It's possible Linode to continue using Cloudflare.
mercer 4y ago
My domains with DNSimple, and my servers are on Hetzner. There should be no dependency on CloudFlare, and yet they are down too.
- sadhorse 4y ago
  Be enlightened by the truth.
  - mercer 4y ago
    Shouldn't you hit me with a stick first, or break a pot?
- sph 4y ago
  Good to see how decentralised the Internet truly is.
- danachow 4y ago
  Puhleeze. DNSimple uses Cloudflare DNS firewall product - this is not a secret. If you don’t like it use an alternative DNS provider, there are plenty.
rtpg 4y ago
What is your setup to where you are isolated from "other person making a mistake"? Even if you're a box in a colocated datacenter you're still able to get knocked off the net from some maintenance on the surrounding pipes. Hell, hosting your own box doesn't cause Comcast DNS issues to not knock off a bunch of people either. I do think there is some holistic overview of hosting stuff on the internet, where you could label each extra actor that can break things, mitigation strategies, and costs of such. Someone better than me would be able to place relative risk (and I think in that model laying out various provider uptimes/issues would be great!) and offer a smart way of dealing with the buy vs. build question on this.
- mwcampbell 4y ago
  Fair point. Still, based on my anecdotal experience using leased dedicated servers, mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline.
  - rtpg 4y ago
    It does feel like more "hosted" environments are trying to do more fancy stuff inside the network, so have more failure cases. Or perhaps services that do a lot of things, even if you end up just using simple server components. I still have a fun memory of half of IBM Cloud's servers falling over, meaning that our production app was luckily still up but our staging server fell over. I could get to their website, but their login stuff was all messed up. I believe that one was also a "routing stuff got messed up" issue....
  - gtirloni 4y ago
    > mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline Not in my experience. Things break all the time, the difference is nobody notices because either the colocating ISP is too small or we are.
- toast0 4y ago
  If you need fault tolerance/isolation, you want to have a second box in a different colo (preferably in a different city; a different coast/continent if it's important). If you can live with dns round robin between the two, then you can easily host the DNS with multiple providers and avoid SPOF (could maybe host it on the two boxes you already have, too). You're still at risk of domain registry/registrar failures, and failures of their tld nameservers (very rare for well run tlds) and the root servers (not sure if they ever had a widespread failure). And of course, simultaneous failure of both locations isn't impossible, just less likely. On Comcast DNS failures... Most of the recent ones I've heard of manifested as users on Comcast can't resolve X, but were really X had bad DNSSEC records and Comcast DNS refused to return records that weren't signed properly. It's easy to avoid that by not using DNSSEC. In the general case of working despite bad ISP dns, you can't do much (anything?) for web browsers, but if you build apps, you can hard code fallback IPs for when DNS doesn't work... But you need to have IPs that stick around for the lifetime of your app downloads.
Tiddles-the2nd 4y ago
Not a useful comment 20 minutes into an outage. The internet is an interconnected web of dependencies. Unless you are Cloudflare/Akamai/Amazon/Google there is no self-hosted anymore. You can host in your basement if you like but you're still dependent on your ISP.
- Santosh83 4y ago
  > The internet is an interconnected web of dependencies. Ironically this is exactly what increasingly centralisation weakens. The huge cloud providers have eroded "an interconnected web of dependencies" into few huge server farms servicing everyone else.
  - andrewnyr 4y ago
    Unfortunately, bad actors have made centralisation necessary for many sites to survive against even low-traffic attacks
dx034 4y ago
Today's actually the first time my site is down and it's Cloudflare's fault instead of my own. Obviously this outage is huge, but so far I've been really impressed with their reliability.
gonzo41 4y ago
Yeah but if the internet is widely down, the network effect is that people probably aren't using your site because everything else is down and they'll wait for confirmation from sites like facebook and their internet banking and netflix to make sure things are back to normal.
gtirloni 4y ago
> at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong. It's likely to be taken offline by yourself more often than not though.

shric 4y ago

Are there places to host an independent bare-metal box where the internet provider for that box is more reliable than cloudflare?

Semaphor 4y ago
We’ve been on Hetzner for several years now. So far the only outages we had were from us moving servers (yeah, we don’t have high availability or load balancing, just a single beefy dedicated server). So, yes?
- jonatron 4y ago
  Last company I worked for, we had many Hetzner servers. We had many drive failures and CPU fan failures. It's fine if you can deal with a relatively high chance of hardware failure.
js4ever 4y ago
According to my monitoring, yes and by a large difference
gfosco 4y ago
Any local data center.
Santosh83 4y ago
Perhaps not, but those who want to avoid Cloudflare for technical or idealogical reasons won't realistically expect identical performance from smaller alternatives. Same as using Linux. People use it knowing fully well it may not support the latest & greatest consumer gadgets like Windows, but unless people use alternatives despite minor downsides, we shouldn't be distressed when we eventually reach a point of global near-monopoly.
- guerrilla 4y ago
  > People use it knowing fully well it may not support the latest & greatest consumer gadgets like Windows, Like what exactly?
HelloNurse 4y ago
Your box running your web server is far less complicated than using a CDN and worrying about countless additional points of failure. Network problems are only a minor risk.
- ranguna 4y ago
  And where will I host my box? I'm my apartment? My Internet goes down at least twice a year and my electricity goes down even more, specially in the winter. So no, this is not more reliable than cloudflare.
  - HelloNurse 4y ago
    In a discussion about using a CDN, it's implicit that it represents an addition to "professional" hosting with servers in a well managed data center that has, at least, redundant high-bandwidth network connections, not to a domestic network connection. Note that your home network could be good enough for a personal web site that nobody pays you to respect a SLA on.
    - ranguna 4y ago
      Soooo... Cloudflare?
      - mwcampbell 4y ago
        No, we're talking about a colocation provider, or a leased dedicated server provider. I went with OVHcloud US for my latest deployment. HN is at m5hosting.com.
        ranguna 4y ago
        OVH had some server fires that caused some amount of user downtime. I'm not really sure how that's gonna help. Unless you have fallback with multi cloud deployments.
  - zahllos 4y ago
jwr 4y ago
I have been running my business on Hetzner bare-metal servers for the last 7 years. During that time there were several brief network outages, on the order of minutes. I think one network outage was 30 minutes. Other than that, no problems. Given the price and performance difference between bare-metal and everything else, I am puzzled as to why small businesses that do not need scalability do not go with bare metal. And given the speeds of todays hardware, if you are not doing something stupid and you have a B2B SaaS, it's really difficult to need "scalability" beyond several bare-metal servers. To be clear, I do not consider my bare-metal boxes "reliable", I have a multi-server setup managed by ansible, with a distributed database, and I can take a single-node failure without problems. I also have a staging setup that can be converted to production quickly, and a terraform setup that can quickly spin up a Digital Ocean cluster if needed.

buro9 4y ago

I'm on Linode. Linode is down because Cloudflare is down. Can't login to their control panel, etc. You'd need to go fully independent and roll your own, with zero dependencies, to really make this work.

stephenr 4y ago
Linode control panel being down doesn't mean that the servers they host are down. For *most* web facing apps/sites, a site hosted on e.g. Linode like this, but not using Cloudflare, would be unaffected by such an outage.
- Operyl 4y ago
  I guess it depends. If you scale up and down via the API and can’t access the API .. you have a pretty good chance of a down scenario if you had a traffic spike you can’t scale for.
feross 4y ago
Does Linode really use Cloudflare? They were bought by Akamai earlier this year.
- buro9 4y ago
  Feel free to try https://login.linode.com/login https://login.linode.com/login Whilst the incident is happening you'll see the Cloudflare 522 page.
  - archon810 4y ago
    Interestingly enough, I'm already logged in, and the homepage as well as the rest of the Linode dashboard are operational. It seems only the login page is down.
- fomine3 4y ago
  Google's Firebase uses Fastly even after acquisition. It's possible Linode to continue using Cloudflare.

mercer 4y ago

My domains with DNSimple, and my servers are on Hetzner. There should be no dependency on CloudFlare, and yet they are down too.

sadhorse 4y ago
Be enlightened by the truth.
- mercer 4y ago
  Shouldn't you hit me with a stick first, or break a pot?
sph 4y ago
Good to see how decentralised the Internet truly is.
danachow 4y ago
Puhleeze. DNSimple uses Cloudflare DNS firewall product - this is not a secret. If you don’t like it use an alternative DNS provider, there are plenty.

rtpg 4y ago

What is your setup to where you are isolated from "other person making a mistake"? Even if you're a box in a colocated datacenter you're still able to get knocked off the net from some maintenance on the surrounding pipes. Hell, hosting your own box doesn't cause Comcast DNS issues to not knock off a bunch of people either. I do think there is some holistic overview of hosting stuff on the internet, where you could label each extra actor that can break things, mitigation strategies, and costs of such. Someone better than me would be able to place relative risk (and I think in that model laying out various provider uptimes/issues would be great!) and offer a smart way of dealing with the buy vs. build question on this.

mwcampbell 4y ago
Fair point. Still, based on my anecdotal experience using leased dedicated servers, mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline.
- rtpg 4y ago
  It does feel like more "hosted" environments are trying to do more fancy stuff inside the network, so have more failure cases. Or perhaps services that do a lot of things, even if you end up just using simple server components. I still have a fun memory of half of IBM Cloud's servers falling over, meaning that our production app was luckily still up but our staging server fell over. I could get to their website, but their login stuff was all messed up. I believe that one was also a "routing stuff got messed up" issue....
- gtirloni 4y ago
  > mistakes at that networking layer seem to happen less often than mistakes that take AWS us-east-1 or one of the big CDNs offline Not in my experience. Things break all the time, the difference is nobody notices because either the colocating ISP is too small or we are.
toast0 4y ago
If you need fault tolerance/isolation, you want to have a second box in a different colo (preferably in a different city; a different coast/continent if it's important). If you can live with dns round robin between the two, then you can easily host the DNS with multiple providers and avoid SPOF (could maybe host it on the two boxes you already have, too). You're still at risk of domain registry/registrar failures, and failures of their tld nameservers (very rare for well run tlds) and the root servers (not sure if they ever had a widespread failure). And of course, simultaneous failure of both locations isn't impossible, just less likely. On Comcast DNS failures... Most of the recent ones I've heard of manifested as users on Comcast can't resolve X, but were really X had bad DNSSEC records and Comcast DNS refused to return records that weren't signed properly. It's easy to avoid that by not using DNSSEC. In the general case of working despite bad ISP dns, you can't do much (anything?) for web browsers, but if you build apps, you can hard code fallback IPs for when DNS doesn't work... But you need to have IPs that stick around for the lifetime of your app downloads.

Tiddles-the2nd 4y ago

Not a useful comment 20 minutes into an outage. The internet is an interconnected web of dependencies. Unless you are Cloudflare/Akamai/Amazon/Google there is no self-hosted anymore. You can host in your basement if you like but you're still dependent on your ISP.

Santosh83 4y ago
> The internet is an interconnected web of dependencies. Ironically this is exactly what increasingly centralisation weakens. The huge cloud providers have eroded "an interconnected web of dependencies" into few huge server farms servicing everyone else.
- andrewnyr 4y ago
  Unfortunately, bad actors have made centralisation necessary for many sites to survive against even low-traffic attacks

dx034 4y ago

Today's actually the first time my site is down and it's Cloudflare's fault instead of my own. Obviously this outage is huge, but so far I've been really impressed with their reliability.

gonzo41 4y ago

Yeah but if the internet is widely down, the network effect is that people probably aren't using your site because everything else is down and they'll wait for confirmation from sites like facebook and their internet banking and netflix to make sure things are back to normal.

gtirloni 4y ago

> at least my service isn't likely to be taken offline by someone else's configuration error or deployment gone wrong. It's likely to be taken offline by yourself more often than not though.