5 ms·

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running

I have run high availability (HA) systems in prem and your statement vastly understates the difficulty and expense. You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. Or the ISP themselves could suffer an outage. You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages. You absolutely do not want to run out of diesel! You need redundancy of every piece of hardware AND you need to test that failover works as expected because the last thing you need is a core switch to fail and traffic not to route over secondary core switch like expected. You need your multiple air con units and them to be powered off different mains inputs so if the electrics fail on one unit it doesn’t take out the others. I guarantee you that if the air cons will fail, it will be on the hottest day of the year a month amount of portable units will stop your servers from overheating. You need beefy UPS with multiple batteries. Ideally multiple UPSs with each UPS powering a different rail on your racks so that if one UPS fails your hardware is still powered from the other rail. And you need to regularly check the battery status and loads on the UPS. Remember that the back up generator takes a second or two to kick in so you need something to keep the power to the servers and networking hardware to be uninterrupted. And since all your hardware is powered via the UPS, if that dies you still lose power even if the building is powered. And you then need to duplicate all of the above in second location just in case the first location still goes down. By the way, all of the possible failure points I’ve raised above HAVE failed on me when managing HA on prem. The reason people move to the cloud for HA is because rolling your own is like rolling your own encryption: it’s hard, error prone, expensive, and even when you have the right people on the team there’s still a good chance you’ll fuck it up. AWS, for all its faults, does make this side of the job easier.

Nextgrid 5y ago
That's true for on-prem infrastructure, but is all already handled for you if you rent servers from hosting providers such as OVH/Hetzner or even rent colocation space in an existing DC, and is still cheaper than the cloud equivalent (and as we saw recently, actually more reliable as well).
- nostrebored 5y ago
  That’s not what you’ve seen recently. When Hetzner goes down nobody cares, because nobody with important workloads and brain cells is running then there. Colo space assumes that the colo is operating more efficiently than AWS/Azure/GCP when in reality you’re comparing apples and oranges.
- laumars 5y ago
  But then you’re still reliant on those hosting providers not fscking up; just like with cloud providers. Literally the same complaint the GP was making about AWS applies for OVH et al too. In fact I used to run some hobby projects in OVH (as an aside, I really liked their services) so I’m aware that they have their own failures too.
  - Nextgrid 5y ago
    Old-school hosting providers have a lot less moving parts than cloud providers. They have their outages, but they’re usually less frequent.
    - laumars 5y ago
      Are they though? Let’s look at what the recent AWS outages have been: a single region (but AWS makes multi-region easy). The biggest impact to most people is the AWS console, something that one seldom actually needs given AWS is API driven. If the same type of outage happened on OVH then you’d lose KVM to your physical servers. But you seldom need those either. The Azure outage was just AD service but you can roll your own there if you wanted. Plus if you want to talk about SaaS then OVH et al have their own SaaS too. In fact the difference between OVH and AWS is more about scale than it is about reliability (with AWS you can buy hardware and rack it in AWS just like with OVH too). Or maybe by “old skool” you mean the few independent hosts that don’t offer SaaS. However they’re usually pretty small fry and this outages are less likely to be reported. Whereas any AWS service going down is massive news. I’m not a cloud-fanboy by any means (I actually find AWS the least enjoyable to manage from a purely superficial perspective) but I’ve worked across a number of different hosting providers as well as building out HA systems on prem and the anti-cloud sentiment here really misses the pragmatic reality of things.
tzs 5y ago
> You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages. I don't see why the petrol station needs to be a short distance away. Unless the plan is to walk to the petrol station and back (which should not be the plan[1]), anyplace within reasonable driving distance should do. [1] long duration electrical outages will often take out everything a short distance away, and the petrol stations usually have electric pumps.
- laumars 5y ago
  Because there are laws on what you can and cannot fill with fuel. So you may find you have to make smaller but more frequent visits. Also buying fuel for a petrol station is going to be more expensive than having a commercial tanker refill it. So ideally you wouldn’t be making large top ups from the local petrol station except under exceptional outages. As for wider power outages affecting the fuel pumps, I suspect they might have their own generators too. But even if they don’t, outages can still be localised (eg road works accidentally cutting through the mains for that street - I’ve had that happen before too). So there’s still a benefit in having a petrol station near by. To be clear, I’m not suggesting those petrol stations should be 5 minutes walking distance. Just close enough to drive there and back in under half an hour.
  - vinay_ys 5y ago
    A typical multi-MW power-hungry high-tech facility (datacenters, manufacturing, hospitals etc) will have large underground fuel storage tanks big enough to run the full load on generators for couple of days and they are continuously kept refilled via fuel tanker trucks through contracts with bulk fuel distributors. They usually have an SLA of a 40KL tanker in 4 hour notice. In case of advance warning of heavy-rains/floods or other natural disasters that can disrupt road networks, they can have more fuel trucks situated close-by as stand-by. Depending on your contract, you may have priority over other customers in the area. These are fairly standard practices.
    - laumars 5y ago
phil21 5y ago
You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done. If you are going to the level of the above, you go with co-location in purpose built centers at a wholesale level. The "layer1" is all done to the specs you state and you don't have to worry about it. On-prem rarely actually means physically on-prem at any scale beyond a small IT office room. It means co-locating in purpose built datacenters. I'm sure examples exist, but the days of large corporate datacenters are pretty much long over - just inertia keeping the old ones going before they move to somewhere like Equinix or DRT. With the wholesalers you can basically design things to spec, and they build out 10ksqft 2MW critical load room for you a few months later. A few organizations will find it worthwhile to continue to build at this scale (e.g. Visa, the government) but it's exceptionally small.
- laumars 5y ago
  > You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done. Then you’re not running HA and thus the argument about cloud downtime being “worse” than on prem is moot. Obviously if your SLA is basically “we will do our best” then there are all sorts of short cuts one can take. ;)
  - phil21 5y ago
    My (late) point is that when people speak about "on-prem" these days, they are not talking about building corporate datacenters on campus. 30 years ago when you talked on-prem that's what this meant. It's now shifted to on-prem meaning your own hardware in massive shared facilities that handle all that "hard stuff" like redundant power and cooling for you. Bespoke datacenter builds for true-on prem certainly exist, but it's not what that term typically means any longer - at least in my line of business. When I'm selling racks of colo now, my customers are calling that their on-prem facilities. In fact a large part of my previous business was dismantling true "on-prem" facilities to move to such large shared wholesalers.
vinay_ys 5y ago
It is much easier than you think. There are well-defined standards and trained trades people and whole host of companies who make great products and provide after-sales services to do it. Every major financial services, telecom and high-precision manufacturing companies run their infra this way. It is definitely less niche than rolling your own encryption.
- laumars 5y ago
  My analogy wasn’t saying it’s niche. It was comparing the difficulty. And yes, there are trained people (I’m one of them ;) ) but that doesn’t make it easy, cheap, nor even less error prone than using cloud services. Which was my point. Also the reasons those companies usually run their own infra is historically down to legislation more than preference. At least that’s been the case with almost all of the companies I’ve built on prem HA systems for.
- nightpool 5y ago
  financial services, telecom and high-precision manufacturing companies One of these things is not like the other, one of these things is not the same... What use does a CNC shop have for an extensive on-prem multi-DC with failover and high availability? It'd be like buying your own snowplows to make sure that the road is clear so your employees can get to work. Maybe necessary if you live in a place with very bad snowplows and no existing infrastructure, but in most places, just a waste of money.
jrockway 5y ago
> You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. At my last job we provided redundant paths (including entry to your building) as an add-on service. So you might not need two ISPs if you're only worried about fiber cuts. You could still be worried about things like "we think all Juniper routers in the world will die at the exact same instant", in which case you need to make sure you pick an ISP that uses Cisco equipment. And of course, it's possible that your ISP pushes a bad route and breaks the entirety of their link to the rest of the Internet.
H1Supreme 5y ago
> You need a back up generator and to be a short distance away from a petrol station My building has a natural gas backup generator.
- Johnny555 5y ago
  Does it have its own gas well? My sister has a home backup generator, they lost power during some cold snap and some pumping component failed and her neighborhood lost gas too. The only house in the neighborhood that had heat/power had a big propane tank because it was built before the neighborhood got gas. I’ve never seen a data center with natural gas backup power. But I don't know if that's because of reliability or if it's too expensive for a big natural gas hookup that's used rarely. Though I have heard of the opposite -- using natural gas turbines as primary power and utility power as backup.

Nextgrid 5y ago

That's true for on-prem infrastructure, but is all already handled for you if you rent servers from hosting providers such as OVH/Hetzner or even rent colocation space in an existing DC, and is still cheaper than the cloud equivalent (and as we saw recently, actually more reliable as well).

nostrebored 5y ago
That’s not what you’ve seen recently. When Hetzner goes down nobody cares, because nobody with important workloads and brain cells is running then there. Colo space assumes that the colo is operating more efficiently than AWS/Azure/GCP when in reality you’re comparing apples and oranges.
laumars 5y ago
But then you’re still reliant on those hosting providers not fscking up; just like with cloud providers. Literally the same complaint the GP was making about AWS applies for OVH et al too. In fact I used to run some hobby projects in OVH (as an aside, I really liked their services) so I’m aware that they have their own failures too.
- Nextgrid 5y ago
  Old-school hosting providers have a lot less moving parts than cloud providers. They have their outages, but they’re usually less frequent.
  - laumars 5y ago
    Are they though? Let’s look at what the recent AWS outages have been: a single region (but AWS makes multi-region easy). The biggest impact to most people is the AWS console, something that one seldom actually needs given AWS is API driven. If the same type of outage happened on OVH then you’d lose KVM to your physical servers. But you seldom need those either. The Azure outage was just AD service but you can roll your own there if you wanted. Plus if you want to talk about SaaS then OVH et al have their own SaaS too. In fact the difference between OVH and AWS is more about scale than it is about reliability (with AWS you can buy hardware and rack it in AWS just like with OVH too). Or maybe by “old skool” you mean the few independent hosts that don’t offer SaaS. However they’re usually pretty small fry and this outages are less likely to be reported. Whereas any AWS service going down is massive news. I’m not a cloud-fanboy by any means (I actually find AWS the least enjoyable to manage from a purely superficial perspective) but I’ve worked across a number of different hosting providers as well as building out HA systems on prem and the anti-cloud sentiment here really misses the pragmatic reality of things.

tzs 5y ago

> You need a back up generator and to be a short distance away from a petrol station so you can refuel quickly and regularly when suffering from longer durations of power outages. I don't see why the petrol station needs to be a short distance away. Unless the plan is to walk to the petrol station and back (which should not be the plan[1]), anyplace within reasonable driving distance should do. [1] long duration electrical outages will often take out everything a short distance away, and the petrol stations usually have electric pumps.

laumars 5y ago
Because there are laws on what you can and cannot fill with fuel. So you may find you have to make smaller but more frequent visits. Also buying fuel for a petrol station is going to be more expensive than having a commercial tanker refill it. So ideally you wouldn’t be making large top ups from the local petrol station except under exceptional outages. As for wider power outages affecting the fuel pumps, I suspect they might have their own generators too. But even if they don’t, outages can still be localised (eg road works accidentally cutting through the mains for that street - I’ve had that happen before too). So there’s still a benefit in having a petrol station near by. To be clear, I’m not suggesting those petrol stations should be 5 minutes walking distance. Just close enough to drive there and back in under half an hour.
- vinay_ys 5y ago
  A typical multi-MW power-hungry high-tech facility (datacenters, manufacturing, hospitals etc) will have large underground fuel storage tanks big enough to run the full load on generators for couple of days and they are continuously kept refilled via fuel tanker trucks through contracts with bulk fuel distributors. They usually have an SLA of a 40KL tanker in 4 hour notice. In case of advance warning of heavy-rains/floods or other natural disasters that can disrupt road networks, they can have more fuel trucks situated close-by as stand-by. Depending on your contract, you may have priority over other customers in the area. These are fairly standard practices.
  - laumars 5y ago

phil21 5y ago

You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done. If you are going to the level of the above, you go with co-location in purpose built centers at a wholesale level. The "layer1" is all done to the specs you state and you don't have to worry about it. On-prem rarely actually means physically on-prem at any scale beyond a small IT office room. It means co-locating in purpose built datacenters. I'm sure examples exist, but the days of large corporate datacenters are pretty much long over - just inertia keeping the old ones going before they move to somewhere like Equinix or DRT. With the wholesalers you can basically design things to spec, and they build out 10ksqft 2MW critical load room for you a few months later. A few organizations will find it worthwhile to continue to build at this scale (e.g. Visa, the government) but it's exceptionally small.

laumars 5y ago
> You really don't need almost any of this stuff. If you have small on-prem needs just grab a couple fiber links, try for diversity on paths for them (good luck), add some power backup if it fits your needs, and be done. Then you’re not running HA and thus the argument about cloud downtime being “worse” than on prem is moot. Obviously if your SLA is basically “we will do our best” then there are all sorts of short cuts one can take. ;)
- phil21 5y ago
  My (late) point is that when people speak about "on-prem" these days, they are not talking about building corporate datacenters on campus. 30 years ago when you talked on-prem that's what this meant. It's now shifted to on-prem meaning your own hardware in massive shared facilities that handle all that "hard stuff" like redundant power and cooling for you. Bespoke datacenter builds for true-on prem certainly exist, but it's not what that term typically means any longer - at least in my line of business. When I'm selling racks of colo now, my customers are calling that their on-prem facilities. In fact a large part of my previous business was dismantling true "on-prem" facilities to move to such large shared wholesalers.

vinay_ys 5y ago

It is much easier than you think. There are well-defined standards and trained trades people and whole host of companies who make great products and provide after-sales services to do it. Every major financial services, telecom and high-precision manufacturing companies run their infra this way. It is definitely less niche than rolling your own encryption.

laumars 5y ago
My analogy wasn’t saying it’s niche. It was comparing the difficulty. And yes, there are trained people (I’m one of them ;) ) but that doesn’t make it easy, cheap, nor even less error prone than using cloud services. Which was my point. Also the reasons those companies usually run their own infra is historically down to legislation more than preference. At least that’s been the case with almost all of the companies I’ve built on prem HA systems for.
nightpool 5y ago
financial services, telecom and high-precision manufacturing companies One of these things is not like the other, one of these things is not the same... What use does a CNC shop have for an extensive on-prem multi-DC with failover and high availability? It'd be like buying your own snowplows to make sure that the road is clear so your employees can get to work. Maybe necessary if you live in a place with very bad snowplows and no existing infrastructure, but in most places, just a waste of money.

jrockway 5y ago

> You need multiple physical links in running to different ISPs because builders working on properties further down the street could accidentally cut through your fibre. At my last job we provided redundant paths (including entry to your building) as an add-on service. So you might not need two ISPs if you're only worried about fiber cuts. You could still be worried about things like "we think all Juniper routers in the world will die at the exact same instant", in which case you need to make sure you pick an ISP that uses Cisco equipment. And of course, it's possible that your ISP pushes a bad route and breaks the entirety of their link to the rest of the Internet.

H1Supreme 5y ago

> You need a back up generator and to be a short distance away from a petrol station My building has a natural gas backup generator.

Johnny555 5y ago
Does it have its own gas well? My sister has a home backup generator, they lost power during some cold snap and some pumping component failed and her neighborhood lost gas too. The only house in the neighborhood that had heat/power had a big propane tank because it was built before the neighborhood got gas. I’ve never seen a data center with natural gas backup power. But I don't know if that's because of reliability or if it's too expensive for a big natural gas hookup that's used rarely. Though I have heard of the opposite -- using natural gas turbines as primary power and utility power as backup.