4 ms·

What if your hardware fails? What if you need to scale up (or scale down again)? Making servers run as reliable as in cloud datacenters is really hard work an

What if your hardware fails? What if you need to scale up (or scale down again)? Making servers run as reliable as in cloud datacenters is really hard work and imposes additional cost. I.e. you need not one, but two or three datacenters with 2-3 times the amount of servers you actually use. Then you need admin staff who knows their work and keeps things running smoothly and reliably. You need not one but at least 3 of that staff because they might get sick or go on vacation. Your cost savings probably comes from cutting corners. Depending on the structure of your applications, downtime requirements and recovery plans, that may even be OK. You don't always need a fleet of tanks to deliver a box of milk bottles.

ex3ndr 10y ago
Hardware fails? It will start on other node automagically. May be we will need to switch psql instance. What kind of additional cost? Do you ever hit something really awful with k8s on self-hosted servers? What can go wrong except losing node?
- static_noise 10y ago
  So, you have redundancy and failover at the application level. You don't need reliable servers then.
  - empthought 10y ago
    Reliability is with respect to some SLA; it could very well be that one would _like_ to timeout more quickly than can be reasonably expected of cloud infrastructure, but well within the capabilities of hardware itself.
ZenoArrow 10y ago
> "Making servers run as reliable as in cloud datacenters is really hard work and imposes additional cost." Not really. It's easy enough to have dedicated server hardware that runs just as smoothly as cloud hosted hardware. There are many options for doing so. The key is to pay for support. For example, if you go with Microsoft or Red Hat or Ubuntu servers, you can pay for support contracts to get help from experts in configuring systems. Such options can work out cheaper than cloud hosting (depending on the IT infrastructure requirements), with the added benefit of having hardware more directly under your control.
- Pyxl101 10y ago
  What happens when you run into trouble with one of the servers? Like it's suddenly stopped responding and you can't SSH? For my systems built on EC2, I might not need to do anything at all for typical hardware failures, if I've set up EC2 auto recovery. It transparently relaunches my instance on another server. If I'm not using auto recovery, then I might just stop and restart the instance, in which case I'm also migrated to another physical server if the first one wasn't working. It has the same identity, by the way: same IP address, same disk contents, and all that, and all the machine perceived was an OS reboot (or maybe kernel panic followed by reboot, depending on the failure). For stateless systems I don't even need to bother with that. I'll just configure it to spin up a replacement instance if the desired number of servers I want to have online is not met because one of them failed. When I have 100+ servers in a fleet, this is really convenient. I don't need to keep track of them individually, I just say: "This is the image I'd like to run, and I want to have 100 of them". Hardware failures that would bring down a normal physical machine don't need to involve me at all. The old server hardware that's broken is now in the hands of EC2 to repair or swap out, and my server's back up and running in minutes, possibly with no action on my part. Is something like that easy to achieve with colocation and support offerings with those vendors? Genuine question - I've never operated in colo or worked with those vendors. I realize that colo facilities can take care of repairing and swapping servers for me, but the benefit I get from the cloud is that I can perform those actions in moments with an API, or they can happen automatically, and I don't need to engage other humans. I can't imagine spending my time coordinating with vendors, getting on the phone, opening support tickets, waiting for people to do things, etc... right now I can do everything at the speed of computing with APIs and automation. It'd be tough to give that up.
  - static_noise 10y ago
    If you start running EC2 clusters, you're more in the "cloud storage" dominion and not really in the "self hosting" area anymore. Not everybody needs that and not everybody who does need it also realizes it before a major failure strikes out one of their machines and there is no working failover and their one admin is not reachable and when they finally reach him he has to be flown in, then they need to wait for replacement parts, which all in all delays recovery by several days.
user5994461 10y ago
> You don't always need a fleet of tanks to deliver a box of milk bottles. A bottle of milk filled with Gold is worth about $800k. By extension the 6-pack is $4.8M If tanks can fit seamlessly in the budget, we shall give tanks a serious thoughts! :D

ex3ndr 10y ago

Hardware fails? It will start on other node automagically. May be we will need to switch psql instance. What kind of additional cost? Do you ever hit something really awful with k8s on self-hosted servers? What can go wrong except losing node?

static_noise 10y ago
So, you have redundancy and failover at the application level. You don't need reliable servers then.
- empthought 10y ago
  Reliability is with respect to some SLA; it could very well be that one would _like_ to timeout more quickly than can be reasonably expected of cloud infrastructure, but well within the capabilities of hardware itself.

ZenoArrow 10y ago

> "Making servers run as reliable as in cloud datacenters is really hard work and imposes additional cost." Not really. It's easy enough to have dedicated server hardware that runs just as smoothly as cloud hosted hardware. There are many options for doing so. The key is to pay for support. For example, if you go with Microsoft or Red Hat or Ubuntu servers, you can pay for support contracts to get help from experts in configuring systems. Such options can work out cheaper than cloud hosting (depending on the IT infrastructure requirements), with the added benefit of having hardware more directly under your control.

Pyxl101 10y ago
What happens when you run into trouble with one of the servers? Like it's suddenly stopped responding and you can't SSH? For my systems built on EC2, I might not need to do anything at all for typical hardware failures, if I've set up EC2 auto recovery. It transparently relaunches my instance on another server. If I'm not using auto recovery, then I might just stop and restart the instance, in which case I'm also migrated to another physical server if the first one wasn't working. It has the same identity, by the way: same IP address, same disk contents, and all that, and all the machine perceived was an OS reboot (or maybe kernel panic followed by reboot, depending on the failure). For stateless systems I don't even need to bother with that. I'll just configure it to spin up a replacement instance if the desired number of servers I want to have online is not met because one of them failed. When I have 100+ servers in a fleet, this is really convenient. I don't need to keep track of them individually, I just say: "This is the image I'd like to run, and I want to have 100 of them". Hardware failures that would bring down a normal physical machine don't need to involve me at all. The old server hardware that's broken is now in the hands of EC2 to repair or swap out, and my server's back up and running in minutes, possibly with no action on my part. Is something like that easy to achieve with colocation and support offerings with those vendors? Genuine question - I've never operated in colo or worked with those vendors. I realize that colo facilities can take care of repairing and swapping servers for me, but the benefit I get from the cloud is that I can perform those actions in moments with an API, or they can happen automatically, and I don't need to engage other humans. I can't imagine spending my time coordinating with vendors, getting on the phone, opening support tickets, waiting for people to do things, etc... right now I can do everything at the speed of computing with APIs and automation. It'd be tough to give that up.
- static_noise 10y ago
  If you start running EC2 clusters, you're more in the "cloud storage" dominion and not really in the "self hosting" area anymore. Not everybody needs that and not everybody who does need it also realizes it before a major failure strikes out one of their machines and there is no working failover and their one admin is not reachable and when they finally reach him he has to be flown in, then they need to wait for replacement parts, which all in all delays recovery by several days.

user5994461 10y ago

> You don't always need a fleet of tanks to deliver a box of milk bottles. A bottle of milk filled with Gold is worth about $800k. By extension the 6-pack is $4.8M If tanks can fit seamlessly in the budget, we shall give tanks a serious thoughts! :D