8 ms·

The dedicated server deployments I worked on at smallish software companies 10+ years ago wound up being really annoying. I enjoy sysadmin type stuff and this i

The dedicated server deployments I worked on at smallish software companies 10+ years ago wound up being really annoying. I enjoy sysadmin type stuff and this idea has tempted me, but I think it’s a false economy in most cases. The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc. I want to worry about whether our product is good, not about middle of the night hardware failures, spinning up a new server and having it not work right because there was no incentive to use configuration management with a single box, having a single runaway task cause OOM or full disk and break everything instead of just that task’s VM, fear of restarting a machine that has been up 1000 days etc.

pas 4y ago
Aaand that's why k8s seems like a good step in the right direction. It has all - okay, absolutely not all, but a good number of the important ones - of these concepts "built-in". Sure, the actual pile of Go(o) that drives it can be improved (and it is indeed improving). That said, the hard truth is that your approach is the correct one, almost all businesses (startups!) overbuild instead of focusing on providing value, identifying the actual niche, etc. (Yes, there's also the demand side of this, as the VC money inflated startups overbuild they want to depend on 3rd parties that can take the load, scale with them, SLAs and whatnot must be advertised.)
- alex_sf 4y ago
  k8s is almost the polar opposite of this idea for most companies. At minimum, a half dozen daemons spread across multiple hosts for.. what? Making the easy things built-in and everything else much more complex than it needs to be?
  - kakoni 4y ago
    And those daemons use constantly 25-30% CPU.https://github.com/k3s-io/k3s/issues/294 https://github.com/k3s-io/k3s/issues/294
  - yrro 4y ago
    I think it depends on the distribution. Microshift for instance seems like a good embodying of the concept that the parent poster was referring to.
- wkat4242 4y ago
  I don't agree, Ansible would be a much saner approach for a few dedicated boxes than k8s
  - riku_iki 4y ago
    What about deploying prebuilt docker image on dedicated server?
    - allarm 4y ago
      What about it? It can be easily done with Ansible. I managed a distributed infra with containers orchestrated by Ansible, worked fine.
      - riku_iki 4y ago
        it would be much less moving/fragile parts in prod. I am not an expert in this topic, and trying to learn about different approaches.
    - wkat4242 4y ago
      This amazing playbook spins up a ton of dockers: https://github.com/spantaleev/matrix-docker-ansible-deploy https://github.com/spantaleev/matrix-docker-ansible-deploy
  - pas 4y ago
    (I haven't used it in a few years, when I tried to deploy OpenStack with it, so if this is outdated, please, correct me!) Ansible is imperative, it can work toward a static state, and that's it. If it runs into a problem it throws up its SSH connection, and cries tons of python errors, and gives up forever. Even with its inventory it's far from declarative. k8s is a bunch of control loops that try to make progress toward their declared state. Failure is not a problem, it'll retry. It'll constantly check its own state, it has a nice API to report it, and thanks to a lot of reified concepts it's harder to have strange clashes between deployed components. (Whereas integrating multiple playbooks with Ansible is ... not trivial.)
- scarface74 4y ago
  And then you have to deal with the complexities of K8s…
  - pas 4y ago
    Yes, ... and? Those are much more "standardized" than whatever else any team cooks up. (And k8s along with Go is steadily improving, so I don't see this as "let's use WordPress because its the platform that has the most answers on StackOverflow".) And even if k8s puts on too many legacy-ness, there are upcoming slimmer manifestations of the core ideas. (Eg. https://github.com/aurae-runtime/aurae https://github.com/aurae-runtime/aurae )
    - scarface74 4y ago
      It doesn’t matter if it adds unneeded complexity as long as it is standardized? And now you’re suggesting that people use a “simpler” non “standard” implementation?
      - pas 4y ago
        You need to solve a bunch of SRE/DevOps problem, fearless deploys, rollbacks, canary rollout, blue/green deployment, HA/LB, backups, dev environments, supporting multiple teams, yadda-yadda. So of course you can implement a minimal complexity solution or you can use something "off the shelf". k8s is complexity and some of it is definitely unneeded for your situation, but it's also rather general, flexible, well supported, etc. > you’re suggesting that people use a “simpler” non “standard” implementation? What I suggested is that if k8s the project/software gets too big to fail, then folks can switch to an alternative. Luckily the k8s concepts and API is open source, so they can be implemented in other projects, and I gave such an example to illustrate that picking k8s is not an Oracle-like vendor lock-in.
- jwr 4y ago
  > k8s seems like a good step in the right direction Kubernetes is fantastic! Whenever I see a potentially competing business start using Kubernetes, I am immediately relieved, as I know I don't have to worry about them anymore. They are about to disappear in a pile of unsolvable technical problems, trying to fix issues that can't be traced down in a tech stack of unbelievable complexity and working around limitations imposed by a system designed for businesses with traffic two-three orders of magnitude larger. Also, their COGS will be way higher than mine, which means they will need to price their product higher, unless they are just burning through VC money (in which case they are a flash in the pan). Good stuff all around.
goneri 4y ago
If you've got your configuration properly automated, nothing prevent you from redeploying your bare-metal nodes every weeks. The opposite can actually also be true, it may become risky at some point to reboot a VM with a high uptime.
- macNchz 4y ago
  Totally, though I think in a fast paced/under-resourced environment incentives matter, and ephemeral cloud vms incentive admins to use configuration management, whereas it's easy to take a shortcut in the moment and quickly manually install something/edit config on a rarely-changing dedicated host. Obviously the same can be said for long running VMs, and this can be solved by having a disciplined team, but I think it's generally more likely in an environment with a single long running dedicated machine.
- Firmwarrior 4y ago
  I'm completely clueless about any server administration beyond a LAMP stack on a single machine serving a few dozen people I always hear about all the great stuff you get practically for free from cloud providers. Is that stuff actually that easy to set up and use? Any time I tried to set up my LAMP stack on a cloud service it was such a confusing and frightening process that I ended up giving up. I'm wondering if I just need to push a little harder and I'll get to Cloud Heaven
  - galaxyLogic 4y ago
    I think the problem is configuration. Every thing that has its own configuration is basically a new language you must be able to speak perfectly. Configuration languages are not statically typed so when you make an error it typically does not tell you you made a mistake, and where, and thy it is a mistake. With Cloud stuff you have more configuration to do because it is about configuring virtual servers etc. Instead of carrying the PC in a box to your room you must "configure it" to make it available.
  - Stranger43 4y ago
    The big problem for server operations is not the damned servers but all of the infrastructure you need to keep them running, like power cooling network/storage and internet connectivity. The cloud is not easy but damn trying to get cooling and power efficiency of an small server room anywhere near the efficiency levels most big data-center publish is next to impossible as is multi vendor internet connectivity. With the cloud all of that kind of goes away as it's managed by whatever data-center operator that cloud is running on but what people forget is that that is also true for old fashioned colocation services which is often offering a better cost/value then cloud. And while it's definitely harder to manage stuff like AWS or Azure because it bleeds a lot of abstractions small scall vpc providers hide from you or that you dont really get with a single home server, it's not hard on the scale of having to run a couple of racks worth of vmware servers with SAN based storage.
wongarsu 4y ago
I think there's a lot of room to bring the best of both worlds together. Some are moving on this already, for example Hetzner Cloud load balancers can also have dedicated servers as targets.
antihero 4y ago
What about simply running k8s on metal? You wouldn’t be able to auto-scale (unless you automated buying and selling the hardware provision from the DS) but it would be nice.
- foepys 4y ago
  You can still auto-scale. Use dedicated servers for your base load and when sudden demand arises, switch to a cloud provider juggling VMs.
- tstrimple 4y ago
  Then you have to manage k8s, and based on the number of companies I've helped move away from that model (to PaaS mostly) it's no fun either.
  - code_biologist 4y ago
    What PaaS providers / platforms do you like? I want to move some personal shell scripted boxes to PaaS, but dropped the idea with the decline of Heroku. Tried Dokku and struggled, but it seems like I should give it another try.
    - geektips 4y ago
      Maybe try coolify[0] its provides a better than dokku in my personal experience. 0. https://coolify.io/ https://coolify.io/
      - password4321 4y ago
        I need ARM, is this still the right choice? Nothing is coming up when I hit the site.
      - 0cf8612b2e1e 4y ago
        Anything of note between the two which you found favorable? I am early-days into designing a Dokku backed system. Dokku has a lot going for it that makes it appealing to me, but happy to learn what I am missing.
deleted 4y ago
[deleted]
thegeomaster 4y ago
> The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc. Hetzner has all of this except managed databases.
- exsysadmin 4y ago
  On their managed servers you can install data bases via KonsoleH and they manage it. (Never used the managed servers myself. MySQL and PostgreSQL?) https://www.hetzner.com/managed-server https://www.hetzner.com/managed-server The webhosting packages also include 1..unlimited DBs (MySQL and PostgreSQL) https://www.hetzner.com/webhosting/ https://www.hetzner.com/webhosting/
  - riku_iki 4y ago
    > install data bases does it have automatic backup and fail over?
    - twic 4y ago
      Looks like it has backups, to an extent [1]: > With booked daily backup or the backup included in the type of server, all data is backed up daily and retained for a maximum of 14 days. Recovery of backups (Restore) is possible via the konsoleH administration interface. But i get the impression that the databases on managed servers are intended for use by apps running on that server, so there isn't really a concept of failover. https://www.hetzner.com/legal/managed-server/ https://www.hetzner.com/legal/managed-server/
- riku_iki 4y ago
  which is most complicated part.
chillfox 4y ago
The key is to use some kind of config management, even for a single box. The problem you need to solve is basically to separate your configurations from the system default and make it easy to bring the system into a known good state.
- regularfry 4y ago
  I do think there's something in there about the immediacy of a replacement if a currently-live box falls over. Rough rule of thumb says it's hours for a physical box, minutes for a VM, seconds for a k8s container, a hopefully smaller number of seconds for a serverless cold start. Each of those has ergonomics different enough that it might matter even with config management that rhymes across the options.
  - miah_ 4y ago
    We should design a protocol that bootstraps newly booted systems via some protocol.. We could call it the Bootstrap Protocol, or bootp for short...
  - chillfox 4y ago
    Sure, but if you are considering a physical box then you have presumably already decided that you are OK with it taking longer to replace.
- p_j_w 4y ago
  This is now eating into the cost savings over a cloud deployment.
  - speedgoose 4y ago
    But most cloud deployments have config management too. Using Ansible/Terraform/Kubernetes on bare metal isn't much different than using Ansible/Terraform/Kubernetes in the cloud.
  - doesnt_know 4y ago
    How so? You should still be doing those things with cloud deployment anyway. The alternative is (the pretty terribly named) "click ops" where devops folks have to fiddle with some cloud UI settings to make production changes or go through wizards to attempt to replicate an environment. If you are doing your environment config as code it ultimately shouldn't matter if your target is a dedicated server, a vm or some cloud specific setup configured via an api.
jabart 4y ago
EC2 or on-premise you need monitoring to detect issues before they become issues. A single drive on a single server failing should never cause a production outage. A lot of configuration issues can be tracked down to self-contained deployments. Does that font file or JRE really need to be installed on the whole server or can you bundle it into your deployment. Our deployments use on-premise and EC2 targets. The deployment script isn't different, only the IP for the host is. Now, I will say if I can use S3 for something I 100% will. There is not an on-premise alternative for it with the same feature set.
- orangepurple 4y ago
  Fine, but RAID-Z2 in a dedicated server is good enough or overkill for most people and it's free.
  - jabart 4y ago
    HackerNews went down because two SSDs in a mirror went down at the same time. This was due to a hours on SSD bug. While Raid-Z2 works, and provides great reliability, two-servers replicated, both with raid-z2 and a nightly offsite backup and as long as you are testing your backups it's a solid plan.
    - bsder 4y ago
      And AWS goes down about once a year because US-East-1 has some weird failure. Your point? The point of DevOps is "Cattle, not pets." Put a bullet in your server once a week to find your failure points.
neeleshs 4y ago
100%. There are definitely cases where managing your own dedicated server infra (not hardware, just the software part) makes sense. For me, the allure of the cloud is not some FAANG scalability fad when its not required, but automatic OS patching, automatic load balancing, NAT services, centralized, analyzable logs, database services that I don't have to manage, out of the box rolling upgrades to services, managing crypto-keys and secrets, and of course, object storage. (These are all the things we use in our startup) I'd go on a limb and say dedicated servers may be very viable/cost effective when we reach a certain scale, instead of the other way round. For the moment, the cloud expenses are worth it for my startup.

pas 4y ago

Aaand that's why k8s seems like a good step in the right direction. It has all - okay, absolutely not all, but a good number of the important ones - of these concepts "built-in". Sure, the actual pile of Go(o) that drives it can be improved (and it is indeed improving). That said, the hard truth is that your approach is the correct one, almost all businesses (startups!) overbuild instead of focusing on providing value, identifying the actual niche, etc. (Yes, there's also the demand side of this, as the VC money inflated startups overbuild they want to depend on 3rd parties that can take the load, scale with them, SLAs and whatnot must be advertised.)

alex_sf 4y ago
k8s is almost the polar opposite of this idea for most companies. At minimum, a half dozen daemons spread across multiple hosts for.. what? Making the easy things built-in and everything else much more complex than it needs to be?
- kakoni 4y ago
  And those daemons use constantly 25-30% CPU.https://github.com/k3s-io/k3s/issues/294 https://github.com/k3s-io/k3s/issues/294
- yrro 4y ago
  I think it depends on the distribution. Microshift for instance seems like a good embodying of the concept that the parent poster was referring to.
wkat4242 4y ago
I don't agree, Ansible would be a much saner approach for a few dedicated boxes than k8s
- riku_iki 4y ago
  What about deploying prebuilt docker image on dedicated server?
  - allarm 4y ago
    What about it? It can be easily done with Ansible. I managed a distributed infra with containers orchestrated by Ansible, worked fine.
    - riku_iki 4y ago
      it would be much less moving/fragile parts in prod. I am not an expert in this topic, and trying to learn about different approaches.
  - wkat4242 4y ago
    This amazing playbook spins up a ton of dockers: https://github.com/spantaleev/matrix-docker-ansible-deploy https://github.com/spantaleev/matrix-docker-ansible-deploy
- pas 4y ago
  (I haven't used it in a few years, when I tried to deploy OpenStack with it, so if this is outdated, please, correct me!) Ansible is imperative, it can work toward a static state, and that's it. If it runs into a problem it throws up its SSH connection, and cries tons of python errors, and gives up forever. Even with its inventory it's far from declarative. k8s is a bunch of control loops that try to make progress toward their declared state. Failure is not a problem, it'll retry. It'll constantly check its own state, it has a nice API to report it, and thanks to a lot of reified concepts it's harder to have strange clashes between deployed components. (Whereas integrating multiple playbooks with Ansible is ... not trivial.)
scarface74 4y ago
And then you have to deal with the complexities of K8s…
- pas 4y ago
  Yes, ... and? Those are much more "standardized" than whatever else any team cooks up. (And k8s along with Go is steadily improving, so I don't see this as "let's use WordPress because its the platform that has the most answers on StackOverflow".) And even if k8s puts on too many legacy-ness, there are upcoming slimmer manifestations of the core ideas. (Eg. https://github.com/aurae-runtime/aurae https://github.com/aurae-runtime/aurae )
  - scarface74 4y ago
    It doesn’t matter if it adds unneeded complexity as long as it is standardized? And now you’re suggesting that people use a “simpler” non “standard” implementation?
    - pas 4y ago
      You need to solve a bunch of SRE/DevOps problem, fearless deploys, rollbacks, canary rollout, blue/green deployment, HA/LB, backups, dev environments, supporting multiple teams, yadda-yadda. So of course you can implement a minimal complexity solution or you can use something "off the shelf". k8s is complexity and some of it is definitely unneeded for your situation, but it's also rather general, flexible, well supported, etc. > you’re suggesting that people use a “simpler” non “standard” implementation? What I suggested is that if k8s the project/software gets too big to fail, then folks can switch to an alternative. Luckily the k8s concepts and API is open source, so they can be implemented in other projects, and I gave such an example to illustrate that picking k8s is not an Oracle-like vendor lock-in.
jwr 4y ago
> k8s seems like a good step in the right direction Kubernetes is fantastic! Whenever I see a potentially competing business start using Kubernetes, I am immediately relieved, as I know I don't have to worry about them anymore. They are about to disappear in a pile of unsolvable technical problems, trying to fix issues that can't be traced down in a tech stack of unbelievable complexity and working around limitations imposed by a system designed for businesses with traffic two-three orders of magnitude larger. Also, their COGS will be way higher than mine, which means they will need to price their product higher, unless they are just burning through VC money (in which case they are a flash in the pan). Good stuff all around.

goneri 4y ago

If you've got your configuration properly automated, nothing prevent you from redeploying your bare-metal nodes every weeks. The opposite can actually also be true, it may become risky at some point to reboot a VM with a high uptime.

macNchz 4y ago
Totally, though I think in a fast paced/under-resourced environment incentives matter, and ephemeral cloud vms incentive admins to use configuration management, whereas it's easy to take a shortcut in the moment and quickly manually install something/edit config on a rarely-changing dedicated host. Obviously the same can be said for long running VMs, and this can be solved by having a disciplined team, but I think it's generally more likely in an environment with a single long running dedicated machine.
Firmwarrior 4y ago
I'm completely clueless about any server administration beyond a LAMP stack on a single machine serving a few dozen people I always hear about all the great stuff you get practically for free from cloud providers. Is that stuff actually that easy to set up and use? Any time I tried to set up my LAMP stack on a cloud service it was such a confusing and frightening process that I ended up giving up. I'm wondering if I just need to push a little harder and I'll get to Cloud Heaven
- galaxyLogic 4y ago
  I think the problem is configuration. Every thing that has its own configuration is basically a new language you must be able to speak perfectly. Configuration languages are not statically typed so when you make an error it typically does not tell you you made a mistake, and where, and thy it is a mistake. With Cloud stuff you have more configuration to do because it is about configuring virtual servers etc. Instead of carrying the PC in a box to your room you must "configure it" to make it available.
- Stranger43 4y ago
  The big problem for server operations is not the damned servers but all of the infrastructure you need to keep them running, like power cooling network/storage and internet connectivity. The cloud is not easy but damn trying to get cooling and power efficiency of an small server room anywhere near the efficiency levels most big data-center publish is next to impossible as is multi vendor internet connectivity. With the cloud all of that kind of goes away as it's managed by whatever data-center operator that cloud is running on but what people forget is that that is also true for old fashioned colocation services which is often offering a better cost/value then cloud. And while it's definitely harder to manage stuff like AWS or Azure because it bleeds a lot of abstractions small scall vpc providers hide from you or that you dont really get with a single home server, it's not hard on the scale of having to run a couple of racks worth of vmware servers with SAN based storage.

wongarsu 4y ago

I think there's a lot of room to bring the best of both worlds together. Some are moving on this already, for example Hetzner Cloud load balancers can also have dedicated servers as targets.

antihero 4y ago

What about simply running k8s on metal? You wouldn’t be able to auto-scale (unless you automated buying and selling the hardware provision from the DS) but it would be nice.

foepys 4y ago
You can still auto-scale. Use dedicated servers for your base load and when sudden demand arises, switch to a cloud provider juggling VMs.
tstrimple 4y ago
Then you have to manage k8s, and based on the number of companies I've helped move away from that model (to PaaS mostly) it's no fun either.
- code_biologist 4y ago
  What PaaS providers / platforms do you like? I want to move some personal shell scripted boxes to PaaS, but dropped the idea with the decline of Heroku. Tried Dokku and struggled, but it seems like I should give it another try.
  - geektips 4y ago
    Maybe try coolify[0] its provides a better than dokku in my personal experience. 0. https://coolify.io/ https://coolify.io/
    - password4321 4y ago
      I need ARM, is this still the right choice? Nothing is coming up when I hit the site.
    - 0cf8612b2e1e 4y ago
      Anything of note between the two which you found favorable? I am early-days into designing a Dokku backed system. Dokku has a lot going for it that makes it appealing to me, but happy to learn what I am missing.

deleted 4y ago

[deleted]

thegeomaster 4y ago

> The incremental cost of being on a cloud is totally worth it to me to have managed databases, automatic snapshots, hosted load balancers, plug and play block storage etc. Hetzner has all of this except managed databases.

exsysadmin 4y ago
On their managed servers you can install data bases via KonsoleH and they manage it. (Never used the managed servers myself. MySQL and PostgreSQL?) https://www.hetzner.com/managed-server https://www.hetzner.com/managed-server The webhosting packages also include 1..unlimited DBs (MySQL and PostgreSQL) https://www.hetzner.com/webhosting/ https://www.hetzner.com/webhosting/
- riku_iki 4y ago
  > install data bases does it have automatic backup and fail over?
  - twic 4y ago
    Looks like it has backups, to an extent [1]: > With booked daily backup or the backup included in the type of server, all data is backed up daily and retained for a maximum of 14 days. Recovery of backups (Restore) is possible via the konsoleH administration interface. But i get the impression that the databases on managed servers are intended for use by apps running on that server, so there isn't really a concept of failover. https://www.hetzner.com/legal/managed-server/ https://www.hetzner.com/legal/managed-server/
riku_iki 4y ago
which is most complicated part.

chillfox 4y ago

The key is to use some kind of config management, even for a single box. The problem you need to solve is basically to separate your configurations from the system default and make it easy to bring the system into a known good state.

regularfry 4y ago
I do think there's something in there about the immediacy of a replacement if a currently-live box falls over. Rough rule of thumb says it's hours for a physical box, minutes for a VM, seconds for a k8s container, a hopefully smaller number of seconds for a serverless cold start. Each of those has ergonomics different enough that it might matter even with config management that rhymes across the options.
- miah_ 4y ago
  We should design a protocol that bootstraps newly booted systems via some protocol.. We could call it the Bootstrap Protocol, or bootp for short...
- chillfox 4y ago
  Sure, but if you are considering a physical box then you have presumably already decided that you are OK with it taking longer to replace.
p_j_w 4y ago
This is now eating into the cost savings over a cloud deployment.
- speedgoose 4y ago
  But most cloud deployments have config management too. Using Ansible/Terraform/Kubernetes on bare metal isn't much different than using Ansible/Terraform/Kubernetes in the cloud.
- doesnt_know 4y ago
  How so? You should still be doing those things with cloud deployment anyway. The alternative is (the pretty terribly named) "click ops" where devops folks have to fiddle with some cloud UI settings to make production changes or go through wizards to attempt to replicate an environment. If you are doing your environment config as code it ultimately shouldn't matter if your target is a dedicated server, a vm or some cloud specific setup configured via an api.

jabart 4y ago

EC2 or on-premise you need monitoring to detect issues before they become issues. A single drive on a single server failing should never cause a production outage. A lot of configuration issues can be tracked down to self-contained deployments. Does that font file or JRE really need to be installed on the whole server or can you bundle it into your deployment. Our deployments use on-premise and EC2 targets. The deployment script isn't different, only the IP for the host is. Now, I will say if I can use S3 for something I 100% will. There is not an on-premise alternative for it with the same feature set.

orangepurple 4y ago
Fine, but RAID-Z2 in a dedicated server is good enough or overkill for most people and it's free.
- jabart 4y ago
  HackerNews went down because two SSDs in a mirror went down at the same time. This was due to a hours on SSD bug. While Raid-Z2 works, and provides great reliability, two-servers replicated, both with raid-z2 and a nightly offsite backup and as long as you are testing your backups it's a solid plan.
  - bsder 4y ago
    And AWS goes down about once a year because US-East-1 has some weird failure. Your point? The point of DevOps is "Cattle, not pets." Put a bullet in your server once a week to find your failure points.

neeleshs 4y ago

100%. There are definitely cases where managing your own dedicated server infra (not hardware, just the software part) makes sense. For me, the allure of the cloud is not some FAANG scalability fad when its not required, but automatic OS patching, automatic load balancing, NAT services, centralized, analyzable logs, database services that I don't have to manage, out of the box rolling upgrades to services, managing crypto-keys and secrets, and of course, object storage. (These are all the things we use in our startup) I'd go on a limb and say dedicated servers may be very viable/cost effective when we reach a certain scale, instead of the other way round. For the moment, the cloud expenses are worth it for my startup.