5 ms·

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kuberne

I did/do run both myself, Kubernetes and Nomad, and it was a million times easier to set up Nomad (including Consul) on bare metal than it was to set up Kubernetes. Kubernetes offers more features, but you most likely don't need them and the increase in complexity makes it a pain to maintain. I'm running a three-node cluster on Hetzner [0] for Pirsch [1] right now and haven't had any difficulties whatsoever when upgrading and modifing the cluster. Adding Vault is a bit tricky, but I'm sure I'll get that done too. I highly recommend trying it if you think about maintaining your own infrastructure! Let me know if you have any questions. I plan to write an article/kind-of-tutorial about the setup. [0] https://www.hetzner.com/de/cloud https://www.hetzner.com/de/cloud [1] https://pirsch.io/ https://pirsch.io/

mad_vill 6y ago
bear metal sounds so sick. I'm in for an industrywide rename :D.
- marvinblum 6y ago
  Woops :D
- qchris 6y ago
  There's probably a startup name in there somewhere. Bear Metal Semiconductor. Bear Metal Fabrication. Bear Metal Labs. You could get so creative with the branding and logo.
- fomine3 6y ago
  It's sold as Thermal Grizzly Conductonaut
throwaway894345 6y ago
I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested in how Nomad compares. EDIT: Downvoters, I'm really curious what you're objecting to above, specifically.
- BurritoAlPastor 6y ago
  You have to bring all of those same things to a Nomad deployment as well. It’s generally more lightweight than Kubernetes, so it might be easier to wire those other components in, but you do still need to do that work either way.
  - 64mb 6y ago
    > it might be easier to wire those other components in IMO the few lines of yaml to set the path/host for an Ingress definition seems cleaner to me than using consul-template to spit out some LB config (as in the post's example). For simplicity, a few years ago I preferred Traefik + Swarm. Add a label or two and you're done. But Swarm died :/
- ForHackernews 6y ago
  I'd be really interested, too. Have you looked at k3s at all? We're considering trying to run https://github.com/rancher/k3os https://github.com/rancher/k3os on bare metal servers.
  - throwaway894345 6y ago
    Yeah, that's what I used. It comes with some providers out of the box, but they strike me as toys. For example, it gives you support for node-local volumes, but I don't really want to have to rely on my pods being scheduled on specific nodes (the nodes with the data). Even if you're okay with this, you still have to solve for data redundancy and/or backup yourself. The Rancher folks have a solution for this in the form of LongHorn, so maybe we can expect that to be integrated into k3s in the future. There's also no external DNS support at all, and IIRC the default load-balancer provider (Klipper LB, which itself seems to be not very well documented) assigns node IPs to services (at random, as far as I can tell) so it's difficult to bind a DNS name to a service without something dynamically updating the records whenever k8s changes the service's external IP address (and even then, this is not a recipe for "high availability" since the DNS caches will be out of date for some period of time). Basically k8s is still immature for bare metal; distributions will catch up in time, but for now a lot of the hype outpaces reality.
mahesh_rm 6y ago
>> I plan to write an article/kind-of-tutorial about the setup. Please post it here when you do. :)
GordonS 6y ago
Are you using Nomad to schedule containers on those nodes? I'd also be really interested in a blog post or write-up about your setup!
- marvinblum 6y ago
  Sure, that's what it is for :) We also run cron jobs, Postgres, and Traefik.
jokethrowaway 6y ago
I had the opposite experience (in 2 different companies). Setting up K8s was quite straightforward and docs were helpful. We ended up building a deployment UI for it though. Consul is nice and easy to use. Nomad has been a painful experience: the default UI is confusing (people accidentally killed live containers), we have some small bits and pieces that don't quite behave as we expect and have no idea how to fix them. Error rate is too low to care and there are more pressing issues so likely WONTFIX. We often found ourselves looking into github issues for edge cases or over-allocating resources to overcome scheduling problems. We considered just switching to their paid offering, just not to have to worry about this. It kind of feels like that's their business model: attract engineers with OSS software and then upsell the paid version without all the warts.
- tutfbhuf 6y ago
  Yup. Setting up k8s with kubeadm on bare-metal is very straight forward and can be done within a few minutes on any linux host that is supported by kubeadm and docker + ssh access.
rugwirobaker 6y ago
Nice product by the way. I might try it later on of my upcoming projects

mad_vill 6y ago

bear metal sounds so sick. I'm in for an industrywide rename :D.

marvinblum 6y ago
Woops :D
qchris 6y ago
There's probably a startup name in there somewhere. Bear Metal Semiconductor. Bear Metal Fabrication. Bear Metal Labs. You could get so creative with the branding and logo.
fomine3 6y ago
It's sold as Thermal Grizzly Conductonaut

throwaway894345 6y ago

I would be very interested in a more detailed write-up on Nomad vs Kubernetes for bare metal. I'm working through getting Kubernetes stood up, but I'm running into a dearth of features--namely you have to bring your own load balancer provider, storage provider, ingress controller, external DNS, monitoring, secret encryption, etc, etc before you can run any real world applications on top of it. I would be interested in how Nomad compares. EDIT: Downvoters, I'm really curious what you're objecting to above, specifically.

BurritoAlPastor 6y ago
You have to bring all of those same things to a Nomad deployment as well. It’s generally more lightweight than Kubernetes, so it might be easier to wire those other components in, but you do still need to do that work either way.
- 64mb 6y ago
  > it might be easier to wire those other components in IMO the few lines of yaml to set the path/host for an Ingress definition seems cleaner to me than using consul-template to spit out some LB config (as in the post's example). For simplicity, a few years ago I preferred Traefik + Swarm. Add a label or two and you're done. But Swarm died :/
ForHackernews 6y ago
I'd be really interested, too. Have you looked at k3s at all? We're considering trying to run https://github.com/rancher/k3os https://github.com/rancher/k3os on bare metal servers.
- throwaway894345 6y ago
  Yeah, that's what I used. It comes with some providers out of the box, but they strike me as toys. For example, it gives you support for node-local volumes, but I don't really want to have to rely on my pods being scheduled on specific nodes (the nodes with the data). Even if you're okay with this, you still have to solve for data redundancy and/or backup yourself. The Rancher folks have a solution for this in the form of LongHorn, so maybe we can expect that to be integrated into k3s in the future. There's also no external DNS support at all, and IIRC the default load-balancer provider (Klipper LB, which itself seems to be not very well documented) assigns node IPs to services (at random, as far as I can tell) so it's difficult to bind a DNS name to a service without something dynamically updating the records whenever k8s changes the service's external IP address (and even then, this is not a recipe for "high availability" since the DNS caches will be out of date for some period of time). Basically k8s is still immature for bare metal; distributions will catch up in time, but for now a lot of the hype outpaces reality.

mahesh_rm 6y ago

>> I plan to write an article/kind-of-tutorial about the setup. Please post it here when you do. :)

GordonS 6y ago

Are you using Nomad to schedule containers on those nodes? I'd also be really interested in a blog post or write-up about your setup!

marvinblum 6y ago
Sure, that's what it is for :) We also run cron jobs, Postgres, and Traefik.

jokethrowaway 6y ago

I had the opposite experience (in 2 different companies). Setting up K8s was quite straightforward and docs were helpful. We ended up building a deployment UI for it though. Consul is nice and easy to use. Nomad has been a painful experience: the default UI is confusing (people accidentally killed live containers), we have some small bits and pieces that don't quite behave as we expect and have no idea how to fix them. Error rate is too low to care and there are more pressing issues so likely WONTFIX. We often found ourselves looking into github issues for edge cases or over-allocating resources to overcome scheduling problems. We considered just switching to their paid offering, just not to have to worry about this. It kind of feels like that's their business model: attract engineers with OSS software and then upsell the paid version without all the warts.

tutfbhuf 6y ago
Yup. Setting up k8s with kubeadm on bare-metal is very straight forward and can be done within a few minutes on any linux host that is supported by kubeadm and docker + ssh access.

rugwirobaker 6y ago

Nice product by the way. I might try it later on of my upcoming projects