5 ms·

Congrats on the beta and everything, but honestly who isn’t using k8s at this point in time? Even running your own on prem is simple as with things like ranche

Congrats on the beta and everything, but honestly who isn’t using k8s at this point in time? Even running your own on prem is simple as with things like rancher these days.

kawsper 6y ago
We're running Nomad, it's doing one thing and one thing well, when we looked at Kubernetes and all the moving parts it comes with plus all the opioniated stacks people tend to run with it, it was a daunting learning experience, we managed to get it running but the upgrade processes felt brittle and it seemed like a full-time job just keeping it updated and happy. Nomad + Consul have been one of the best decisions we've made, it's simple to reason about, easy to maintain, the upgrade path haven't been stressful and it requires a minimum of maintenance.
- MrSaints 6y ago
  I'm quite curious about your scale / size of your team. Most of the moving parts for k8s are easily handled if you go for a managed service. Using Nomad alone is fine, but with Consul in the mix, it requires quite a bit of set-up (especially for some degree of HA), and from my experience, is no less harder than using something like k3s. Overall from my perspective, it just seems like Nomad + Consul sits in two places. Either for an org. small enough where HA is not a concern, so setting it up, and running it "on-premise" is trivial. Or, for an org. large enough where you can have a team dedicated to setting it up, and managing it to ensure it meets various SLAs. Genuinely curious to know what's your experience been like, and if it matches up with this.
  - kawsper 6y ago
    We're 1.5 people to do ops works, but no one is full time on it, I'm the main responsible and I have someone that helps me when he's interested. We are 5 developers. Our whole platform is between 10-50 EC2 machines running a Nomad cluster, Nomad manages our Docker containers and with services backed by RDS. I think managed services were in their infancy when we did our initial research back in 2017/2018, Tectonic+Kubernetes with CoreOS looked promising but they were bought by Red Hat and probably rebranded/merged/disappeared into OpenShift. EKS was in beta an only available in the US (we're in EU). We did try Rancher but we hit issues with it. I don't know if K3S existed yet, but just looking at the diagram on their website it does look quite interesting. We launched Consul first and started defining all of our services, and after that we started moving applications into Nomad. HA has been quite easy with Terraform on EC2. We build "golden-images" with Packer, and then launch them with Terraform, upgrading Consul is adding 3 new servers, making sure things are stable, and then removing the 3 old servers.
  - mrweasel 6y ago
    > Most of the moving parts for k8s are easily handled if you go for a managed service. Which large parts of Europe currently cannot do, or are at least unwilling to risk doing, due to Schrems II. Amazon, Google and Azure are the three providers with the best managed Kubernetes services, or that least those who require least involvement from the user. Otherwise I completely agree, Kubernetes is less of an overhead, if you can rely on managed services. I do question the idea that Nomad can't easily do HA. If you can build an HA environment for Kubernetes, then you can just as easily do the same for Nomad.
bajsejohannes 6y ago
My next project will use Nomad over Kubernetes, and the reason is that I want to keep complexity down. I haven't used rancher, but another layer won't remove complexity, just hide it (until the abstraction leaks).
heipei 6y ago
There are plenty of shops running Nomad, Cloudflare is probably one of the most prominent ones. Also, due to its simplicity and easy of deployment compare to k8s, my guess is that there is a large amount of small shops running Nomad because they can actually manage it, but those are the exact shops that don't have a public presence or write engineering blog posts.
- MattJ100 6y ago
  Definitely. As a solo founder I really want to keep complexity down. I didn't already have significant experience with k8s, but what experience I had steered me away from it for a one-person setup*. I opted for Nomad and haven't regretted it so far. I know k8s is more popular (meaning there is broader community support and a larger ecosystem around it), and has an extremely diverse set of features available. I don't doubt it is a sensible choice for many deployments, but I really don't think that these properties are a reason to dismiss alternatives. Nomad works well, and even supports various use-cases that k8s doesn't. * for this particular project managed options weren't really an option.
jen20 6y ago
> but honestly who isn’t using k8s at this point in time Most of the market outside of a small 'cloud-native' echo chamber. I use Kubernetes currently, and almost every aspect of the everything related to it would be simpler, more scalable, and generally less bad using Nomad.
rickette 6y ago
I've introduced Nomad some ~4 years ago in an organisation that requires everything to run on-prem. This has been pretty successful, with adoption expanding beyond its original use-case. Not dismissing k8s here but if you're running Docker Swarm (dead) or Mesos (also dead) on-prem I'd think it's wise to also take a look at Nomad.
42wim 6y ago
Another person here who isn't using k8s. We're running 1000+ containers on it and it just works, needs almost no maintenance and the Nomad upgrades are with zero downtime.
chucky_z 6y ago
We cycle north of a million containers a day through all our Nomad clusters. Scales, functions, breaks, and is repairable all in predictable ways. The same is true for Vault and Consul. Also if you run Windows, Nomad is your absolute best choice. I understand k8s work on Windows, however I don't know a single individual running Windows containers.
mrweasel 6y ago
We're still struggling to figure out why people want to use Kubernetes. I don't mean that is a "Kubernetes" is bad kinda way and we sell a lot of Kubernetes, both running clusters and just general consulting. It's really really hard to find customers who couldn't just run their containers on one or two VMs. More and more we see the developers getting Kubernetes added to the requirement, and then they show up with as little as ONE container. The argument I often hear is: Then we can scale up during peak hour. Well.. yes, but this is a cluster limited to just you. You still need to pay for the workers nodes and the control plan. Those don't just turn off. Most people seem to not grasp that running Kubernetes is often more expensive than running VMs or EC2 instances. You need to have a large number of smaller containers and a lot of interaction between these containers for Kubernetes to make much sense, in my opinion. It's a solution for a small subset of problem, and if your solution is outside that subset, the management overhead might not be worth it.
- maccard 6y ago
  > It's really really hard to find customers who couldn't just run their containers on one or two VMs If it runs on two VMs, how do you handle networking between them, or updating the containers? Once you actually start to run a container, kubernetes solves a bunch of problems straight away that you have to solve yourself if you use something elelse > You still need to pay for the workers nodes and the control plan. Surely if you're running k8s at any scale you're auto scaling your worker nodes, and if you're spending enough to actually scale your workers the cost of the control plane is negligible compared to the rest (i.e. on aws it's $30/month per cluster)
  - mrweasel 6y ago
    > If it runs on two VMs, how do you handle networking between them, or updating the containers? You do know that networking existed before containers right? In our case though: We already have a large multi-tenant load balancer, so any traffic that needs be able to hit either containers is routed that way. Inter-container communication is just run over the networks as you would any normal application. Deployment, that's a little tricky, and here Kubernetes do present good, but confusing way of managing deployments. However, we find that for most setups Ansible is actually just fine. Remove the VM from the load balancer, upgrade that VM, swap it out for the other VM and upgrade that. This is something we sovled before Kubernetes was available. It's my belief though that many of those developers who want Kubernetes actually want kubectl, because it's an easy way for them to deploy code to production. They in fact don't really care what runs their code in the end. I mostly deal in onprem Kubernetes, so autoscaling workers isn't really a thing. The hardware NEEDS to be present at all time. Your point about the cost of the control plan VMs is correct, but what if your application runs fine on roughly the same resources as the a control plan node? (something like 4GB of RAM vs. 2GB and maybe 2 CPUs) My issue is that I see people pick Kubernetes for insanely small projects.
hugofirth 6y ago
Competition is good, and from my experience with both Nomad feels like a more complete product than kubernetes
hnarn 6y ago
> honestly who isn’t using k8s at this point in time? The HN bubble can be fascinating at times. The answer is: a lot of people, and a lot of companies. Like A LOT.

Congrats on the beta and everything, but honestly who isn’t using k8s at this point in time? Even running your own on prem is simple as with things like rancher these days.

kawsper 6y ago

We're running Nomad, it's doing one thing and one thing well, when we looked at Kubernetes and all the moving parts it comes with plus all the opioniated stacks people tend to run with it, it was a daunting learning experience, we managed to get it running but the upgrade processes felt brittle and it seemed like a full-time job just keeping it updated and happy. Nomad + Consul have been one of the best decisions we've made, it's simple to reason about, easy to maintain, the upgrade path haven't been stressful and it requires a minimum of maintenance.

MrSaints 6y ago
I'm quite curious about your scale / size of your team. Most of the moving parts for k8s are easily handled if you go for a managed service. Using Nomad alone is fine, but with Consul in the mix, it requires quite a bit of set-up (especially for some degree of HA), and from my experience, is no less harder than using something like k3s. Overall from my perspective, it just seems like Nomad + Consul sits in two places. Either for an org. small enough where HA is not a concern, so setting it up, and running it "on-premise" is trivial. Or, for an org. large enough where you can have a team dedicated to setting it up, and managing it to ensure it meets various SLAs. Genuinely curious to know what's your experience been like, and if it matches up with this.
- kawsper 6y ago
  We're 1.5 people to do ops works, but no one is full time on it, I'm the main responsible and I have someone that helps me when he's interested. We are 5 developers. Our whole platform is between 10-50 EC2 machines running a Nomad cluster, Nomad manages our Docker containers and with services backed by RDS. I think managed services were in their infancy when we did our initial research back in 2017/2018, Tectonic+Kubernetes with CoreOS looked promising but they were bought by Red Hat and probably rebranded/merged/disappeared into OpenShift. EKS was in beta an only available in the US (we're in EU). We did try Rancher but we hit issues with it. I don't know if K3S existed yet, but just looking at the diagram on their website it does look quite interesting. We launched Consul first and started defining all of our services, and after that we started moving applications into Nomad. HA has been quite easy with Terraform on EC2. We build "golden-images" with Packer, and then launch them with Terraform, upgrading Consul is adding 3 new servers, making sure things are stable, and then removing the 3 old servers.
- mrweasel 6y ago
  > Most of the moving parts for k8s are easily handled if you go for a managed service. Which large parts of Europe currently cannot do, or are at least unwilling to risk doing, due to Schrems II. Amazon, Google and Azure are the three providers with the best managed Kubernetes services, or that least those who require least involvement from the user. Otherwise I completely agree, Kubernetes is less of an overhead, if you can rely on managed services. I do question the idea that Nomad can't easily do HA. If you can build an HA environment for Kubernetes, then you can just as easily do the same for Nomad.

bajsejohannes 6y ago

My next project will use Nomad over Kubernetes, and the reason is that I want to keep complexity down. I haven't used rancher, but another layer won't remove complexity, just hide it (until the abstraction leaks).

heipei 6y ago

There are plenty of shops running Nomad, Cloudflare is probably one of the most prominent ones. Also, due to its simplicity and easy of deployment compare to k8s, my guess is that there is a large amount of small shops running Nomad because they can actually manage it, but those are the exact shops that don't have a public presence or write engineering blog posts.

MattJ100 6y ago
Definitely. As a solo founder I really want to keep complexity down. I didn't already have significant experience with k8s, but what experience I had steered me away from it for a one-person setup*. I opted for Nomad and haven't regretted it so far. I know k8s is more popular (meaning there is broader community support and a larger ecosystem around it), and has an extremely diverse set of features available. I don't doubt it is a sensible choice for many deployments, but I really don't think that these properties are a reason to dismiss alternatives. Nomad works well, and even supports various use-cases that k8s doesn't. * for this particular project managed options weren't really an option.

jen20 6y ago

> but honestly who isn’t using k8s at this point in time Most of the market outside of a small 'cloud-native' echo chamber. I use Kubernetes currently, and almost every aspect of the everything related to it would be simpler, more scalable, and generally less bad using Nomad.

rickette 6y ago

I've introduced Nomad some ~4 years ago in an organisation that requires everything to run on-prem. This has been pretty successful, with adoption expanding beyond its original use-case. Not dismissing k8s here but if you're running Docker Swarm (dead) or Mesos (also dead) on-prem I'd think it's wise to also take a look at Nomad.

42wim 6y ago

Another person here who isn't using k8s. We're running 1000+ containers on it and it just works, needs almost no maintenance and the Nomad upgrades are with zero downtime.

chucky_z 6y ago

We cycle north of a million containers a day through all our Nomad clusters. Scales, functions, breaks, and is repairable all in predictable ways. The same is true for Vault and Consul. Also if you run Windows, Nomad is your absolute best choice. I understand k8s work on Windows, however I don't know a single individual running Windows containers.

mrweasel 6y ago

We're still struggling to figure out why people want to use Kubernetes. I don't mean that is a "Kubernetes" is bad kinda way and we sell a lot of Kubernetes, both running clusters and just general consulting. It's really really hard to find customers who couldn't just run their containers on one or two VMs. More and more we see the developers getting Kubernetes added to the requirement, and then they show up with as little as ONE container. The argument I often hear is: Then we can scale up during peak hour. Well.. yes, but this is a cluster limited to just you. You still need to pay for the workers nodes and the control plan. Those don't just turn off. Most people seem to not grasp that running Kubernetes is often more expensive than running VMs or EC2 instances. You need to have a large number of smaller containers and a lot of interaction between these containers for Kubernetes to make much sense, in my opinion. It's a solution for a small subset of problem, and if your solution is outside that subset, the management overhead might not be worth it.

maccard 6y ago
> It's really really hard to find customers who couldn't just run their containers on one or two VMs If it runs on two VMs, how do you handle networking between them, or updating the containers? Once you actually start to run a container, kubernetes solves a bunch of problems straight away that you have to solve yourself if you use something elelse > You still need to pay for the workers nodes and the control plan. Surely if you're running k8s at any scale you're auto scaling your worker nodes, and if you're spending enough to actually scale your workers the cost of the control plane is negligible compared to the rest (i.e. on aws it's $30/month per cluster)
- mrweasel 6y ago
  > If it runs on two VMs, how do you handle networking between them, or updating the containers? You do know that networking existed before containers right? In our case though: We already have a large multi-tenant load balancer, so any traffic that needs be able to hit either containers is routed that way. Inter-container communication is just run over the networks as you would any normal application. Deployment, that's a little tricky, and here Kubernetes do present good, but confusing way of managing deployments. However, we find that for most setups Ansible is actually just fine. Remove the VM from the load balancer, upgrade that VM, swap it out for the other VM and upgrade that. This is something we sovled before Kubernetes was available. It's my belief though that many of those developers who want Kubernetes actually want kubectl, because it's an easy way for them to deploy code to production. They in fact don't really care what runs their code in the end. I mostly deal in onprem Kubernetes, so autoscaling workers isn't really a thing. The hardware NEEDS to be present at all time. Your point about the cost of the control plan VMs is correct, but what if your application runs fine on roughly the same resources as the a control plan node? (something like 4GB of RAM vs. 2GB and maybe 2 CPUs) My issue is that I see people pick Kubernetes for insanely small projects.

hugofirth 6y ago

Competition is good, and from my experience with both Nomad feels like a more complete product than kubernetes

hnarn 6y ago

> honestly who isn’t using k8s at this point in time? The HN bubble can be fascinating at times. The answer is: a lot of people, and a lot of companies. Like A LOT.