4 ms·

Nomad looked like a good product until I used it. I was so looking forward to a simpler take on orchestration, having spent years in k8s-land and generally bein

Nomad looked like a good product until I used it. I was so looking forward to a simpler take on orchestration, having spent years in k8s-land and generally being a fan of Hashicorp in the past. My tldr now is nomad should be considered harmful.

nanmu42 3y ago
Recent version of Nomad has introduced many QoL features like secrets and service discovery, though logging collection remains unsupported.
- mdaniel 3y ago
  > though logging collection remains unsupported. Do you happen to have a link to the issue (I guess or docs, if it's their official stance)? I'd enjoy reading how they ended up in that circumstance
  - kawsper 3y ago
    Logcollection doesn't come out of the box, and you're kinda on your own to set it up, this article (although a bit dated) shows you what you must go through: https://atodorov.me/2021/07/09/logging-on-nomad-and-log-aggregation-with-loki/ https://atodorov.me/2021/07/09/logging-on-nomad-and-log-aggr... It also doesn't help that Nomad handles everything from Docker containers to managing regular applications.
  - jen20 3y ago
    Also worth noting that nomad’s closest competitor (Kubernetes) does not do this either. It’s a huge oversight in both ecosystems.
    - mdaniel 3y ago
      We may be talking about different things, but GP's comment said "unsupported" whereas in kubernetes it's just not part of the control plane offered by kubernetes itself, in exactly the same way that cloud load balancer or even cloud auto-scaling isn't provided by kubernetes itself Having read the sibling's comment, I think I better understand the situation since Nomad isn't a container orchestrator and thus there isn't one homogeneous place from which _to_ collect the logs, but in any sane k8s setup that's not true since both dockerd and containerd have mechanisms to influence that behavior and k8s is perfectly able, and supports, scheduling per-Node utility plumbing stuff like that without drama
      - jen20 3y ago
        Nomad has equivalent mechanisms for collecting logs to Kubernetes. None are built in, in either platform.
        nanmu42 3y ago
        Would you like to provide a link? It would help me a lot. Thanks.
        jen20 3y ago
        https://developer.hashicorp.com/nomad/docs/job-specification/logs https://developer.hashicorp.com/nomad/docs/job-specification... covers much of it from the collection side
      - nanmu42 3y ago
        Yes k8s does not provide a out-of-box logging collection solution but since everything is in containers, we can piggyback on Docker or Containerd via deamonset to get all the logs. Nomad on the other hand support various payload(native exec, exec via chroot, containers, even Firecracker VM by community support), so doing logging collection by end users is trickier. It worth noting that Nomad UI(a official web admin panel) has log tailing utility built-in so maybe partial work has already been done. The developers may have other concerns. The related issue is https://github.com/hashicorp/nomad/issues/10220 https://github.com/hashicorp/nomad/issues/10220
        jen20 3y ago
        All of the work has been done; not partial work.
ochoseis 3y ago
I’ve started playing with Nomad recently just to see what it’s all about. There are some annoying things with networking, but overall it’s been pretty fun so far. What made you consider it harmful?
- erulabs 3y ago
  The number one issue is that deployments are blocking, and scaling is a deployment. That means during a deployment, scaling is disabled. This makes safely deploying to thousands of containers during scale-in hours extremely dangerous. Combined with nomad-autoscaler being a SPOF and crash-happy, the always-be-deploying startup with a daily traffic pattern should not use Nomad. It’s also nontrivial to get nomad to be resilient. For example, if your template uses keys from consul or vault, even with “noop” mode, and consul or vault are unreachable, Nomad will happily start killing running containers that had been rendered correctly in the past, because it can’t re-render the template. This pattern has only recently been addressed by “nomad_retry” but there have been several bugs with it, and 1.6.2 will currently kill all running containers if some template resource can’t be reached. Under the hood this uses consul-template, which does support infinite retry, but getting nomad to use consul template safely is non trivial. Eg: vault tokens expire so infinite retry for vault doesn’t work as-is. Node_pools just landed (2023!), and are still broken when using Levant (another abandoned nomad tool - think kustomize with a much more horrifying DSL). Bonus issues: they’re still trying to get cgroups v2 working, the current version finally doesn’t DOS your backend services in an infinite loop, the UI lies, deployments can get stuck forever because “progress_deadline” is more of a suggestion than a deadline, nomad-autoscaler is not highly available, crashes very easily, often scales faster than its “cooldown” window, and is simply stupid. AWS Karpenter feels a full decade into the future. And all that so I can write my own Nomad specs instead of installing a vendors helm chart in a network isolated namespace. Boo. To reiterate, I’ve liked Hashi for years and years, and I don’t have any ill will towards them. Just shocked at how poor nomad is compared to k8s. It’s definitely more fun than k8s when getting started, for sure.

nanmu42 3y ago

Recent version of Nomad has introduced many QoL features like secrets and service discovery, though logging collection remains unsupported.

mdaniel 3y ago
> though logging collection remains unsupported. Do you happen to have a link to the issue (I guess or docs, if it's their official stance)? I'd enjoy reading how they ended up in that circumstance
- kawsper 3y ago
  Logcollection doesn't come out of the box, and you're kinda on your own to set it up, this article (although a bit dated) shows you what you must go through: https://atodorov.me/2021/07/09/logging-on-nomad-and-log-aggregation-with-loki/ https://atodorov.me/2021/07/09/logging-on-nomad-and-log-aggr... It also doesn't help that Nomad handles everything from Docker containers to managing regular applications.
- jen20 3y ago
  Also worth noting that nomad’s closest competitor (Kubernetes) does not do this either. It’s a huge oversight in both ecosystems.
  - mdaniel 3y ago
    We may be talking about different things, but GP's comment said "unsupported" whereas in kubernetes it's just not part of the control plane offered by kubernetes itself, in exactly the same way that cloud load balancer or even cloud auto-scaling isn't provided by kubernetes itself Having read the sibling's comment, I think I better understand the situation since Nomad isn't a container orchestrator and thus there isn't one homogeneous place from which _to_ collect the logs, but in any sane k8s setup that's not true since both dockerd and containerd have mechanisms to influence that behavior and k8s is perfectly able, and supports, scheduling per-Node utility plumbing stuff like that without drama
    - jen20 3y ago
      Nomad has equivalent mechanisms for collecting logs to Kubernetes. None are built in, in either platform.
      - nanmu42 3y ago
        Would you like to provide a link? It would help me a lot. Thanks.
        jen20 3y ago
        https://developer.hashicorp.com/nomad/docs/job-specification/logs https://developer.hashicorp.com/nomad/docs/job-specification... covers much of it from the collection side
    - nanmu42 3y ago
      Yes k8s does not provide a out-of-box logging collection solution but since everything is in containers, we can piggyback on Docker or Containerd via deamonset to get all the logs. Nomad on the other hand support various payload(native exec, exec via chroot, containers, even Firecracker VM by community support), so doing logging collection by end users is trickier. It worth noting that Nomad UI(a official web admin panel) has log tailing utility built-in so maybe partial work has already been done. The developers may have other concerns. The related issue is https://github.com/hashicorp/nomad/issues/10220 https://github.com/hashicorp/nomad/issues/10220
      - jen20 3y ago
        All of the work has been done; not partial work.

ochoseis 3y ago

I’ve started playing with Nomad recently just to see what it’s all about. There are some annoying things with networking, but overall it’s been pretty fun so far. What made you consider it harmful?

erulabs 3y ago
The number one issue is that deployments are blocking, and scaling is a deployment. That means during a deployment, scaling is disabled. This makes safely deploying to thousands of containers during scale-in hours extremely dangerous. Combined with nomad-autoscaler being a SPOF and crash-happy, the always-be-deploying startup with a daily traffic pattern should not use Nomad. It’s also nontrivial to get nomad to be resilient. For example, if your template uses keys from consul or vault, even with “noop” mode, and consul or vault are unreachable, Nomad will happily start killing running containers that had been rendered correctly in the past, because it can’t re-render the template. This pattern has only recently been addressed by “nomad_retry” but there have been several bugs with it, and 1.6.2 will currently kill all running containers if some template resource can’t be reached. Under the hood this uses consul-template, which does support infinite retry, but getting nomad to use consul template safely is non trivial. Eg: vault tokens expire so infinite retry for vault doesn’t work as-is. Node_pools just landed (2023!), and are still broken when using Levant (another abandoned nomad tool - think kustomize with a much more horrifying DSL). Bonus issues: they’re still trying to get cgroups v2 working, the current version finally doesn’t DOS your backend services in an infinite loop, the UI lies, deployments can get stuck forever because “progress_deadline” is more of a suggestion than a deadline, nomad-autoscaler is not highly available, crashes very easily, often scales faster than its “cooldown” window, and is simply stupid. AWS Karpenter feels a full decade into the future. And all that so I can write my own Nomad specs instead of installing a vendors helm chart in a network isolated namespace. Boo. To reiterate, I’ve liked Hashi for years and years, and I don’t have any ill will towards them. Just shocked at how poor nomad is compared to k8s. It’s definitely more fun than k8s when getting started, for sure.