6 ms·

I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment in

I worked at a hedge fund that also used nomad. The problem, however, is not how well it scales or whatever, but the fact that all the accompanying deployment info and literature is for kubernetes, and k8s has far more features. I like the quality of products from HashiCorp, but k8s is far, far, ahead of where nomad is. What I really want is better integration for Terraform and kubernetes. The current TF k8s support leaves much to be desired, too many things are missing or broken and I find there are several bugs that result in deployment flapping (i.e., constantly re-deploying the same thing when there are no changes).

api 6y ago
Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.
- 4636760295 6y ago
  The main missing feature is that it's not kubernetes.
  - aganame 6y ago
    You have an extra word there.
- SteveNuts 6y ago
  For us it's not a feature missing, but the mindshare is low, and there's not really any prebuilt examples of how to run and maintain services long term reliably.
- DavyJone 6y ago
  Does it have "easy" side-cars/init/post? Last time I checked those were missing for example.
  - adadgar 6y ago
    It does! Here is the relevant documentation and examples: https://www.nomadproject.io/docs/job-specification/lifecycle/ https://www.nomadproject.io/docs/job-specification/lifecycle... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob
    - DavyJone 6y ago
      Ill check that out, I really think K8s is becoming a bit too complex for many customers. Thanks!
- q3k 6y ago
  The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.
  - chucky_z 6y ago
    Preemption is coming in the next release in OSS :)
    - deleted 6y ago
      [deleted]
  - BurritoAlPastor 6y ago
    We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.
    - q3k 6y ago
      For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service. Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.
  - schmichael 6y ago
    Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion. (I'm the Nomad Team Lead at HashiCorp)
- syllogism 6y ago
  A few long-standing feature requests I've noticed a lot: * No autoscaling * Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289 https://github.com/hashicorp/nomad/issues/289 * No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419 https://github.com/hashicorp/nomad/issues/419
  - jambay 6y ago
    Autoscaling is in Tech Preview as of March: https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-tech-preview/ https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t... Disclaimer: I work at HashiCorp in Product Management.
    - syllogism 6y ago
      Nice! It's definitely been one of the bigger holes. I hope you'll also consider sequential tasks in a single jobfile. Without it it's kind of awkward because you have to be passing shell scripts around just to run a setup step before your actual workload.
      - nickethier 6y ago
        We’ve also started to implement this with a new task dependencies feature in the latest release. Check out: https://www.hashicorp.com/resources/preview-of-nomad-0-11-task-dependencies/ https://www.hashicorp.com/resources/preview-of-nomad-0-11-ta... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob There’s more work going on there to further improve the feature in upcoming releases. Disclaimer: I’m an engineer on the Nomad team.
monus 6y ago
Crossplane https://crossplane.io https://crossplane.io might be what you’re looking for with its bunch of controllers and a nice composition API. One of the best features is that you can bundle a CR to request a MySQL database and it will be satisfied with whatever config is in your cluster so that app only declares the need but not care how it’s done. Disclaimer: I’m one of the maintainers of Crossplane.
- mleonard 6y ago
  Any chance you plan to integrate with Google's Config Connector? It's very similar to crossplane (but gcp specific). https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview
  - monus 6y ago
    I think that'd be a bit challenging because Config Connector is highly opinionated, for example, Kubernetes Namespace corresponds to GCP project. Though it might be enabled to be used as part of a Composition when we support namespaced CRs to be used as composition member.
    - rossmohax 6y ago
      Every config connector resource can be annotated with the project for GCP resource to be created in. Namespece to GCP project mapping is encouraged, but not enforced, you are still free to create resources in multiple projects from a single namespace as well as in a single project from multiple namespaces.
      - monus 6y ago
        Oh, thanks for letting me know!
closeparen 6y ago
"Kubernetes but with less stuff" is a valuable niche that I'm glad someone is targeting.
- 4636760295 6y ago
  That's a fair point, I guess it depends on your use case. The risk, however, is that the powers that be at HashiCorp one day decide to abandon Nomad once they realize it will never be a profit centre for them.
  - otterley 6y ago
    Nomad is open source (or, at least, a significant subset of it is). Anyone is able to continue to improve it, even if Hashicorp is no longer paying people to work on it.
    - 4636760295 6y ago
      I used to do sales for an enterprise "open source" software product, so I get it, but the truth is as soon as someone stops paying people to keep the project going it will die.
      - toomuchtodo 6y ago
        Who pays for k8s development?
        closeparen 6y ago
        Google. Commoditizing the cloud is their strategy to make GCP relevant.
        toomuchtodo 6y ago
        Question was rhetorical, I still appreciate the effort though.
sciurus 6y ago
Looked at https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform https://www.hashicorp.com/blog/deploy-any-resource-with-the-... ?
halbritt 6y ago
Why would you want TF in k8s? That hurts my soul. Isn't exactly declarative, nor really imperative, mostly but not entirely idempotent, and worst of all, it has faux-state that may or may not reflect the actual state of the services. I used to love Terraform, but it broke my heart. Now we're done.

api 6y ago

Which features is Nomad missing? Feature count comparisons are meaningless unless the features are tied to actual important use cases. Lots of software is encrusted with rarely used features that just add complexity.

4636760295 6y ago
The main missing feature is that it's not kubernetes.
- aganame 6y ago
  You have an extra word there.
SteveNuts 6y ago
For us it's not a feature missing, but the mindshare is low, and there's not really any prebuilt examples of how to run and maintain services long term reliably.
DavyJone 6y ago
Does it have "easy" side-cars/init/post? Last time I checked those were missing for example.
- adadgar 6y ago
  It does! Here is the relevant documentation and examples: https://www.nomadproject.io/docs/job-specification/lifecycle/ https://www.nomadproject.io/docs/job-specification/lifecycle... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob
  - DavyJone 6y ago
    Ill check that out, I really think K8s is becoming a bit too complex for many customers. Thanks!
q3k 6y ago
The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.
- chucky_z 6y ago
  Preemption is coming in the next release in OSS :)
  - deleted 6y ago
    [deleted]
- BurritoAlPastor 6y ago
  We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.
  - q3k 6y ago
    For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service. Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.
- schmichael 6y ago
  Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion. (I'm the Nomad Team Lead at HashiCorp)
syllogism 6y ago
A few long-standing feature requests I've noticed a lot: * No autoscaling * Can't reserve entire CPU core: https://github.com/hashicorp/nomad/issues/289 https://github.com/hashicorp/nomad/issues/289 * No way to run jobs sequentially: https://github.com/hashicorp/nomad/issues/419 https://github.com/hashicorp/nomad/issues/419
- jambay 6y ago
  Autoscaling is in Tech Preview as of March: https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-tech-preview/ https://www.hashicorp.com/blog/hashicorp-nomad-autoscaling-t... Disclaimer: I work at HashiCorp in Product Management.
  - syllogism 6y ago
    Nice! It's definitely been one of the bigger holes. I hope you'll also consider sequential tasks in a single jobfile. Without it it's kind of awkward because you have to be passing shell scripts around just to run a setup step before your actual workload.
    - nickethier 6y ago
      We’ve also started to implement this with a new task dependencies feature in the latest release. Check out: https://www.hashicorp.com/resources/preview-of-nomad-0-11-task-dependencies/ https://www.hashicorp.com/resources/preview-of-nomad-0-11-ta... https://learn.hashicorp.com/nomad/task-deps/interjob https://learn.hashicorp.com/nomad/task-deps/interjob There’s more work going on there to further improve the feature in upcoming releases. Disclaimer: I’m an engineer on the Nomad team.

monus 6y ago

Crossplane https://crossplane.io https://crossplane.io might be what you’re looking for with its bunch of controllers and a nice composition API. One of the best features is that you can bundle a CR to request a MySQL database and it will be satisfied with whatever config is in your cluster so that app only declares the need but not care how it’s done. Disclaimer: I’m one of the maintainers of Crossplane.

mleonard 6y ago
Any chance you plan to integrate with Google's Config Connector? It's very similar to crossplane (but gcp specific). https://cloud.google.com/config-connector/docs/overview https://cloud.google.com/config-connector/docs/overview
- monus 6y ago
  I think that'd be a bit challenging because Config Connector is highly opinionated, for example, Kubernetes Namespace corresponds to GCP project. Though it might be enabled to be used as part of a Composition when we support namespaced CRs to be used as composition member.
  - rossmohax 6y ago
    Every config connector resource can be annotated with the project for GCP resource to be created in. Namespece to GCP project mapping is encouraged, but not enforced, you are still free to create resources in multiple projects from a single namespace as well as in a single project from multiple namespaces.
    - monus 6y ago
      Oh, thanks for letting me know!

closeparen 6y ago

"Kubernetes but with less stuff" is a valuable niche that I'm glad someone is targeting.

4636760295 6y ago
That's a fair point, I guess it depends on your use case. The risk, however, is that the powers that be at HashiCorp one day decide to abandon Nomad once they realize it will never be a profit centre for them.
- otterley 6y ago
  Nomad is open source (or, at least, a significant subset of it is). Anyone is able to continue to improve it, even if Hashicorp is no longer paying people to work on it.
  - 4636760295 6y ago
    I used to do sales for an enterprise "open source" software product, so I get it, but the truth is as soon as someone stops paying people to keep the project going it will die.
    - toomuchtodo 6y ago
      Who pays for k8s development?
      - closeparen 6y ago
        Google. Commoditizing the cloud is their strategy to make GCP relevant.
        toomuchtodo 6y ago
        Question was rhetorical, I still appreciate the effort though.

sciurus 6y ago

Looked at https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform https://www.hashicorp.com/blog/deploy-any-resource-with-the-... ?

halbritt 6y ago

Why would you want TF in k8s? That hurts my soul. Isn't exactly declarative, nor really imperative, mostly but not entirely idempotent, and worst of all, it has faux-state that may or may not reflect the actual state of the services. I used to love Terraform, but it broke my heart. Now we're done.