3 ms·

The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job)

by q3k 6y ago

The free version of Nomad is missing, IIUC: - no quotas for teams/projects/organizations - no preemption (ie higher priority job preempts lower priority job) - no namespacing So generally it's somewhat useless in organizations where there are multiple different teams that should be able to coexist on a cluster without stepping on eachothers' toes, or even where you want a CI system to access the cluster in a safe manner.

chucky_z 6y ago
Preemption is coming in the next release in OSS :)
- deleted 6y ago
  [deleted]
BurritoAlPastor 6y ago
We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.
- q3k 6y ago
  For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service. Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.
schmichael 6y ago
Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion. (I'm the Nomad Team Lead at HashiCorp)
- tetha 6y ago
  This is going to make our AI team very happy because they can just dump experiments into the cluster at low-priority so those'll be done when those are done. It's also going to make the operators very unhappy because it'll be harder to monitor actual memory utilization (allocated memory vs memory really in use) in order to plan cluster extensions. Are there some tools around or work planned to make this kind of scaling and utilization easier?
  - schmichael 6y ago
    I was going to say "yes!" because we do have some telemetry/metrics improvements coming up, but then I realized those won't address your use case. It seems like aggregating cluster resource usage by job priority is what we need for that. Our UI might be able to calculate that, but I don't think our metrics include job priority so there's no way for your existing tooling to display it. Please file an issue or link me to an existing issue of you have time. This seems really compelling!
- deleted 6y ago
  [deleted]
dmwilcox 6y ago
Enterprise version has all of those things ;) For my personal use I don't need them and for a business Enterprise won't break the bank. You'd be surprised how much your management might be interested in having an escalation path if everything goes south and you need a hot fix. I can vouch that Enterprise support is worth it if you're on a small team that can't spend all day on this stuff

by q3k 6y ago

chucky_z 6y ago

Preemption is coming in the next release in OSS :)

deleted 6y ago
[deleted]

BurritoAlPastor 6y ago

We run everything in Nomad for all our teams, but we don’t use any of the features you mention, and it’s not causing issues. We wrote some metrics for how much memory all the allocations are claiming, and we throw another EC2 instance in the cluster when we’re running out. Works pretty well. Preemption seems like a nice feature, but I imagine getting teams to coordinate their preemption values would be a political nightmare.

q3k 6y ago
For me the ability to squeeze in some jobs in between the cracks at best effort priority is crucial (ie. any sort of high latency batch processing / experiments). I wouldn't want these extremely low priority jobs to compete with, I don't know, an actual customer facing service. Also, we run on bare metal, so there really isn't a way to request extra capacity within seconds.

schmichael 6y ago

Preemption is going OSS! https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md https://github.com/hashicorp/nomad/blob/master/CHANGELOG.md We fully intend to migrate more features to OSS in the future -- especially as we build out more enterprise features. As you can imagine building a sustainable business is quite the balancing act, and there's constant internal discussion. (I'm the Nomad Team Lead at HashiCorp)

tetha 6y ago
This is going to make our AI team very happy because they can just dump experiments into the cluster at low-priority so those'll be done when those are done. It's also going to make the operators very unhappy because it'll be harder to monitor actual memory utilization (allocated memory vs memory really in use) in order to plan cluster extensions. Are there some tools around or work planned to make this kind of scaling and utilization easier?
- schmichael 6y ago
  I was going to say "yes!" because we do have some telemetry/metrics improvements coming up, but then I realized those won't address your use case. It seems like aggregating cluster resource usage by job priority is what we need for that. Our UI might be able to calculate that, but I don't think our metrics include job priority so there's no way for your existing tooling to display it. Please file an issue or link me to an existing issue of you have time. This seems really compelling!
deleted 6y ago
[deleted]

dmwilcox 6y ago

Enterprise version has all of those things ;) For my personal use I don't need them and for a business Enterprise won't break the bank. You'd be surprised how much your management might be interested in having an escalation path if everything goes south and you need a hot fix. I can vouch that Enterprise support is worth it if you're on a small team that can't spend all day on this stuff