3 ms·

What options are there for managing persistent storage in Nomad? I've been through the very basic initial getting started tutorial, but haven't had a chance to

by jerrac 4y ago

What options are there for managing persistent storage in Nomad? I've been through the very basic initial getting started tutorial, but haven't had a chance to dig any deeper. When I was trying kube, I found Longhorn to be the one thing I liked. Is there anything like that for Nomad? I've been trying to get a MariaDB Galera cluster running on Docker Swarm for the past couple weeks and have been somewhat stymied a few things. Something like Longhorn would help a lot.

schmichael 4y ago
Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas. That being said they’re often glaring. So please report an issue to Nomad and the upstream of you hit an issue! Nomad has 2 other storage features: 1. Host volumes where nodes can advertise a volume by name and jobs can request being placed on nodes with that volume available. I think a lot of folks run databases this way so they can statically define their database nodes and not worry about the numerous runtime points of failure CSI introduces. 2. Ephemeral disk stickiness and migration: during deployments an instance of a job’s intrinsic local storage can be set to either get reused on the same node (sticky) or migrated to wherever the new instance is placed. Ephemeral disks is the oldest storage solution and is only a best effort (unlike CSI which may be able to reliably reattach and existing volume to a new node in the case of node failure). However I’ve heard of a lot of people happily using it for already distributed databases like Elasticsearch or Cassandra where the database can tolerate nomad’s best effort failing. (Sorry for lack of links as I’m on mobile. Googling the key phrases you care about should get you to our docs quickly and easily)
- jerrac 4y ago
  Thanks! that gives a really good start.

by jerrac 4y ago

schmichael 4y ago

Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas. That being said they’re often glaring. So please report an issue to Nomad and the upstream of you hit an issue! Nomad has 2 other storage features: 1. Host volumes where nodes can advertise a volume by name and jobs can request being placed on nodes with that volume available. I think a lot of folks run databases this way so they can statically define their database nodes and not worry about the numerous runtime points of failure CSI introduces. 2. Ephemeral disk stickiness and migration: during deployments an instance of a job’s intrinsic local storage can be set to either get reused on the same node (sticky) or migrated to wherever the new instance is placed. Ephemeral disks is the oldest storage solution and is only a best effort (unlike CSI which may be able to reliably reattach and existing volume to a new node in the case of node failure). However I’ve heard of a lot of people happily using it for already distributed databases like Elasticsearch or Cassandra where the database can tolerate nomad’s best effort failing. (Sorry for lack of links as I’m on mobile. Googling the key phrases you care about should get you to our docs quickly and easily)

jerrac 4y ago
Thanks! that gives a really good start.