3 ms·

We run our databases (ScyllaDB, Elasticsearch) on our workload orchestrator (Nomad), just like any other job. The only difference is that the servers running th

by heipei 4y ago

We run our databases (ScyllaDB, Elasticsearch) on our workload orchestrator (Nomad), just like any other job. The only difference is that the servers running the database workloads are not available for scheduling any other jobs, that way the database will be the only thing running on them. I can't overstate how much easier this has made database operations for us. We get to keep all of our config in our Nomad repo, use the same mechanism for secrets management, service discovery, the fleet-wide logging and auditing sidecars, etc. Plus Nomad allows rolling updates with automatic revert which makes upgrading 10+ nodes of Elasticsearch one-by-one with healthchecks a total breeze. Since we've had this setup we are able to trigger database release updates with a one-line change and a single command.

motoboi 4y ago
Those database are distributed cluster native databases. They have failover, cluster elections, node auto joining mechanisms. Vanilla postgres have none of that.
- heipei 4y ago
  True, that makes running these databases a bit easier since there are multiple nodes that can be restarted in any order. However the main point still stands: You could just as well run Postgres on Nomad/k8s by pinning it to a specific node and excluding that node from the general job scheduling pool. We do that for a single-node Redis instance for example. Still vastly superior to managing unique snowflakes nodes just for the DBs via ansible/systemd.
nanmu42 4y ago
> secrets management, service discovery, the fleet-wide logging and auditing sidecars Interesting, I have run my hobby projects by Nomad satisfiedly and are looking for ways to run serious workloads. Would you like to share more wisdom? How do you accomplish things above? Thanks.
- MoOmer 4y ago
  Consul + Vault!
- jxfive 4y ago
  Not OP but the last couple releases of Nomad have added quite a few QoL features without having to reach for/or setup Consul or Vault depending on your needs. At least in regard to basic service discovery and secrets management. I'm unsure if something like open policy agent can directly work with the orchestrator and may have to be at the application level. https://www.hashicorp.com/blog/nomad-service-discovery https://www.hashicorp.com/blog/nomad-service-discovery https://www.hashicorp.com/blog/nomad-1-4-adds-nomad-variables-and-updates-service-discovery https://www.hashicorp.com/blog/nomad-1-4-adds-nomad-variable...
deleted 4y ago
[deleted]

by heipei 4y ago

motoboi 4y ago

Those database are distributed cluster native databases. They have failover, cluster elections, node auto joining mechanisms. Vanilla postgres have none of that.

heipei 4y ago
True, that makes running these databases a bit easier since there are multiple nodes that can be restarted in any order. However the main point still stands: You could just as well run Postgres on Nomad/k8s by pinning it to a specific node and excluding that node from the general job scheduling pool. We do that for a single-node Redis instance for example. Still vastly superior to managing unique snowflakes nodes just for the DBs via ansible/systemd.

nanmu42 4y ago

> secrets management, service discovery, the fleet-wide logging and auditing sidecars Interesting, I have run my hobby projects by Nomad satisfiedly and are looking for ways to run serious workloads. Would you like to share more wisdom? How do you accomplish things above? Thanks.

MoOmer 4y ago
Consul + Vault!
jxfive 4y ago
Not OP but the last couple releases of Nomad have added quite a few QoL features without having to reach for/or setup Consul or Vault depending on your needs. At least in regard to basic service discovery and secrets management. I'm unsure if something like open policy agent can directly work with the orchestrator and may have to be at the application level. https://www.hashicorp.com/blog/nomad-service-discovery https://www.hashicorp.com/blog/nomad-service-discovery https://www.hashicorp.com/blog/nomad-1-4-adds-nomad-variables-and-updates-service-discovery https://www.hashicorp.com/blog/nomad-1-4-adds-nomad-variable...

deleted 4y ago

[deleted]