5 ms·

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 (https://www.hashicorp.com/bl

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 (https://www.hashicorp.com/blog/announcing-hashicorp-consul-1-9 https://www.hashicorp.com/blog/announcing-hashicorp-consul-1...) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for long enough that the asset caches went fully cold, so spinning back up to 100% capacity would put far more load than usual on the servers meaning recovery needed to be a very slow incremental rollout to warm them. Messy situation all around, but if you want the real root cause pay attention to the consul change log over the next few weeks I’d say.

ZephyrBlu 5y ago
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
- gorbypark 5y ago
  Consul is a service mesh, vault is secret storage and nomad is for workload orchestration.
  - ZephyrBlu 5y ago
    So relating to things I do know about, consul is like cloud infra, vault is a service for secret storage and nomad is like Kubernetes?
    - dilyevsky 5y ago
      Consul is like etcd but has some extra features built in like service discovery and l7 proxy so they market it as full blown service mesh a la istio, the other two are spot on
      - mianos 5y ago
        I think this is more correct. Like Etcd in that it is a distributed kv store with consensus. It has a dns interface that facilitates service discovery but it can be used for so much more. Infrastructure like global locks, coordination or even just a fault tolerant kv database, as it is for vault itself.
- judge2020 5y ago
  The home pages for the products are helpful https://www.consul.io/ https://www.consul.io/ https://www.vaultproject.io/ https://www.vaultproject.io/ https://www.nomadproject.io/ https://www.nomadproject.io/
- inside_edition 5y ago
  Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler. You tell it what you need to run and what their memory/cpu requirements are and it finds a space in your physical infrastructure to run that. The apps it runs normally needs secrets from vault and communicate with services discovered via consul. They all are well integrated and build off of each other, kind of like layers an onion where your app services are the outer layer of the onion. Consul failing this badly is like the core of the onion going rotten. There’s not much saving it at that point, you need to grow a new onion from the inside out, but that takes time.
  - ZephyrBlu 5y ago
    Thanks for the explanation. I see how that would cause a cascading failure.
  - colordrops 5y ago
    > Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?
    - lmm 5y ago
      Dynamic-first, better support for failover etc.. It's probably not doing anything that you couldn't build on top of DNS, but it's designed from the ground up for clustered deployments.
    - anilgulecha 5y ago
      Really high level, it doesn't. It's an internal dynamic DNS, essentially.
      - zeeZ 5y ago
        One of their primary query interfaces actually even is DNS.
- maccam94 5y ago
  I'd recommend reading the official Hashicorp website for that, but the gist is that nomad is another way of deploying apps in containers (similar to kubernetes), consul is how apps find other apps to talk to (kinda like DNS but with health checking built-in so they don't get stale info), and vault is how apps retrieve the credentials they need to connect to other services.
- Rebelgecko 5y ago
  I haven't used most of these but here's my possibly-flawed understanding: Consul: It's used to help manage cloud networking so that your application doesn't have to worry about IP addresses, datacenter locations, punching through firewalls, etc. Think of the situation where you have a zillion microservices talking to each other on different machines- it makes it easier for them to find each other. It also includes a distributed key-value store Vault: If you've used a password manager in your browser, imagine that but distributed and on steroids. You can use it to share credentials with groups. It also has some APIs to help with encryption/decryption and includes a key-value store Nomad: Roughly similar to Kubernetes (although with good support for non-containerized software). It's used to orchestrate software. For example when you have a bunch of programs you want to run 24/7 , you can specify what machines they should run on (for example at the datacenter/region level), what resources to give the programs, how to handle hardware failures, etc.
- softveda 5y ago
  https://www.hashicorp.com/cloud-operating-model https://www.hashicorp.com/cloud-operating-model
dilyevsky 5y ago
> no consul/vault meant no nomad, no nomad meant no application servers. Not super familiar with nomad but it seems like that would not necessarily follow. For example if etcd goes fully dark in kubernetes cluster things will mostly continue running for a while unless something also crashes the servers
- inside_edition 5y ago
  True for long running services, but not necessarily ephemeral workloads. Those need a system to schedule them or else once they complete they just stop running. Also, as long running services crash or reboot naturally they need to be rescheduled or else your cluster slowly dies, and as your cluster size increases the mtbf decreases and the need to reschedule workloads continually increases.
terom 5y ago
Choice quotes from their PR piece: https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox > We didn’t want to choose any technology that requires the company to drive deep expertise, almost to the point where you have to be a code contributor back into the project to get what you want. Nomad is just very easy to adopt. Better be damn sure you have your 24/7 vendor support contracts in order if and when shit does hit the fan.
- jpgvm 5y ago
  To be fair Nomad failed because it's reliant on Consul. This would be the equivalent of having a k8s outage because etcd is down. Their mistake here is thinking that because you can understand the code of a higher level service that somehow you don't need deep knowledge of it's dependencies. That is a deeply naive view and they paid the price.
- xorcist 5y ago
  > > We didn’t want to choose any technology that requires the company to drive deep expertise, That's a beautifully concise quote which neatly summarizes what contemporary IT values.
  - op00to 5y ago
    You don’t need deep etcd experience to run k8s. Why is that bad?
    - 0xbadcafebee 5y ago
      You've never had to work a 3 day weekend to recover an etcd failure? Guess I'm just lucky
      - op00to 5y ago
        I’m on the vendor side now, so every day is helping customers recover from etcd failures caused by slow storage due to them not listening to documented requirements. ;)
tyingq 5y ago
>No functional consul meant no vault Vault does support other backends, like postgres, etcd, zookeeper, and others. Though if Consul is the backend, Vault is also registered as a service within the Consul mesh.

ZephyrBlu 5y ago

As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?

gorbypark 5y ago
Consul is a service mesh, vault is secret storage and nomad is for workload orchestration.
- ZephyrBlu 5y ago
  So relating to things I do know about, consul is like cloud infra, vault is a service for secret storage and nomad is like Kubernetes?
  - dilyevsky 5y ago
    Consul is like etcd but has some extra features built in like service discovery and l7 proxy so they market it as full blown service mesh a la istio, the other two are spot on
    - mianos 5y ago
      I think this is more correct. Like Etcd in that it is a distributed kv store with consensus. It has a dns interface that facilitates service discovery but it can be used for so much more. Infrastructure like global locks, coordination or even just a fault tolerant kv database, as it is for vault itself.
judge2020 5y ago
The home pages for the products are helpful https://www.consul.io/ https://www.consul.io/ https://www.vaultproject.io/ https://www.vaultproject.io/ https://www.nomadproject.io/ https://www.nomadproject.io/
inside_edition 5y ago
Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler. You tell it what you need to run and what their memory/cpu requirements are and it finds a space in your physical infrastructure to run that. The apps it runs normally needs secrets from vault and communicate with services discovered via consul. They all are well integrated and build off of each other, kind of like layers an onion where your app services are the outer layer of the onion. Consul failing this badly is like the core of the onion going rotten. There’s not much saving it at that point, you need to grow a new onion from the inside out, but that takes time.
- ZephyrBlu 5y ago
  Thanks for the explanation. I see how that would cause a cascading failure.
- colordrops 5y ago
  > Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?
  - lmm 5y ago
    Dynamic-first, better support for failover etc.. It's probably not doing anything that you couldn't build on top of DNS, but it's designed from the ground up for clustered deployments.
  - anilgulecha 5y ago
    Really high level, it doesn't. It's an internal dynamic DNS, essentially.
    - zeeZ 5y ago
      One of their primary query interfaces actually even is DNS.
maccam94 5y ago
I'd recommend reading the official Hashicorp website for that, but the gist is that nomad is another way of deploying apps in containers (similar to kubernetes), consul is how apps find other apps to talk to (kinda like DNS but with health checking built-in so they don't get stale info), and vault is how apps retrieve the credentials they need to connect to other services.
Rebelgecko 5y ago
I haven't used most of these but here's my possibly-flawed understanding: Consul: It's used to help manage cloud networking so that your application doesn't have to worry about IP addresses, datacenter locations, punching through firewalls, etc. Think of the situation where you have a zillion microservices talking to each other on different machines- it makes it easier for them to find each other. It also includes a distributed key-value store Vault: If you've used a password manager in your browser, imagine that but distributed and on steroids. You can use it to share credentials with groups. It also has some APIs to help with encryption/decryption and includes a key-value store Nomad: Roughly similar to Kubernetes (although with good support for non-containerized software). It's used to orchestrate software. For example when you have a bunch of programs you want to run 24/7 , you can specify what machines they should run on (for example at the datacenter/region level), what resources to give the programs, how to handle hardware failures, etc.
softveda 5y ago
https://www.hashicorp.com/cloud-operating-model https://www.hashicorp.com/cloud-operating-model

dilyevsky 5y ago

> no consul/vault meant no nomad, no nomad meant no application servers. Not super familiar with nomad but it seems like that would not necessarily follow. For example if etcd goes fully dark in kubernetes cluster things will mostly continue running for a while unless something also crashes the servers

inside_edition 5y ago
True for long running services, but not necessarily ephemeral workloads. Those need a system to schedule them or else once they complete they just stop running. Also, as long running services crash or reboot naturally they need to be rescheduled or else your cluster slowly dies, and as your cluster size increases the mtbf decreases and the need to reschedule workloads continually increases.

terom 5y ago

Choice quotes from their PR piece: https://www.hashicorp.com/case-studies/roblox https://www.hashicorp.com/case-studies/roblox > We didn’t want to choose any technology that requires the company to drive deep expertise, almost to the point where you have to be a code contributor back into the project to get what you want. Nomad is just very easy to adopt. Better be damn sure you have your 24/7 vendor support contracts in order if and when shit does hit the fan.

jpgvm 5y ago
To be fair Nomad failed because it's reliant on Consul. This would be the equivalent of having a k8s outage because etcd is down. Their mistake here is thinking that because you can understand the code of a higher level service that somehow you don't need deep knowledge of it's dependencies. That is a deeply naive view and they paid the price.
xorcist 5y ago
> > We didn’t want to choose any technology that requires the company to drive deep expertise, That's a beautifully concise quote which neatly summarizes what contemporary IT values.
- op00to 5y ago
  You don’t need deep etcd experience to run k8s. Why is that bad?
  - 0xbadcafebee 5y ago
    You've never had to work a 3 day weekend to recover an etcd failure? Guess I'm just lucky
    - op00to 5y ago
      I’m on the vendor side now, so every day is helping customers recover from etcd failures caused by slow storage due to them not listening to documented requirements. ;)

tyingq 5y ago

>No functional consul meant no vault Vault does support other backends, like postgres, etcd, zookeeper, and others. Though if Consul is the backend, Vault is also registered as a service within the Consul mesh.