4 ms·

Ask Roblox employees how they feel about Hashicorp products. Terraform is probably the most solid product they have seconded by vault, but after hearing the con

Ask Roblox employees how they feel about Hashicorp products. Terraform is probably the most solid product they have seconded by vault, but after hearing the consul and nomad horror stories, I don't think I could take their products seriously ever, not when kubernetes is setting right there.

vilkkala 3y ago
Could you elaborate on these horror stories?
CaptArmchair 3y ago
Well, here's the link to the post-mortem of said "horror" story on the Roblox blog: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-... TL;DR The outage was caused by (a) they enabled a new streaming feature in Consul under unusualy high read-and-write load, (b) the load conditions triggered a pathological issue in the third-party BoltDB system upon which Consul relies and (c) all of that was exacerbated by having one consul cluster supporting multiple workloads. I'd use quotes to catalog this as a "horror" story, because this was clearly a very specific issue triggered by a specific and complex set of circumstances. The blogpost also mentions that Roblox worked closely with Hashicorp engineers to mitigate the issue, and work towards structural solutions; the post also affirms their choice to manage their infra themselves rather then moving into a public cloud solution. Sure, Kubernetes covers loads of territory. But there definitely are niches where products like Consul & Nomad do add value.
throwaway2023rb 3y ago
Hashicorp enterprise support is rock solid for Nomad, Consul, Vault. If there is a P0 problem, they will root cause, and usually have a fix identified in < 48 hours. All three of those products are taken very seriously - running 10,000+ servers in a single cluster.
busterarm 3y ago
> not when kubernetes is setting right there. Enjoy spending the rest of your life trying to get etcd to cooperate. If you think operating Kubernetes at scale is a cakewalk, you don't have the scale problems you think you do. I'll take consul over etcd ten million times out of ten.
- pcthrowaway 3y ago
  Realistically you're either deploying consul on top of 1-N kubernetes clusters, or Nomad. If deploying on kubernetes, you now have all the problems that come with kubernetes, plus additional problems of trying to get consul working. I spent a week just trying to stand up a federated multi-datacenter deployment of consul on EKS before my company decided it was too much hassle
  - busterarm 3y ago
    the OP was suggesting that it's just obvious to use Kubernetes instead of Nomad. I was saying that anyone who operates large scale Kubernetes knows that you will forever be dealing with tuning etcd and fighting to keep etcd alive. It's an underpinning service of Kubernetes. Roblox's outage was related to the intricacies of running consul and mistakes that they made. The point I was making was that I would rather, at this scale of operation, be running Nomad and optionally Consul and optionally dealing with the intricacies of Consul than running Kubernetes and being _forced_ to deal with what a miserable pain in the ass etcd is. I was saying that running Kubernetes is _not_ the obvious choice -- at least once you're at 10^5+ systems.
    - bastardoperator 3y ago
      Until you have 3 days of downtime and end up on a special build of consul that nobody else has while nearly decimating your companies stock price, reputation, and employee morale, the alternatives start looking better. The point I'm making is that it doesn't handle their scale today and that projects with larger communities, support, and usage exist today. No one said k8s was a silver bullet, it's just an alternative to an already failing infrastructure.

vilkkala 3y ago

Could you elaborate on these horror stories?

CaptArmchair 3y ago

Well, here's the link to the post-mortem of said "horror" story on the Roblox blog: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-... TL;DR The outage was caused by (a) they enabled a new streaming feature in Consul under unusualy high read-and-write load, (b) the load conditions triggered a pathological issue in the third-party BoltDB system upon which Consul relies and (c) all of that was exacerbated by having one consul cluster supporting multiple workloads. I'd use quotes to catalog this as a "horror" story, because this was clearly a very specific issue triggered by a specific and complex set of circumstances. The blogpost also mentions that Roblox worked closely with Hashicorp engineers to mitigate the issue, and work towards structural solutions; the post also affirms their choice to manage their infra themselves rather then moving into a public cloud solution. Sure, Kubernetes covers loads of territory. But there definitely are niches where products like Consul & Nomad do add value.

throwaway2023rb 3y ago

Hashicorp enterprise support is rock solid for Nomad, Consul, Vault. If there is a P0 problem, they will root cause, and usually have a fix identified in < 48 hours. All three of those products are taken very seriously - running 10,000+ servers in a single cluster.

busterarm 3y ago

> not when kubernetes is setting right there. Enjoy spending the rest of your life trying to get etcd to cooperate. If you think operating Kubernetes at scale is a cakewalk, you don't have the scale problems you think you do. I'll take consul over etcd ten million times out of ten.

pcthrowaway 3y ago
Realistically you're either deploying consul on top of 1-N kubernetes clusters, or Nomad. If deploying on kubernetes, you now have all the problems that come with kubernetes, plus additional problems of trying to get consul working. I spent a week just trying to stand up a federated multi-datacenter deployment of consul on EKS before my company decided it was too much hassle
- busterarm 3y ago
  the OP was suggesting that it's just obvious to use Kubernetes instead of Nomad. I was saying that anyone who operates large scale Kubernetes knows that you will forever be dealing with tuning etcd and fighting to keep etcd alive. It's an underpinning service of Kubernetes. Roblox's outage was related to the intricacies of running consul and mistakes that they made. The point I was making was that I would rather, at this scale of operation, be running Nomad and optionally Consul and optionally dealing with the intricacies of Consul than running Kubernetes and being _forced_ to deal with what a miserable pain in the ass etcd is. I was saying that running Kubernetes is _not_ the obvious choice -- at least once you're at 10^5+ systems.
  - bastardoperator 3y ago
    Until you have 3 days of downtime and end up on a special build of consul that nobody else has while nearly decimating your companies stock price, reputation, and employee morale, the alternatives start looking better. The point I'm making is that it doesn't handle their scale today and that projects with larger communities, support, and usage exist today. No one said k8s was a silver bullet, it's just an alternative to an already failing infrastructure.