3 ms·

Sounds like they need to switch to Kubernetes? I kid of course. One of the best post-mortems I've seen. I'm sure there are K8s horror stories out there of etcd

• schoolornot 5y ago
  The one thing you can say about Nomad is that's generally incredibly scalable compared to Kubernetes. At 1000+ nodes over multiple datacenters, things in Kube seem to break down.
  • tapoxi 5y ago
    Do they still? GKE supports 15,000 nodes per cluster.
• spydum 5y ago
  you joke, but it's precisely this: >Critical monitoring systems that would have provided better visibility into the cause of the outage relied on affected systems, such as Consul. This combination severely hampered the triage process. which gives me goosebumps whenever I hear people proselytizing everything run on Kubernetes. At some point, it makes good sense to keep capabilities isolated from each other, especially when those functions are key to keeping the lights on. Mapping out system dependencies (either systems, software components, etc) is really the soft underbelly of most tech stacks.
• YATA0 5y ago
  >Sounds like they need to switch to Kubernetes? Hah! Good one!