4 ms·

Related advice based on my days working at Basho: find a way to recognize, and terminate, slow-running (or erratically-behaving) servers. A dead server is much

Related advice based on my days working at Basho: find a way to recognize, and terminate, slow-running (or erratically-behaving) servers. A dead server is much better for a distributed system than a misbehaving one. The latter can bring down your entire application.

rcxdude 1y ago
Indeed, which is why I've heard of failover setups where the backup has a means to make very sure that the main system is off before it takes over (often by cutting the power).
- owl_vision 1y ago
  I did this systematically: at the first sign of outlier in performance one system would move itself to another platform and shut itself down. The shutdown meant turn all services off and let someone log in to investigate and rearrange it again. This system allowed different roles to be assigned to different platform. The platform was bare metal or bhyve vm. It worked perfect.
- xyzzy_plugh 1y ago
  Usually we call this STONITH
westurner 1y ago
Docker and Kubernetes have health check mechanisms to help solve for this; Docker docs > Dockerfile HEALTHCHECK instruction: https://docs.docker.com/reference/dockerfile/#healthcheck https://docs.docker.com/reference/dockerfile/#healthcheck Podman docs > podman-healthcheck-run, docker-healthcheck-run: https://docs.podman.io/en/v5.4.0/markdown/podman-healthcheck-run.1.html https://docs.podman.io/en/v5.4.0/markdown/podman-healthcheck... Kubernetes docs > "Configure Liveness, Readiness and Startup Probes" https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con...
Hortinstein 1y ago
Always thought Riak was a really cool database, distributed by default kinda blew my mind at the time! Would love to hear more about basho
- macintux 1y ago
  10 years later, any details I could offer would be relatively suspect; my memory is pretty poor. Still, I'll make an attempt. Don't write a book based on anything below. I can honestly say it's the best job I ever had, working with the smartest collection of people (I've been fortunate to work at multiple startups, and while Ian Murdock put together a solid crew at Progeny, Basho was just chock full of brilliant people). Riak was designed for rock solid, massive scale data. The problem was that most companies didn't really need that, and by the time someone reached that size, they had a significant investment in databases that were much more developer-friendly. We attempted to make it easier to work with by introducing CRDTs, so developers wouldn't necessarily have to write as much application logic to handle conflicts. Still, it's hard to sell a database when you can't demonstrate simple queries, because your choices were primarily "update static views as you add new data so you don't actually have to run queries in real-time" or "write map-reduce logic in Erlang". And, it was hard to simulate massive scale databases, so while our resilience and operations stories were pretty good, it was challenging to anticipate how, and where, performance might start breaking down. I miss it dearly, but I was also lucky enough to never be on a support call when someone was trying to rescue a failing cluster that served millions of customers. I will say that Erlang made that last scenario much more manageable: the ability to connect to a live node and see exactly what was happening, and the ability to hot patch a function on the fly, was incredibly helpful.

rcxdude 1y ago

Indeed, which is why I've heard of failover setups where the backup has a means to make very sure that the main system is off before it takes over (often by cutting the power).

owl_vision 1y ago
I did this systematically: at the first sign of outlier in performance one system would move itself to another platform and shut itself down. The shutdown meant turn all services off and let someone log in to investigate and rearrange it again. This system allowed different roles to be assigned to different platform. The platform was bare metal or bhyve vm. It worked perfect.
xyzzy_plugh 1y ago
Usually we call this STONITH

westurner 1y ago

Docker and Kubernetes have health check mechanisms to help solve for this; Docker docs > Dockerfile HEALTHCHECK instruction: https://docs.docker.com/reference/dockerfile/#healthcheck https://docs.docker.com/reference/dockerfile/#healthcheck Podman docs > podman-healthcheck-run, docker-healthcheck-run: https://docs.podman.io/en/v5.4.0/markdown/podman-healthcheck-run.1.html https://docs.podman.io/en/v5.4.0/markdown/podman-healthcheck... Kubernetes docs > "Configure Liveness, Readiness and Startup Probes" https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ https://kubernetes.io/docs/tasks/configure-pod-container/con...

Hortinstein 1y ago

Always thought Riak was a really cool database, distributed by default kinda blew my mind at the time! Would love to hear more about basho

macintux 1y ago
10 years later, any details I could offer would be relatively suspect; my memory is pretty poor. Still, I'll make an attempt. Don't write a book based on anything below. I can honestly say it's the best job I ever had, working with the smartest collection of people (I've been fortunate to work at multiple startups, and while Ian Murdock put together a solid crew at Progeny, Basho was just chock full of brilliant people). Riak was designed for rock solid, massive scale data. The problem was that most companies didn't really need that, and by the time someone reached that size, they had a significant investment in databases that were much more developer-friendly. We attempted to make it easier to work with by introducing CRDTs, so developers wouldn't necessarily have to write as much application logic to handle conflicts. Still, it's hard to sell a database when you can't demonstrate simple queries, because your choices were primarily "update static views as you add new data so you don't actually have to run queries in real-time" or "write map-reduce logic in Erlang". And, it was hard to simulate massive scale databases, so while our resilience and operations stories were pretty good, it was challenging to anticipate how, and where, performance might start breaking down. I miss it dearly, but I was also lucky enough to never be on a support call when someone was trying to rescue a failing cluster that served millions of customers. I will say that Erlang made that last scenario much more manageable: the ability to connect to a live node and see exactly what was happening, and the ability to hot patch a function on the fly, was incredibly helpful.