5 ms·

Infrastructure can never get to any reasonable level of resilience alone without programs being designed for resilience and such ignorance will likely lead to n

Infrastructure can never get to any reasonable level of resilience alone without programs being designed for resilience and such ignorance will likely lead to nasty stuff, like cascading failures, because of massive load differences between normal operations and attempts to handle errors by recreating entire services. But Kubernetes itself is not known for its resilience, on the contrary, its reliability reputation is pretty bad at this point, unlike that of Erlang. Design for resilience is not a small thing. I suggest to read Joe Armstrong's "Making reliable distributed systems in the presence of software errors" [1] as a starting point on this. [1] http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf

Thaxll 8y ago
I'm not saying that you should not add resilience in your program, but that OP claims about "what if printf crashes" well that's it it will crash deal with it. You should ensure your platform will recover if you program crash not try everything possible to not make your program not crash. "Pet vs Cattle" The thing I don't like about Erlang is about the runtime that is mixed between code and infra which I think is not a good idea, it was designed before we made progress with HA platform like Kubernetes. https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md https://github.com/kubernetes/community/blob/master/sig-scal... It should be separated and it's what pretty much everyone is doing nowdays.
- CMCDragonkai 8y ago
  I originally thought the same until I started building ML pipelines. The best would be a direct implementation of first class module systems to handle infrastructure.
- ramchip 8y ago
  Say you have a bug in a minor feature, that most people wouldn’t care about, but triggers the crash. Now someone’s trying to use this feature once or twice per minute. If you rely on the infra for recovery, you’re going to be in serious trouble. There may be 300k users connected to an app instance, and you’re going to be kicking all of them out and restarting an instance every time the minor feature is called. This turns a minor bug into a full-blown outage. Now we’re talking about a “printf crash” as an example, but in my experience the issues are more subtle. I’ve seen a Python app leak file descriptors due to a bug in an object’s destructor, which only happened when an exception was thrown in a certain place. This caused a service outage as the app looked “fine” from the infra’s point of view (it could respond to health check requests), but the functionality was dead. With Erlang processes, using a process per connection, the VM guarantees resources are cleaned up when a process dies, so that kind of issue doesn’t happen. The Zen of Erlang goes a bit deeper into transient issues and why the supervision model helps: https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html

Thaxll 8y ago

I'm not saying that you should not add resilience in your program, but that OP claims about "what if printf crashes" well that's it it will crash deal with it. You should ensure your platform will recover if you program crash not try everything possible to not make your program not crash. "Pet vs Cattle" The thing I don't like about Erlang is about the runtime that is mixed between code and infra which I think is not a good idea, it was designed before we made progress with HA platform like Kubernetes. https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md https://github.com/kubernetes/community/blob/master/sig-scal... It should be separated and it's what pretty much everyone is doing nowdays.

CMCDragonkai 8y ago
I originally thought the same until I started building ML pipelines. The best would be a direct implementation of first class module systems to handle infrastructure.
ramchip 8y ago
Say you have a bug in a minor feature, that most people wouldn’t care about, but triggers the crash. Now someone’s trying to use this feature once or twice per minute. If you rely on the infra for recovery, you’re going to be in serious trouble. There may be 300k users connected to an app instance, and you’re going to be kicking all of them out and restarting an instance every time the minor feature is called. This turns a minor bug into a full-blown outage. Now we’re talking about a “printf crash” as an example, but in my experience the issues are more subtle. I’ve seen a Python app leak file descriptors due to a bug in an object’s destructor, which only happened when an exception was thrown in a certain place. This caused a service outage as the app looked “fine” from the infra’s point of view (it could respond to health check requests), but the functionality was dead. With Erlang processes, using a process per connection, the VM guarantees resources are cleaned up when a process dies, so that kind of issue doesn’t happen. The Zen of Erlang goes a bit deeper into transient issues and why the supervision model helps: https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html