3 ms·
> A single regression within a monolithic code base can bring the whole system (in this case, all of Uber) down. Serious question: why though? I can't think of
by nivertech 3y ago
> A single regression within a monolithic code base can bring the whole system (in this case, all of Uber) down.
Serious question: why though? I can't think of a backend system where a single regression could bring the whole thing down. If something blows it up, it rarely takes the whole application down. What's this scenario exactly? Maybe I'm only thinking of certain types of applications, but a single failure somewhere needn't bring the whole thing down.
A crash in a stateless distributed tier (e.g. web servers) will not take the whole system down.
But a crash while processing any request in a stateful non-distributed multithreaded system, can bring it down. It's much harder to make stateful systems resilient and fault tolerant. Usually it's solved by moving the state somewhere else (e.g. databases, message queues, or Kafka-style AOLs).
Exception: monolithic stateful systems using BEAM/OTP/Erlang/Elixir. Here the blast radius is per Erlang process, so in the worst case only a single request/user/session should be affected. There are some classes of errors (e.g. out-of-memory, overloaded process message queues, or crash in native code library) which can bring down an entire Erlang node. This can be mitigated by running a cluster of several Erlang nodes (which is supported out-of-the box by BEAM VM and OTP).
- CharlieYuuu 3y agoI was experience a case : whole system (a stateless distributed tier) down caused by a user input (the user try again and again until all instances down).
- nivertech 3y agoCrashed instances should restart. On EC2 one can put all instances running stateless code into autosacling ground behind load balancer, or ReplicaSet for deployment/service in k8s. BTW: I expirienced the entier us-east region going down just in time when I was demoing my product to a VC. Turns out it was a bug in an AWS Management Console for that region.
- knewter 3y agoErlang/elixir is a very nice middle ground situation where you can build "monolithically" but also inherently be building services that are isolated, and you can scale to 50-150 nodes before getting any more complicated. 150 nodes is a lot. I love the beam.
- nivertech 3y agoThat's for the standard full mesh clustering, Erlang distribution can be further tuned with hidden nodes, etc. I think WhatsApp scaled clusters to 10_000 nodes[1]. I personally only run max 10 nodes Erlang cluster (on physical servers) in production, and ~ 20 on AWS for dev/testing workloads. But I think it's better to have a federation of smaller clusters, or separate cluster per datacenter/AZ. Erlang distribution isn't a silver bullet, though. But for 80% of cases it's enough. --- [1] Scaling Erlang cluster to 10,000 nodes https://codesync.global/media/maxim-fedorov-scaling-erlang-cluster-to-10-000-nodes/ https://codesync.global/media/maxim-fedorov-scaling-erlang-c...
- knewter 3y agoYeah then also there's partisan. I didn't figure anyone that needed to go beyond 150 nodes would fail to learn this stuff, but I appreciate you mentioning it so I don't accidentally drive-by convince someone there's an upper limit rather than engineering tradeoffs :)