3 ms·

Thx .. pretty decent idea. You are right that that there isn't a low-effort, modern way to get reliable datastructures. My team studied hazelcast a few years ba

Thx .. pretty decent idea. You are right that that there isn't a low-effort, modern way to get reliable datastructures. My team studied hazelcast a few years back as part of evaluating the distributed system we built ourselves. It definitely has some nice features. Here is the issue we faced though .. we had a system with great performance but not partition tolerance. We joked that we should call it almost-right consensus (we used CRDTs in some nifty ways). This was a sad joke as there is no way we can publish this.

PaulHoule 5y ago
It seems to me that if you are worried about partition tolerance you are never going to be happy. Perhaps you are ignorant and happy if you decide to accept it. My take is that most people can live with occasional "split brain" problems in the control plane so long as the algorithms involved have a latch structure with a bias towards progressing. (Think of the arguments as to why Bubble Sort completes even though it has free loops.) You might get some work units stuck and performance will be less than optimal but the main fear is that the degraded state is so hard to detect that it goes on for a long time. Data plane problems are worse because people really do expect to get the right answer. There are many things other than "distributed system BS" that lead to incorrectness, such as floating point numbers, but I like the restartability paradigm behind Spark.