4 ms·
Another big takeaway for me: "This isn’t the end of the world–it does illustrate the fragility of a system with three distinct quorums, all of which must be av
by sbilstein 11y ago
Another big takeaway for me:
"This isn’t the end of the world–it does illustrate the fragility of a system with three distinct quorums, all of which must be available and connected to one another, but there will always be certain classes of network failure that can break a distributed scheduler."
As we build increasingly complex distributed from what are essentially modular distributed systems themselves, this is something to be aware of. The failure characteristics of all the underlying systems will bubble up in surprising ways.
- fweespeech 11y agoYes but the number of times I've had to manually heal a cluster in production in my professional life can be counted on one hand. A network partition like Jepsen tests for is a once-every-other-month problem due to network maintenance, hardware failures, etc. So yeah, it is the end of the world for me. I like not having to wake up in the middle of the night every few months.
- SEJeff 11y agoOne thing to note here, is that upstream mesos is working on removing Zookeeper as a dependency. That would at a minimum remove one of the three pieces. Note that I briefly evaluated Chronos, but tossed it aside as a toy when realizing it didn't support any form of constraints. Aurora is a very nice scheduler for HA / distributed cron (with constraints!) and long running services. It would be fantastic if Kyle would test out Aurora, if not only to break it so that upstream fixes it. He is generally successful in breaking ALL THE THINGS.
- bbrazil 11y agoMesos is looking at allowing things other than Zookeeper to be the dependency, I'm not aware of them removing the dependency altogether. https://issues.apache.org/jira/browse/MESOS-1806 https://issues.apache.org/jira/browse/MESOS-1806
- fidget 11y agoI do wonder why the GP would want to, given how ridiculously reliable zookeeper is.
- SEJeff 11y agoOh I love zookeeper and have no issues whatsoever with it. However, it is almost universally the first thing I hear people moan about when someone suggest they use Mesos. People don't like Zookeeper.
- SEJeff 11y agoIn MESOS-1806, it mentions replacing ZK with ReplicatedLog or etcd. The replicated log is a mesos native construct, that has no external dependencies. If you can replace ZK with a native Mesos construct, seems like it allows you to remove ZK entirely. I meant to say "optionally allow removing ZK as a dependency" in the original post. You're totally correct in that regard.
- ianburrell 11y agoThe latest version of Chronos added support for constraints, at least for simple EQUALS constraints. The annoying problem that I also encountered with Chronos was the lack of error reporting from the REST API. Invalid jobs would just return "400 Bad Request" with no error message. The error sometimes wouldn't even be reported in the logs.
- SEJeff 11y agoYeah I was playing with it > 6 months ago, and Aurora was solid then as well.