5 ms·
When people talk about security vulnerabilities, they focus on the most glaring holes. "Memory safety!!!!!" etc. Meanwhile, cyber criminals merrily go about the
by 0xbadcafebee 5y ago
When people talk about security vulnerabilities, they focus on the most glaring holes. "Memory safety!!!!!" etc. Meanwhile, cyber criminals merrily go about their day hijacking massive computer systems without even using a memory safety vuln. Because the criminals don't give a crap what kind of holes there are - they any kind will do.
When people talk about distributed consensus algorithms, they're similarly missing the important point. A useful algorithm isn't the end-all be-all of the system. What's important is that the entire system be operated to a high degree of reliability in a way that meets the needs of a specific application. Paxos or not, if your implementation keeps falling over for reasons other than consensus, or you don't build in the tools to properly deal with the knock-on effects of consensus issues, etc, then the whole thing is going to blow up anyway.
No algorithm on its own will make a better system. You need all the other parts of the system to be just as awesome, or it falls apart just the same.
- tester756 5y agohttps://xkcd.com/538/ https://xkcd.com/538/
- jiggawatts 5y agoThis reminds me of a recent project where I was doing some maintenance work on a system where one component was using ZooKeeper, which uses a distributed consensus algorithm similar to Paxos. Guess which component was the only one that failed to come back up after a simple reboot? Turns out that ZooKeeper has default "boot" timeouts that are too short for typical JVM startup times on some hosting platforms, so it just throws up its hands and never tries again after the first attempt. I want to know who was the person that sat down, cracked his knuckles, and then want about writing the code to deliberately add a failure mode to what is otherwise a very failure-tolerant algorithm.
- lmm 5y agoZooKeeper is consistent at all costs and often sacrifices availability for it, by design.
- jiggawatts 5y agoThis superficially sounds like a good design choice, but makes no sense whatsoever if you stop and think about it for a minute. Clustering systems like ZooKeeper are pointless for scenarios where things never fail. They serve no purpose in a system that is infallible, unchanging, and perfect. They're intended for handling failures and other changes.[1] Power outages happen. Temporary but total network outages happen, sometimes across even multi-region deployments. Cluster-wide reboots may need to occur occasionally. Startup might be slow due to a storm of other systems also starting and slowing down everything. These scenarios have to be handled. It's not optional for any system design for failure scenarios. VMware handled it fine. Windows Failover Clustering handled it fine. SQL Server AlwaysOn Availability Groups handled it fine. Kubernetes was back up and running before anything else. ZooKeeper didn't. It just sat there. No nodes were down. No services had crashed. It wasn't out of disk space. No data was corrupted. No "inconsistency" of any kind had occurred. It just decided not to start. Because it was, what? Too impatient? It saw that 60 seconds had elapsed, and apparently 61 seconds is just too long to wait, so no cluster! No! Too slow! Boot faster next time! What if this occurs at 1am on a system without 24/7, round-the-clock support? ZooKeeper will go down, and stay down. In this case this was a planned outage, so I had to stay up until 3:30 am troubleshooting why ZooKeeper wouldn't act like any other cluster I had ever dealt with. Fun. [1] I got this lesson taught to me by a CommVault backup product designer. He explained that "backup products" aren't. They're recovery products, and their only purpose is to be used in a disaster. If they can't handle a disaster, they're worse than worthless, they give a false sense of confidence! CommVault was designed by people that had operated forward IT bases for the military in the Yugoslav civil war. They had to recover systems from data centres that were literally smoking craters in the ground. ZooKeeper struggles with planned reboots. It's not even vaguely in the same league of trustworthiness.
- polskibus 5y agoIs etcd better than zookeeper in this regard?
- jiggawatts 5y ago
- dikei 5y agoWell, all valid consensus algorithms are guaranteed to eventually reach agreement, but it's not possible to guarantee about deadline in an asynchronous, distributed environment. However, in practice, we don't want our system to stall forever waiting for consensus, so a timeout is often used to terminate the algorithm.
- lamontcg 5y agoIts important to have a distributed consensus protocol like Paxos, which has been validated, as a lego block in building any larger system. We need a lot more of those well engineered, proven, validated lego blocks. Memory safety is a cross cutting concern which makes all those lego blocks much safer. We are about 30 years behind the curve and have a lot of ground to make up. That doesn't make these efforts useless, that just means that we need 10 times the effort.
- drivebycomment 5y agoWhat a strawman. The article doesn't claim or imply that Paxos on its own will make a better system. You can't build secure systems without having solid building blocks and primitives. Just like proving encryption algorithms and cryptographic have functions have certain properties is important and essential, proving properties about Paxos is important and useful.
- johnisgood 5y agoSpeaking of proving encryption algorithms and such: https://github.com/Componolit/libsparkcrypto https://github.com/Componolit/libsparkcrypto Ada/SPARK! They added ECDSA, ECGDSA, too. :)