9 ms·
Jepsen: Radix DLT 1.0-Beta.35.1
- NelsonMinar 5y agoI admire aphyr's ability to dive into a new and complex distributed systems technology and understand it enough to evaluate it for correctness. I hope the Radix developers have ears to listen, the comment in the report "RDX Works informed Jepsen that the blockchain/DLT community had developed idiosyncratic definitions of safety and liveness" is not encouraging.
- daenney 5y ago> To Jepsen’s surprise, RDX Works asserted that phenomena such as aborted read, intermediate read, and lost writes do not constitute safety violations (in the blockchain sense). RDX Works claims that to describe these errors as safety violations would not be understood by readers from a blockchain background; this report is therefore “factually incorrect”. On these grounds, RDX Works requested that Jepsen delete any mention of our findings from the abstract of this report. That certainly does not inspire any confidence. > Jepsen respectfully declines to do so. Thank you for sticking to that.
- Mleekko 5y agoWhat Radix say is true though. In private DBs, reads from the DB node are considered transactions and need to follow the same rules as writes. But on public blockchains(ledgers) only state manipulation is what matters. For example, Metamask obtaining an address balance would be a transaction, but no one calls it that way because it doesn't modify the state.
- daenney 5y agoSure. But they could have asked for the additional clarifications or context to be added to make this clear, instead of requesting a bunch of stuff be removed because they’re concerned it’ll paint them in a bad light.
- faraz85 5y agoAs I understand from the report, no request was made to remove the content but to leave terms like "liveness break" and "safety break" out of the abstract until those terms were defined in the main report.
- moby_click 5y agoI don't think readers with any kind of computational background expect to read FAILED writes. Apparently, neither does RDX Works - claiming to have fixed most of the issues.
- faraz85 5y agoPleased they ignored that request too, although I can see where RDX are coming from. In a distributed ledger it's all about state. The consensus layer of the architecture is rock solid according to this report.
- aphyr 5y agoThe core ledger system lost committed transactions by choosing not to write them to disk before acknowledgement.
- Mleekko 5y agoYeah, the issue is serious but at the same time: "This problem occurred only in cases where every node was killed at roughly the same time". And there are 100 nodes on the network. (and it is fixed now)
- NelsonMinar 5y agoWell, Radix says it is fixed now. That's the gist of their response to this report, "we fixed a lot of things and no one's tested the new code but trust us!"
- Mleekko 5y agonope, if you read it one more time, for this particular issue Jepsen confirms the fix.
- Too 5y agoIsn't this is a rather common configuration for a distributed DB? Given one node dies before flush, you trust that the remaining nodes in the system will not die at the same time and live long enough to flush to their respective disks. It's a gamble yes, but depending on your environment the risk can be smaller than the benefits. For a blockchain ledger, you might want to choose the safer corner of CAP in that equation.
- belter 5y agoJepsen is the Carl Jung of Distributed Systems...
- vessenes 5y agoI like the deadpan ‘radix claims 1.4mm tps; in testing, transactions at more than 1/s can cause slowdowns’ - impressive lack of eye rolling.
- faraz85 5y agoSlightly out of context, the 1.4mm tps test was during an earlier iteration of the sharded architecture (coming in 2023) whereas Jepsen were testing the unsharded mainnet.
- boardwaalk 5y agoIt may imply they’d need 1.4 million (mm is millimeters…) shards to meet that test number, and scale linearly while doing it, though.
- coolsunglasses 5y agomm is mille mille, meaning a thousand thousand. 1,000 * 1,000 is 1 million. 1mm to mean 1 million is common outside of the United States.
- gmalette 5y agoI live outside the United States and have never seen mm mean anything other than millimeter
- Smaug123 5y agoMeanwhile I live in the UK and it's completely standard for monetary quantities.
- boardwaalk 5y agoI’ve seen that use for sure. Perhaps a little telling we’re talking about blockchain tech and that’s what people are using. Is it really about a fast distributed transaction log, or… is it about the money first.
- twsted 5y agoThis is really hilarious: > When asked, RDX Works executives informed Jepsen that blockchain/DLT readers would normally understand present-tense English statements like these to be referring to potential future behavior, rather than the present. > Jepsen is no stranger to ambitious claims, and aims to understand, analyze, and report on systems as they presently behave—in the context of their documentation, marketing, and community understanding. Jepsen encourages vendors and community members alike to take care with this sort of linguistic ambiguity.
- hnuser123456 5y agoThe future is now. (sorry for relatively low effort comment.)
- cormacrelf 5y agoI wonder if Finnish startups benefit from a language that does not have a future tense. (This is not a Finnish startup.)
- faraz85 5y agoI like to think most people are able to read marketing material in the context of the roadmap to understand the difference in performance claimed vs. measured - apples and oranges comes to mind
- camgunz 5y agoHey did you uh, make an account for this thread?
- lostdog 5y agoDo you have any affiliation with the Radix project that you'd like to disclose?
- rad_gruchalski 5y agoMost likely: https://radixtalk.com/u/faraz/summary https://radixtalk.com/u/faraz/summary.
- ceroxylon 5y agoClassic blockchain startup in the 2020s: put a cool name on your version of sharding, act as if it were already live and being used by governments / corporations, and hope enough people put rockets in the discord to keep the VC money coming in.
- AtlasBarfed 5y agoOH YEAH JEPSEN IS TEARING APART CRYPTO!!!! Aphyr is the best. No matter your fave distributed tech and all it's CAP-don't-matter stuff, he shows that... CAP does very much matter and it's really hard. Cassandra? Kafka? MongoDB (bwahahahah)? It's all got edge cases. He should be getting paid a million bucks a year by various auditing/accounting firms and the FTC/SEC for validating crypto claims. It would be a massive public service. I like that its being treated like a database, and that the safety/correctness of any blockchain has to be viewed from a distributed database standpoint.
- bogomipz 5y agoJepsen is most certainly not "tearing apart crypto" nor is that even remotely the intention of this post. From the very first paragraph: >"This work was funded by Radix Tokens (Jersey) Limited, and conducted in collaboration with RDX Works Ltd ..." The post also links to RDX Works Ltd's blog post on this collaboration. Also in the first paragraph.
- wmf 5y agoThat's the best part: they paid him to throw such savage shade on their blockchain. One transaction per second, one million transactions per second, what's the difference? It's just the roadmap, man.
- raphlinus 5y agoMakes you wonder why they ended the collaboration in November and didn't continue to retain him to validate that the new stuff lives up to its claims, doesn't it?
- Mleekko 5y agoIt's not a secret at all. The response from the Radix CEO: "We re-used the part of his testing harness, and then re-created the other critical tests to determine if the errors detected where still present. He is fantastically expensive and booked 3-6 months in advance on average. We'll definite re-deploy this kind of testing again, but we'll save it for another bigger release, rather than a patch where the identified errors can be tested against."
- redwood 5y agoI have a pretty much opposite perspective on Jepsen than most of the folks here. My feeling is that essentially no distributed system is perfect or without trade offs (and certainly no single node system is perfect and wothout tradeoffs) and Jepsen posts basically make that clear over and over again like some form of techie outrage porn... but the tone and implication is as if somehow there is some alternative panacea without tradeoffs... and I find that a little bit misleading. Basically an astute reader of Jepsen testing may deduce "I need to use a single node system" which is one option but without the availability characteristics modern users usually want
- belter 5y agoThat is not what Jepsen demonstrates. Instead it shows systems making claims that are not supported by their implementation and/or algorithms. If you want to take on your claim/suggestion then those systems just have to stop making the claims he is testing for. PS: There is also another trend, that is System claiming they fixed the issues Jepsen finds, without submitting themselves again for analysis...but I digress now...
- redwood 5y agoI respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case. Here's an analogy that may help me communicate how I feel since I realize my message is not landing: Let's say I'm buying a condo in the San Francisco Bay area. And let's say the building that I'm looking to buy in advertisers that it is historic but seismically retrofitted. Then say Jepsen-earthquake-test comes in and shakes the ground beneath the building and shows that the building indeed collapses with enough of an earthquake: would that or would that not be enough information for me to decide whether or not the seismically retrofitted building is good enough for my needs? There's a lot of ambiguity in answer that question.
- belter 5y agoAlthough I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to. When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom
- raphlinus 5y agoIt's funny how the comments here are polarized, some of them claiming that Jepsen slaughtered RDX, others that it proved that the consensus layer is rock solid. Let's appreciate this for what it is. Blockchains are, at their heart, a type of database (or at the very least a ledger which can be the foundation on which some subset of database semantics can be layered). Performance and reliability are empirical claims which can be tested empirically, using the kind of methodology that Jepsen has been innovating for many years. It is very much to the credit of RDX Works that they subjected their product to this type of testing. I'm not saying anything about the way the use the test results in their blog post and marketing materials, though. What I'd like to see going forward is that it's routine for blockchain-based databases to be tested the same way as real databases, based on actual shipping product rather than speculative goals. Whether you think this would be validating or devastating reveals quite a bit about your preconceptions, but either way would be a win for truth and progress.
- deleted 5y ago[deleted]
- serverholic 5y agoI would love for a highly reputable team or individual to test blockchains and their claims. However, the testing should be done in a way that respects the fact that blockchains are a different type of database and should be judged as such. For example, because of the blockchain trilemma, increasing transaction speed isn't always a good thing since it could sacrifice decentralization.
- xwolfi 5y agoRight but there's a noticeable difference between claiming millions of transactions per second on your blockchain home page and having around 5 in an audit. Sure transaction speed isn't always a good thing, and 5 tps might be very good, but then it should be marketed as such and with a slower speed than competitors, rather than say it broke a world record, maybe ?
- 5y ago
- elesbao 5y agoAphyr and jepsen are a blessing both to breakdown software like these as to illustrate complex distribute system concepts with real world evidence. The dist sys class provided by Aphyr is amazing (https://github.com/aphyr/distsys-class https://github.com/aphyr/distsys-class)