4 ms·
> I was also surprised that Jepsen didn’t find critical bugs. From the report..."...we can prove the presence of bugs, but not their absence..."
by belter 3y ago
> I was also surprised that Jepsen didn’t find critical bugs.
From the report..."...we can prove the presence of bugs, but not their absence..."
- vasco 3y agoThat's consistent with the usual definition of "finding" anything.
- cdchn 3y ago"Absence of evidence is not evidence of absence."
- andreareina 3y agoIf you've looked, it is. The more and the better you look, the better evidence it is.
- kelseyfrog 3y agoIf you run it through bayes theorem, it adjusts the posterior very little.
- cwillu 3y agoIf a test almost always finds something, then the failure of that test to find something is strong evidence.
- kelseyfrog 3y agoI'd be happy to see you numbers for estimated likelihood, prior, and marginal probability if you have them. I'm curious what you get.
- kelseyfrog 3y agoThank you. I've updated my initial guess of p(critical bugs | did not find critical bugs) from 0.5 to 0.82 given my estimate of likelihood and base rates.
- nine_k 3y agos/evidence/proof/. Evidence of absence ("we searched really carefully and nothing came up") does update the Bayesian priors significantly, so the probability of absence of bugs can now be estimated as much higher.
- jupp0r 3y agoIn practical terms, if you are a database and Jepsen doesn't find any bugs, that's as much assurance as you are going to get in 2024 short of formal verification.
- pests 3y agoThe work antithesis has been doing here has me really excited as well.
- stuarthalloway 3y agoFormal verification is very powerful but still not full assurance. Fun fact: Testing and monitoring of Datomic has sometimes uncovered design flaws in underlying storages that formal verification missed.
- nine_k 3y agoWhat kind of flaws? I would expect performance problems.
- jupp0r 3y agoTo start with, you usually perform verification on a behavioral model and not on the code itself. This opens up the possibility that there are behavioral differences between the code itself and the model which wouldn't be caught.
- 3y ago