4 ms·

Have you considered presenting the data in a concise manner in addition to the in-depth analyses? That is, a table on the jepsen.io frontpage, or at least on e

by devit 6y ago

Have you considered presenting the data in a concise manner in addition to the in-depth analyses? That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. Finally, there should be tests for the non-broken databases (PostgreSQL for instance, both in single-server mode, deployed with Stolon on Kubernetes and using the multimaster projects) as well to confirm they actually work.

eloff 6y ago
Oh man this would be useful.
aphyr 6y ago
That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). This is a wonderful idea, and I've got no idea how to actually do it in a standardized, rigorous way. Vendor claims are often contradictory, it's hard to get a good idea of anomaly frequency, availability is... a rabbithole, and it's hard to come up with a standard taxonomy of anomalies--most of the analyses I do wind up finding something I've never really seen before, haha. With that in mind, I've wound up letting the reports speak for themselves. Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. I don't know a good way to do this either. Each report is typically the product of months of experimental work; it's not like Jepsen is a pass-fail test suite that gives immediately accurate results. There is, unfortunately, a lot of subtle interpretive work that goes into figuring out if a test is doing something meaningful, and a lot of that work needs to be repeated on each test run. Think, like... staring at the logs and noticing that a certain class of exception is being caught more often than you might have expected, and realizing that a certain type of transaction now triggers a new conflict detection mechanism which causes higher probabilities of aborts; those aborts reduce the frequency with which you can observe database state, allowing a race condition to go un-noticed. That kinda thing. If I'm lucky and the API/setup process haven't changed, I can re-run an analysis in about a week or so. If I'm unlucky, there's been drift in the OS, setup process, APIs, client libraries, error handling, etc. It's not uncommon for a repeat analysis to take months. :-(
- X6S1x6Okd1st 6y ago
  It's probably more snarky than helpful, but it'd be great to have a section where it's just marketing materials or docs that you've corrected with a red pen
  - bcrosby95 6y ago
    It's probably better to keep it professional. Your average employee can afford some snark. But when companies hire you for this sort of consulting, you could turn off a lot of potential clients by including it in materials you produce, even when they didn't pay for it. Because it is a representation of the product they would be paying for. It would be kinda like you including this sort of thing on your resume. Which would also be a bad idea.
    - ashtonkem 6y ago
      For those who don’t know, Kyle makes a living offering these types of analysis to database companies directly. While a lot of us love to dunk on Mongo (myself included), it would be silly to expect Kyle to risk his livelihood.
  - jka 6y ago
    If done accurately and professionally, something like you're suggesting could be really useful to aid people and organizations during vendor selection. https://web.hypothes.is/about/ https://web.hypothes.is/about/ or similar could be used to develop commentary overlays on top of marketing materials.
HappyDreamer 6y ago
> consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell Plus maybe a column indicating what [the company behind the database] claims?

by devit 6y ago

eloff 6y ago

Oh man this would be useful.

aphyr 6y ago

That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). This is a wonderful idea, and I've got no idea how to actually do it in a standardized, rigorous way. Vendor claims are often contradictory, it's hard to get a good idea of anomaly frequency, availability is... a rabbithole, and it's hard to come up with a standard taxonomy of anomalies--most of the analyses I do wind up finding something I've never really seen before, haha. With that in mind, I've wound up letting the reports speak for themselves. Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. I don't know a good way to do this either. Each report is typically the product of months of experimental work; it's not like Jepsen is a pass-fail test suite that gives immediately accurate results. There is, unfortunately, a lot of subtle interpretive work that goes into figuring out if a test is doing something meaningful, and a lot of that work needs to be repeated on each test run. Think, like... staring at the logs and noticing that a certain class of exception is being caught more often than you might have expected, and realizing that a certain type of transaction now triggers a new conflict detection mechanism which causes higher probabilities of aborts; those aborts reduce the frequency with which you can observe database state, allowing a race condition to go un-noticed. That kinda thing. If I'm lucky and the API/setup process haven't changed, I can re-run an analysis in about a week or so. If I'm unlucky, there's been drift in the OS, setup process, APIs, client libraries, error handling, etc. It's not uncommon for a repeat analysis to take months. :-(

X6S1x6Okd1st 6y ago
It's probably more snarky than helpful, but it'd be great to have a section where it's just marketing materials or docs that you've corrected with a red pen
- bcrosby95 6y ago
  It's probably better to keep it professional. Your average employee can afford some snark. But when companies hire you for this sort of consulting, you could turn off a lot of potential clients by including it in materials you produce, even when they didn't pay for it. Because it is a representation of the product they would be paying for. It would be kinda like you including this sort of thing on your resume. Which would also be a bad idea.
  - ashtonkem 6y ago
    For those who don’t know, Kyle makes a living offering these types of analysis to database companies directly. While a lot of us love to dunk on Mongo (myself included), it would be silly to expect Kyle to risk his livelihood.
- jka 6y ago
  If done accurately and professionally, something like you're suggesting could be really useful to aid people and organizations during vendor selection. https://web.hypothes.is/about/ https://web.hypothes.is/about/ or similar could be used to develop commentary overlays on top of marketing materials.

HappyDreamer 6y ago

> consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell Plus maybe a column indicating what [the company behind the database] claims?