8 ms·

Hi folks! Author of the report here. If anyone has questions about detecting transactional anomalies, what those anomalies are in the first place, snapshot isol

by aphyr 6y ago

Hi folks! Author of the report here. If anyone has questions about detecting transactional anomalies, what those anomalies are in the first place, snapshot isolation, etc., I'm happy to answer as best I can.

eternalban 6y ago
"3.4 Duplicate Effects" This section seems to be the most worrying results in your report, Kyle, with no work around. Did I read that correctly?
- aphyr 6y ago
  Yeah, there's no workaround that I can find for 3.4 (duplicate effects), 3.5 (read skew), 3.6 (cyclic information flow), or 3.7 (read own future writes). I've arranged those in "increasingly worrying order"--duplicating writes doesn't feel as bad as allowing transactions to mutually observe each other's effects, for example. The fact that you can't even rely on a single transactions' operations taking place (or, more precisely, appearing to take place) in the order they're written is especially worrying. All of these behaviors occurred with read and write concerns set to snapshot/majority. That's not to say that workarounds don't exist, just that I didn't find any in the documentation or by twiddling config flags in the ~2 weeks I was working on this report. :)
danpalmer 6y ago
Not a question necessarily about the technical side, but I'm interested in your opinion as to the root cause – is it desire to achieve certain results for marketing purposes, lack of understanding/training in the team about distributed systems, just bugs and a lack of testing...? Alternatively does most of this come down to one specific technical choice, and why might they have made that choice? Very happy for (informed) speculation here, I recognise we'll probably never know for certain, but I'm interested to avoid making similar mistakes myself.
- aphyr 6y ago
  There's a few things at play here. One is talking only about the positive results from the previous Jepsen analysis, while not discussing the negative ones. Vendors often try to represent findings in the most positive light, but this was a particularly extreme case. Not discussing default behavior is a significant oversight, and it's especially important given ~80% of people run with default write concern, and 99% run with default read concern. The middle part of the report talks about unexpected but (almost all) documented behavior around read and write concern for transactions. I don't want to conjecture too much about motivations here, but based on my professional experience with a few dozen databases, and surveys of colleagues, I termed it "surprising". The fact that there's explicit documentation for what I'd consider Counterintuitive API Design suggests that this is something MongoDB engineers considered, and possibly debated, internally. The final part of the report talks about what I'm pretty sure are bugs. I'm strongly suspicious of the retry mechanism: it's possible that an idempotency token doesn't exist, isn't properly used, or that MongoDB's client or server layers are improperly interpreting an indeterminate failure as a determinate one. It seems possible that all 4 phenomena we observed stem from the retry mechanism, but as discussed in the report, it's not entirely clear that's the case.
  - danpalmer 6y ago
    Thanks for the thoughts. I get the impression that MongoDB may have hyped themselves into a corner in the early days with poorly made (or misleading) benchmarks. Perhaps they have customers with a lot of influence determining how they think about performance vs consistency. Maybe this combined with patching, re-patching, re-patching again their replication logic/consistency algorithm means that they'll be stuck in this sort of position for a long time.
    - aphyr 6y ago
      Possibly! You're right that path dependence played a role in safety issues: the problems we found in 3.4.0-rc3 were related to grafting the new v1 replication protocol onto a system which made assumptions about how v0 behaved. That said, I don't want to discount that MongoDB has made significant improvements over the years. Single-document linearizability was a long time in the works, and that's nothing to sneeze at! http://jepsen.io/analyses/mongodb-3-4-0-rc3 http://jepsen.io/analyses/mongodb-3-4-0-rc3
teskk123 6y ago
hello :) Where is you found out all information about how to do such testing?
- aphyr 6y ago
  I was lucky to have a good education: my B.A. involved courses in contemporary experimental physics and independent research in nonlinear quantum dynamics (esp. proofs, experimental design, writing), cognitive and social psychology (more experiment design and stats), math structures (proof techniques), philosophy (metaphysics, philosophy of science), and English (rhetoric). All of those helped give me a foundation for doing this kind of experimental work and communicating it to others. Jepsen draws inspiration from a long line of work on property-based testing, especially Quickcheck & co. It also draws on roughly 10 years of experience building & running distributed systems in production. A lot of Jepsen I invented from whole cloth, but some of the checkers in Jepsen are derived from specific research papers, like work by Wing, Gong, and Howe on linearizability checking. Then it's just... a lot of thinking, experimenting, and writing. Jepsen's the product of ~6 years of full-time work. Elle, the system which detected the anomalies in this report, was a research project I've been puzzling over for roughly two years. I write the Jepsen series, and open-source all of the code for these tests, partly as a resource so that other people can learn to do this same kind of work. :-)
  - teskk123 6y ago
    wow. thanks a lot for quick and full answering!
  - throwaway_pdp09 6y ago
    I guess you've answered my question, but to be clear, you do not instrument/analyse the code, you treat it as a black box which you hammer on externally, is that right?
    - aphyr 6y ago
      Pretty much, yeah. There are some cases where Jepsen reaches into the guts of a database or lies to it via LD_PRELOAD shims, but generally these are Just Plain Old Binaries provided by vendors; no instrumentation required.
monstrado 6y ago
Huge fan of your work! I was curious if you've ever attempted to run your (or part of) Mongo test suite against FoundationDB using their DocumentLayer since it's supposed to be Mongo API compatible.
- aphyr 6y ago
  No, I haven't! You can see a full list of analyses here: http://jepsen.io/analyses http://jepsen.io/analyses
- robterrell 6y ago
  IIRC one of the FoundationDB engineers tested with Jepsen and found that it passed in its default configuration, but the blog post seems to have disappeared. https://web.archive.org/web/20150312112556/http://blog.foundationdb.com/foundationdb-vs-the-new-jepsen-and-why-you-should-care https://web.archive.org/web/20150312112556/http://blog.found...
  - monstrado 6y ago
    Thanks for firing up the time machine! I've been using FDB for a little over a year now and can't recommend it enough. Such a solid piece of meticulous engineering.
devit 6y ago
Have you considered presenting the data in a concise manner in addition to the in-depth analyses? That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. Finally, there should be tests for the non-broken databases (PostgreSQL for instance, both in single-server mode, deployed with Stolon on Kubernetes and using the multimaster projects) as well to confirm they actually work.
- eloff 6y ago
  Oh man this would be useful.
- aphyr 6y ago
  That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). This is a wonderful idea, and I've got no idea how to actually do it in a standardized, rigorous way. Vendor claims are often contradictory, it's hard to get a good idea of anomaly frequency, availability is... a rabbithole, and it's hard to come up with a standard taxonomy of anomalies--most of the analyses I do wind up finding something I've never really seen before, haha. With that in mind, I've wound up letting the reports speak for themselves. Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. I don't know a good way to do this either. Each report is typically the product of months of experimental work; it's not like Jepsen is a pass-fail test suite that gives immediately accurate results. There is, unfortunately, a lot of subtle interpretive work that goes into figuring out if a test is doing something meaningful, and a lot of that work needs to be repeated on each test run. Think, like... staring at the logs and noticing that a certain class of exception is being caught more often than you might have expected, and realizing that a certain type of transaction now triggers a new conflict detection mechanism which causes higher probabilities of aborts; those aborts reduce the frequency with which you can observe database state, allowing a race condition to go un-noticed. That kinda thing. If I'm lucky and the API/setup process haven't changed, I can re-run an analysis in about a week or so. If I'm unlucky, there's been drift in the OS, setup process, APIs, client libraries, error handling, etc. It's not uncommon for a repeat analysis to take months. :-(
politician 6y ago
Thank you for all of your work over the years. Your reports have helped me and others stand up to bizdev hype and make better decisions for our companies and customers. Postgres is widely understood to be a robust database with safe defaults. I, and perhaps others, would love to see you aim your array of weapons at Postgres. Do you have any plans to look at stock Postgres?
- aphyr 6y ago
  It's been on my list for a long time, but I've also struggled to find out like... what, exactly, is the right way to do postgres replication? Every time I go into the docs I wind up with a laundry list of different mechanisms for replication and failover, and no idea which one would be most appropriate for a test. I gotta get on this!
  - bsaul 6y ago
    i feel like this is the reaction of everyone having ever tried to setup postgres replication. With your audience, you deciding for a particular setup will probably help a LOT of people, and ultimately the postgres project as well.
    - takeda 6y ago
      If you worry about data, you should not use automatic failover. It's nearly impossible for standby to know why master stopped responding. Maybe there was a hardware failure, or maybe master is just busy. This is why manual failover is better, because you can know the real reason and decide whether you should perform failover or just wait. With tools like repmgr it is just a single command invoked on the standby. If you absolutely don't want to lose any data, you should have two masters in close proximity (so the latency isn't high) set up with synchronous replication, then have one or two standbys with asynchronous replication. This will reduce throughout, but then you can be sure that the other machine has all the same transactions. If something happens to both you then can fallback to the asynchronous one which might be a bit behind.
dilandau 6y ago
You're doing very, very valuable work. Thanks fam, keep those vendors honest, and help us make informed decisions.
staticassertion 6y ago
I've wanted to try building a toy database to learn more about how they work - any suggestions for good resources?
- aphyr 6y ago
  I've actually written a little mini-Jepsen workbench for folks who want to practice implementing a distributed key-value store. Might be worth a spin! https://github.com/jepsen-io/maelstrom https://github.com/jepsen-io/maelstrom
rclayton 6y ago
Hi Kyle! I’ve really enjoyed your work over the years. I was wondering, with all of your testing and experimentation, is there any system that had really impressed you?
- zbentley 6y ago
  I don't presume to speak for him, but his writeup on ZooKeeper was among the most positive in the Jepsen series: https://aphyr.com/posts/291-jepsen-zookeeper https://aphyr.com/posts/291-jepsen-zookeeper My bias: I like and heavily use ZooKeeper in production. HN seems not to like it as much.
- aphyr 6y ago
  I'm kind of impressed any distributed system gets off the ground. These things are hard to write!
rystsov 6y ago
Hi Kyle, thanks for the Elle :) I want to use Elle to check long histories of transactions over small set of keys with read dominant workload, the paper recommends to use lists over registers but when the history becomes long on the one hand it becomes too wasteful to read the register's history on each request on the other hand the Elle's input becomes very large. E.g. when each read should return the whole register's history the size of history grows O(n^2) compared to the case when the reads return just the head. So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists?
- aphyr 6y ago
  E.g. when each read should return the whole register's history the size of history grows O(n^2) compared to the case when the reads return just the head. If you look at Elle's transaction generators, you can cap the size of any individual key, and use an uneven (e.g. exponential) distribution of key choices to get various frequencies. That way keys stay reasonably small (I use 1-10K writes/key), some keys are updated frequently to catch race conditions, and others last hundreds of seconds to catch long-lasting errors. So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists? RW registers are significantly weaker, though I don't know how to quantify the difference. I've still caught errors with registers, but the grounds for inferring anomalies are a.) less powerful and b.) can only be applied in certain circumstances--we talk about some of these details in the paper.

by aphyr 6y ago

eternalban 6y ago

"3.4 Duplicate Effects" This section seems to be the most worrying results in your report, Kyle, with no work around. Did I read that correctly?

aphyr 6y ago
Yeah, there's no workaround that I can find for 3.4 (duplicate effects), 3.5 (read skew), 3.6 (cyclic information flow), or 3.7 (read own future writes). I've arranged those in "increasingly worrying order"--duplicating writes doesn't feel as bad as allowing transactions to mutually observe each other's effects, for example. The fact that you can't even rely on a single transactions' operations taking place (or, more precisely, appearing to take place) in the order they're written is especially worrying. All of these behaviors occurred with read and write concerns set to snapshot/majority. That's not to say that workarounds don't exist, just that I didn't find any in the documentation or by twiddling config flags in the ~2 weeks I was working on this report. :)

danpalmer 6y ago

Not a question necessarily about the technical side, but I'm interested in your opinion as to the root cause – is it desire to achieve certain results for marketing purposes, lack of understanding/training in the team about distributed systems, just bugs and a lack of testing...? Alternatively does most of this come down to one specific technical choice, and why might they have made that choice? Very happy for (informed) speculation here, I recognise we'll probably never know for certain, but I'm interested to avoid making similar mistakes myself.

aphyr 6y ago
There's a few things at play here. One is talking only about the positive results from the previous Jepsen analysis, while not discussing the negative ones. Vendors often try to represent findings in the most positive light, but this was a particularly extreme case. Not discussing default behavior is a significant oversight, and it's especially important given ~80% of people run with default write concern, and 99% run with default read concern. The middle part of the report talks about unexpected but (almost all) documented behavior around read and write concern for transactions. I don't want to conjecture too much about motivations here, but based on my professional experience with a few dozen databases, and surveys of colleagues, I termed it "surprising". The fact that there's explicit documentation for what I'd consider Counterintuitive API Design suggests that this is something MongoDB engineers considered, and possibly debated, internally. The final part of the report talks about what I'm pretty sure are bugs. I'm strongly suspicious of the retry mechanism: it's possible that an idempotency token doesn't exist, isn't properly used, or that MongoDB's client or server layers are improperly interpreting an indeterminate failure as a determinate one. It seems possible that all 4 phenomena we observed stem from the retry mechanism, but as discussed in the report, it's not entirely clear that's the case.
- danpalmer 6y ago
  Thanks for the thoughts. I get the impression that MongoDB may have hyped themselves into a corner in the early days with poorly made (or misleading) benchmarks. Perhaps they have customers with a lot of influence determining how they think about performance vs consistency. Maybe this combined with patching, re-patching, re-patching again their replication logic/consistency algorithm means that they'll be stuck in this sort of position for a long time.
  - aphyr 6y ago
    Possibly! You're right that path dependence played a role in safety issues: the problems we found in 3.4.0-rc3 were related to grafting the new v1 replication protocol onto a system which made assumptions about how v0 behaved. That said, I don't want to discount that MongoDB has made significant improvements over the years. Single-document linearizability was a long time in the works, and that's nothing to sneeze at! http://jepsen.io/analyses/mongodb-3-4-0-rc3 http://jepsen.io/analyses/mongodb-3-4-0-rc3

teskk123 6y ago

hello :) Where is you found out all information about how to do such testing?

aphyr 6y ago
I was lucky to have a good education: my B.A. involved courses in contemporary experimental physics and independent research in nonlinear quantum dynamics (esp. proofs, experimental design, writing), cognitive and social psychology (more experiment design and stats), math structures (proof techniques), philosophy (metaphysics, philosophy of science), and English (rhetoric). All of those helped give me a foundation for doing this kind of experimental work and communicating it to others. Jepsen draws inspiration from a long line of work on property-based testing, especially Quickcheck & co. It also draws on roughly 10 years of experience building & running distributed systems in production. A lot of Jepsen I invented from whole cloth, but some of the checkers in Jepsen are derived from specific research papers, like work by Wing, Gong, and Howe on linearizability checking. Then it's just... a lot of thinking, experimenting, and writing. Jepsen's the product of ~6 years of full-time work. Elle, the system which detected the anomalies in this report, was a research project I've been puzzling over for roughly two years. I write the Jepsen series, and open-source all of the code for these tests, partly as a resource so that other people can learn to do this same kind of work. :-)
- teskk123 6y ago
  wow. thanks a lot for quick and full answering!
- throwaway_pdp09 6y ago
  I guess you've answered my question, but to be clear, you do not instrument/analyse the code, you treat it as a black box which you hammer on externally, is that right?
  - aphyr 6y ago
    Pretty much, yeah. There are some cases where Jepsen reaches into the guts of a database or lies to it via LD_PRELOAD shims, but generally these are Just Plain Old Binaries provided by vendors; no instrumentation required.

monstrado 6y ago

Huge fan of your work! I was curious if you've ever attempted to run your (or part of) Mongo test suite against FoundationDB using their DocumentLayer since it's supposed to be Mongo API compatible.

aphyr 6y ago
No, I haven't! You can see a full list of analyses here: http://jepsen.io/analyses http://jepsen.io/analyses
robterrell 6y ago
IIRC one of the FoundationDB engineers tested with Jepsen and found that it passed in its default configuration, but the blog post seems to have disappeared. https://web.archive.org/web/20150312112556/http://blog.foundationdb.com/foundationdb-vs-the-new-jepsen-and-why-you-should-care https://web.archive.org/web/20150312112556/http://blog.found...
- monstrado 6y ago
  Thanks for firing up the time machine! I've been using FDB for a little over a year now and can't recommend it enough. Such a solid piece of meticulous engineering.

devit 6y ago

Have you considered presenting the data in a concise manner in addition to the in-depth analyses? That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. Finally, there should be tests for the non-broken databases (PostgreSQL for instance, both in single-server mode, deployed with Stolon on Kubernetes and using the multimaster projects) as well to confirm they actually work.

eloff 6y ago
Oh man this would be useful.
aphyr 6y ago
That is, a table on the jepsen.io frontpage, or at least on each product's review page, with database products and configuration on rows and consistency properties on columns, and a nice "Yay!" or "Nope!" mark in the cell, plus links on how to achieve the database configurations in the table (esp. how to configure each database to have the most guarantees). This is a wonderful idea, and I've got no idea how to actually do it in a standardized, rigorous way. Vendor claims are often contradictory, it's hard to get a good idea of anomaly frequency, availability is... a rabbithole, and it's hard to come up with a standard taxonomy of anomalies--most of the analyses I do wind up finding something I've never really seen before, haha. With that in mind, I've wound up letting the reports speak for themselves. Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale. I don't know a good way to do this either. Each report is typically the product of months of experimental work; it's not like Jepsen is a pass-fail test suite that gives immediately accurate results. There is, unfortunately, a lot of subtle interpretive work that goes into figuring out if a test is doing something meaningful, and a lot of that work needs to be repeated on each test run. Think, like... staring at the logs and noticing that a certain class of exception is being caught more often than you might have expected, and realizing that a certain type of transaction now triggers a new conflict detection mechanism which causes higher probabilities of aborts; those aborts reduce the frequency with which you can observe database state, allowing a race condition to go un-noticed. That kinda thing. If I'm lucky and the API/setup process haven't changed, I can re-run an analysis in about a week or so. If I'm unlucky, there's been drift in the OS, setup process, APIs, client libraries, error handling, etc. It's not uncommon for a repeat analysis to take months. :-(

politician 6y ago

Thank you for all of your work over the years. Your reports have helped me and others stand up to bizdev hype and make better decisions for our companies and customers. Postgres is widely understood to be a robust database with safe defaults. I, and perhaps others, would love to see you aim your array of weapons at Postgres. Do you have any plans to look at stock Postgres?

aphyr 6y ago
It's been on my list for a long time, but I've also struggled to find out like... what, exactly, is the right way to do postgres replication? Every time I go into the docs I wind up with a laundry list of different mechanisms for replication and failover, and no idea which one would be most appropriate for a test. I gotta get on this!
- bsaul 6y ago
  i feel like this is the reaction of everyone having ever tried to setup postgres replication. With your audience, you deciding for a particular setup will probably help a LOT of people, and ultimately the postgres project as well.
  - takeda 6y ago
    If you worry about data, you should not use automatic failover. It's nearly impossible for standby to know why master stopped responding. Maybe there was a hardware failure, or maybe master is just busy. This is why manual failover is better, because you can know the real reason and decide whether you should perform failover or just wait. With tools like repmgr it is just a single command invoked on the standby. If you absolutely don't want to lose any data, you should have two masters in close proximity (so the latency isn't high) set up with synchronous replication, then have one or two standbys with asynchronous replication. This will reduce throughout, but then you can be sure that the other machine has all the same transactions. If something happens to both you then can fallback to the asynchronous one which might be a bit behind.

dilandau 6y ago

You're doing very, very valuable work. Thanks fam, keep those vendors honest, and help us make informed decisions.

staticassertion 6y ago

I've wanted to try building a toy database to learn more about how they work - any suggestions for good resources?

aphyr 6y ago
I've actually written a little mini-Jepsen workbench for folks who want to practice implementing a distributed key-value store. Might be worth a spin! https://github.com/jepsen-io/maelstrom https://github.com/jepsen-io/maelstrom

rclayton 6y ago

Hi Kyle! I’ve really enjoyed your work over the years. I was wondering, with all of your testing and experimentation, is there any system that had really impressed you?

zbentley 6y ago
I don't presume to speak for him, but his writeup on ZooKeeper was among the most positive in the Jepsen series: https://aphyr.com/posts/291-jepsen-zookeeper https://aphyr.com/posts/291-jepsen-zookeeper My bias: I like and heavily use ZooKeeper in production. HN seems not to like it as much.
aphyr 6y ago
I'm kind of impressed any distributed system gets off the ground. These things are hard to write!

rystsov 6y ago

Hi Kyle, thanks for the Elle :) I want to use Elle to check long histories of transactions over small set of keys with read dominant workload, the paper recommends to use lists over registers but when the history becomes long on the one hand it becomes too wasteful to read the register's history on each request on the other hand the Elle's input becomes very large. E.g. when each read should return the whole register's history the size of history grows O(n^2) compared to the case when the reads return just the head. So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists?

aphyr 6y ago
E.g. when each read should return the whole register's history the size of history grows O(n^2) compared to the case when the reads return just the head. If you look at Elle's transaction generators, you can cap the size of any individual key, and use an uneven (e.g. exponential) distribution of key choices to get various frequencies. That way keys stay reasonably small (I use 1-10K writes/key), some keys are updated frequently to catch race conditions, and others last hundreds of seconds to catch long-lasting errors. So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists? RW registers are significantly weaker, though I don't know how to quantify the difference. I've still caught errors with registers, but the grounds for inferring anomalies are a.) less powerful and b.) can only be applied in certain circumstances--we talk about some of these details in the paper.