4 ms·

I am tech lead for a project that revolves around multiple terabytes of trading data for one of top ten largest banks in the world. My team has three, 3-node, 3

I am tech lead for a project that revolves around multiple terabytes of trading data for one of top ten largest banks in the world. My team has three, 3-node, 3TB per node MongoDB clusters where we keep huge amount of documents (mostly immutable 1kB to 10kB in size). Majority write/read concern is exactly so that you don't loose data and don't observe stuff that is going to be rolled back. It is important to understand this fact when you evaluate MongoDB for your solution. That it comes with additional downsides is hardly a surprise, otherwise there would be no reason to specify anything else than majority. You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. It is also obvious, when you use majority concern, that some of the nodes may accept the write but then have to roll back when the majority cannot acknowledge the write. It is obvious this may cause some of the writes to fail that would succeed should the write concern be configured to not require majority acknowledgment. The article simply misses the mark by trying to create sensation where there is none to be found. The MongoDB documentation explains the architecture and guarantees provided by MongoDB enough so that you should be able to understand various read/write concerns and that anything below majority does not guarantee much. This is a tradeoff which you are allowed to make provided you understand the consequences.

aphyr 6y ago
You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. Gently, may I suggest that you read the report, or at least the abstract? This is addressed in the second sentence. :-)
- lmilcin 6y ago
  To quote from the report: "Moreover, the snapshot read concern did not guarantee snapshot unless paired with write concern majority—even for read-only transactions." Of course, it doesn't work when you don't pair it with majority read/write concern. You can't expect to get a snapshot of data that wasn't yet acknowledged by majority of the cluster. As to the quote you probably are referring to: "Jepsen evaluated MongoDB version 4.2.6, and found that even at the strongest levels of read and write concern, it failed to preserve snapshot isolation." I did not find any proof of this in the rest of the report. It seems this is mostly complaint of what happens when you mix different read and write concerns. I would also suggest to think a little bit on the concept of snapshot in the context of distributed system. It is not possible to have the same kind of snapshot that you would get with a single-node application with the architecture of MongoDB. MongoDB is a distributed system where you will get different results depending on which node you are asking. The only way you could get close to having a global snapshot is if all nodes agreed on a single truth (for example single log file, block chain, etc.) which would preclude read/write with concern level less than majority.
  - logicchains 6y ago
    Did you see the part about "Operations in a transaction use the transaction-level read concern. That is, any read concern set at the collection and database level is ignored inside the transaction."? "Tansactions without an explicit read concern downgrade any requested read concern at the database or collection level to a default level of local, which offers “no guarantee that the data has been written to a majority of replicas (i.e. may be rolled back).”" The big problem is that, even if somebody correctly sets the read and write concerns to something sensible, the moment they use a transaction these guarantees fly out the window, unless they read the docs carefully enough to realise they have to set the read and write concern for the transaction too. The defaults are very un-intuitive; I can't imagine that the case of somebody needing snapshot isolation in general but being fine with arbitrary data less in transactions is a common case, compared to wanting to avoid data loss both generally and in transactions.
    - lmilcin 6y ago
      It is different to complain about unclear documentation and unintuitive gurantees and to say that it just doesn't work. Yes it works. Yes, you have to read the documentation very carefully.
      - inglor 6y ago
        Not saying you're wrong. As an anecdotal data point - we've read the docs (carefully) and spoke to MongoDB quite a bit when implementing transactions including their highest paid levels of support and still ran into this issue: > transactions running with the strongest isolation levels can exhibit G1c: cyclic information flow. As well as the Node.js API issue (I just checked randomly and their Python API has the same bug lol) listed above.
      - afarrell 6y ago
        It is not different. For a product like mongodb, both the durability guarantees and the documentation explaining them are an integral part of the user experience. If I'm starting a project, I'm making decisions for a junior developer whom I'll hire in 2 years. I care what code that junior developer will be most nudged to write. If the Stripe API had documentation was needlessly unclear in a way which led people to lose a significant amount of money, that would be a bug.
bronson 6y ago
If a database advertises attributes that aren't a part of its default setup, you can expect its docs to make it very simple and clear how to get them. If not, that's misrepresentation.
- lmilcin 6y ago
  The documentation states that very clearly and the attributes are part of every call to the database (as long as you are using native driver). In any case any person that has some experience with distributed systems will understand what it roughly means to get an acknowledgment from just a single node vs. waiting for the majority. Oracle also does not use serializable as its default isolation level, yet it advertises it. This is all part of the product functionality. Whenever you evaluate product for your project you have to understand various options, functionalities and their tradeoffs. Defaults don't mean shit. In a complex clustered product you need to understand all important knobs to decide the correct settings and configurable guarantees are most important knobs there are.
  - bronson 6y ago
    Good point. If there’s a database that rivals Mongo for shady sales tactics, it’s Oracle.
lllr_finger 6y ago
> The article simply misses the mark by trying to create sensation where there is none to be found. As someone who is a tech lead for a large database install, I'd urge you to read the rest of the Jepsen reports. They aren't intended to be hit pieces on technology - they're deep dives into the claims and guarantees of each database. IIRC MDB has explicitly reached out to OP in the past (I doubt they'll continue to do so after this). Why that matters to the rest of us: once I learn all those dials and knobs I'm left wondering why I would choose Mongo over another technology, and how much the design of the default behavior and complexity of said dials/knobs are influenced by their core business.
- lmilcin 6y ago
  I agree. MongoDB has large numbers of peculiarities that you better know before you buy in. It is definitely not so rosy as advertised. In particular it seems the product is not mature (especially if you come from Oracle world) and the features seem slapped on as they go and not thought through.
- afarrell 6y ago
  I would also wonder about the surrounding ecosystem of tooling & libraries. Imagine there was a programming language which had rather inconsistent naming, poor automated testing support, and a history of guiding its users toward security vulnerabilities. A culture would grow up around that language and the most successful members would be those who could best tolerate those properties. People generally self-select into language communities. So unless some powerful influence pushed random programmers to use the language or made it easier to add new tooling, the culture would continue to undervalue what the language originally lacked. I suspect the same social dynamic would apply to a database.
nosequel 6y ago
Since you just leaned all the way in, while repeatedly proving you either will not, or cannot read the posted article at all. Will you let us know what bank you support so at least I can make sure I never use that bank? Thanks, Those of us who care about our banking and investing data.

aphyr 6y ago

You just can't test lower levels of guarantees and then complain you did not get what higher levels of guarantees were designed to provide. Gently, may I suggest that you read the report, or at least the abstract? This is addressed in the second sentence. :-)

lmilcin 6y ago
To quote from the report: "Moreover, the snapshot read concern did not guarantee snapshot unless paired with write concern majority—even for read-only transactions." Of course, it doesn't work when you don't pair it with majority read/write concern. You can't expect to get a snapshot of data that wasn't yet acknowledged by majority of the cluster. As to the quote you probably are referring to: "Jepsen evaluated MongoDB version 4.2.6, and found that even at the strongest levels of read and write concern, it failed to preserve snapshot isolation." I did not find any proof of this in the rest of the report. It seems this is mostly complaint of what happens when you mix different read and write concerns. I would also suggest to think a little bit on the concept of snapshot in the context of distributed system. It is not possible to have the same kind of snapshot that you would get with a single-node application with the architecture of MongoDB. MongoDB is a distributed system where you will get different results depending on which node you are asking. The only way you could get close to having a global snapshot is if all nodes agreed on a single truth (for example single log file, block chain, etc.) which would preclude read/write with concern level less than majority.
- logicchains 6y ago
  Did you see the part about "Operations in a transaction use the transaction-level read concern. That is, any read concern set at the collection and database level is ignored inside the transaction."? "Tansactions without an explicit read concern downgrade any requested read concern at the database or collection level to a default level of local, which offers “no guarantee that the data has been written to a majority of replicas (i.e. may be rolled back).”" The big problem is that, even if somebody correctly sets the read and write concerns to something sensible, the moment they use a transaction these guarantees fly out the window, unless they read the docs carefully enough to realise they have to set the read and write concern for the transaction too. The defaults are very un-intuitive; I can't imagine that the case of somebody needing snapshot isolation in general but being fine with arbitrary data less in transactions is a common case, compared to wanting to avoid data loss both generally and in transactions.
  - lmilcin 6y ago
    It is different to complain about unclear documentation and unintuitive gurantees and to say that it just doesn't work. Yes it works. Yes, you have to read the documentation very carefully.
    - inglor 6y ago
      Not saying you're wrong. As an anecdotal data point - we've read the docs (carefully) and spoke to MongoDB quite a bit when implementing transactions including their highest paid levels of support and still ran into this issue: > transactions running with the strongest isolation levels can exhibit G1c: cyclic information flow. As well as the Node.js API issue (I just checked randomly and their Python API has the same bug lol) listed above.
    - afarrell 6y ago
      It is not different. For a product like mongodb, both the durability guarantees and the documentation explaining them are an integral part of the user experience. If I'm starting a project, I'm making decisions for a junior developer whom I'll hire in 2 years. I care what code that junior developer will be most nudged to write. If the Stripe API had documentation was needlessly unclear in a way which led people to lose a significant amount of money, that would be a bug.

bronson 6y ago

If a database advertises attributes that aren't a part of its default setup, you can expect its docs to make it very simple and clear how to get them. If not, that's misrepresentation.

lmilcin 6y ago
The documentation states that very clearly and the attributes are part of every call to the database (as long as you are using native driver). In any case any person that has some experience with distributed systems will understand what it roughly means to get an acknowledgment from just a single node vs. waiting for the majority. Oracle also does not use serializable as its default isolation level, yet it advertises it. This is all part of the product functionality. Whenever you evaluate product for your project you have to understand various options, functionalities and their tradeoffs. Defaults don't mean shit. In a complex clustered product you need to understand all important knobs to decide the correct settings and configurable guarantees are most important knobs there are.
- bronson 6y ago
  Good point. If there’s a database that rivals Mongo for shady sales tactics, it’s Oracle.

lllr_finger 6y ago

> The article simply misses the mark by trying to create sensation where there is none to be found. As someone who is a tech lead for a large database install, I'd urge you to read the rest of the Jepsen reports. They aren't intended to be hit pieces on technology - they're deep dives into the claims and guarantees of each database. IIRC MDB has explicitly reached out to OP in the past (I doubt they'll continue to do so after this). Why that matters to the rest of us: once I learn all those dials and knobs I'm left wondering why I would choose Mongo over another technology, and how much the design of the default behavior and complexity of said dials/knobs are influenced by their core business.

lmilcin 6y ago
I agree. MongoDB has large numbers of peculiarities that you better know before you buy in. It is definitely not so rosy as advertised. In particular it seems the product is not mature (especially if you come from Oracle world) and the features seem slapped on as they go and not thought through.
afarrell 6y ago
I would also wonder about the surrounding ecosystem of tooling & libraries. Imagine there was a programming language which had rather inconsistent naming, poor automated testing support, and a history of guiding its users toward security vulnerabilities. A culture would grow up around that language and the most successful members would be those who could best tolerate those properties. People generally self-select into language communities. So unless some powerful influence pushed random programmers to use the language or made it easier to add new tooling, the culture would continue to undervalue what the language originally lacked. I suspect the same social dynamic would apply to a database.

nosequel 6y ago

Since you just leaned all the way in, while repeatedly proving you either will not, or cannot read the posted article at all. Will you let us know what bank you support so at least I can make sure I never use that bank? Thanks, Those of us who care about our banking and investing data.