8 ms·

RavenDB 6.0.2 (A Jepsen Report)

195 points by aphyr 3y ago

CJefferson 3y ago
I love Jepsen, but it seriously worries me how bad software turns out to be, and how many outrageous claims companies make that turn out to be so easily proved false. Should there be more serious penalties when companies make claims which turn out to be false as soon as they are tested? I think there should be.
- skyde 3y ago
  Well some company try to implement Raft algorithm with good intentions but do a bad job a at so bugs make it so they fail Jensen test. You can’t force people to test their software correctly unless it’s a regulated field like aviation.
  - okigan 3y ago
    [flagged]
    - dallasg3 3y ago
      Also is Jepsen, not Jensen.
      - myk9001 3y ago
        Named after Carly Rae Jepsen for her song Call Me Maybe.
    - lambda 3y ago
      Aviation has an extremely strong safety record, which has been getting better year by year. Yes, there are misses, but they have been happening increasingly less frequently over time. https://en.wikipedia.org/wiki/Aviation_safety#/media/File:Fatalities_per_revenue_passenger_kilometre_in_air_transport_since_1970.png https://en.wikipedia.org/wiki/Aviation_safety#/media/File:Fa... Aviation isn't perfect; nothing implemented by large groups of fallible humans with budget constraints will be. But it has one of the best safety track records of any industry. Now, why won't aviation style engineering be applied in other fields, like databases? Well, because no one really cares enough. No one dies if some random database used by some ad platform loses the occasional transaction. Yeah, it's frustrating to engineers who are trying to build reliable systems, but in the grand scheme of things losing a few percent of transactions isn't the end of the world for most businesses. You get safety cultures like that in aviation because there are real, substantial risks, so you need to have thorough engineering discipline, properly designed redundancy, etc. For databases that are used for the majority of the business world, efficiency is generally a bigger concern than correctness; they'd rather have cheap and fast databases that lose a few transactions occasionally than something that actually provides consistency. But of course everyone thinks they need consistency, so it's advertised as a selling point while not actually being provided in practice.
      - _a_a_a_ 3y ago
        > For databases that are used for the majority of the business world, efficiency is generally a bigger concern than correctness; they'd rather have cheap and fast databases that lose a few transactions occasionally than something that actually provides consistency 'majority' is a strong claim. Any figures to back it up?
        lambda 3y ago
        No, no figures to back it up. This is just based on anecdotal experience, having never seen very many businesses that actually prioritize consistency or testing or validating it. The fact that Kyle keeps on finding these massive problems in popular databases is part of that anecdotal evidence. Obviously people want correctness; they would like their databases to not randomly lose data. Hence the fact that it's a highly advertised feature. But when it actually comes down to selecting a database, convenience and performance seem to be what people actually compare on, there are very few places that actually hire someone like Kyle to dig in and verify the the consistency claims about a database.
      - CJefferson 3y ago
        I'm positive people want their databases to be correct. There are databases which promise great speed in return for occasionally losing bits of your data, and they get very little use outside of special uses.
        simplyinfinity 3y ago
        want != bet your life on them
        lambda 3y ago
        Exactly; people would like their databases to be correct and consistent, but generally don't care enough to actually do something like hiring someone like Kyle to verify it before buying a database. They just find something with the right features and performance, and go ahead and use it. You see a lot of them in this thread; people who used RavenDB for a while, and then had to migrate away because of issues. If the business actually cared enough about consistency, there would have been some kind of validation and verification before selecting a database.
- candiddevmike 3y ago
  I don't think any company explicitly claims their implementation handles X failure scenarios, certainly not in the EULA or license. You could possibly pick apart their documentation, but I think the warranty clause in most licenses would cover the product. This may not apply to regulated products dealing with health/safety/defense, and IANAL. Either way, best to assume most companies are lying through their teeth about any feature until you or someone you trust has validated it.
romanovcode 3y ago
RavenDB is expensive pay-to-use database. I do not understand why would one choose this over Postgres.
- beoberha 3y ago
  I had never heard of it until today, but a quick Google tells you it’s a Mongo-like JSON database with multi master capabilities. If you really really want those (I’m not sure it’s a good idea), then this seems way better than PG.
  - wmfiv 3y ago
    It's not better because it doesn't actually work. If you're not interested in a science experiment, Cassandra (or Scylla) are the multi-master databases that are mainstream and proven to work and scale. They're not fun or sexy and their feature set is much smaller but they do what they say and they work. Or AWS/GCP/Azure will happily give you an API for one.
    - endisneigh 3y ago
      It’s funny you say this when they’ve both failed jepsen tests. And by fail, I mean there are scenarios where they do not behave as intended - this is not to suggest either are broken inherently.
      - wmfiv 3y ago
        For sure. Virtually every Jepsen test has found bugs even with extremely mature products like Postgres. But fundamentally Cassandra (and Scylla) and most other databases tested have fixed those bugs and improved their documentation with many incorporating Jepsen tests into their ongoing process. That's different from how Raven has just wildly mistated their capabilities.
      - jjirsa 3y ago
        Cassandra’s jepsen is over 10 years old now. That database spent 3-4 years primarily focused on correctness from 2018-2022. The industry moves fast, our memories are slow. But there are millions of instances of cassandra in production across most of the fortune 500, and half of this thread has never heard of RavenDB
RcouF1uZ4gsC 3y ago
> AP systems are known for availability, not safety; I think in 99.9% of cases, you don't want AP. The P only matters when the network is more prone to go down than the machines. For example, if every node goes down, your AP design won't be available. With the massive improvements in network and connectivity and increased redundancy, you should aim for CP. If you really, really need AP, then a ground up design based on CRDTs seems the best, most discipline approach. With CRDT, you can have availability because the operations can be entirely local, and you know you can sync to the other nodes when available without conflict.
- mjb 3y ago
  Completely agree. In-region (e.g. at the scale of a US state), CP seems like a clear winner. For more geo-distributed latency becomes challenging, and in environments like IoT and mobile unreliable connectivity becomes challenging. If you're there, you need a principled way of approaching AP (e.g. CRDTs or CALM https://arxiv.org/pdf/1901.01930.pdf https://arxiv.org/pdf/1901.01930.pdf).
hudo 3y ago
I was using Raven around ver 1-3. Even it was single node and simple app, we observed stale reads and lost writes so had to eventually migrate to SQL Server. It was really weird reading claims from Oren (expert in .NET space, famous from his great work on Nhibernate and few other frameworks), where his db didn't work as advertised at all (back then build with Esent key/value store + full text search for map/reduce, think it was lucene.net - obviously very broken tech for this purpose). Too bad, was really hoped things were fixed by now, Db has really good programming APIs. Interesting trivia: there's "raven db done right" - https://martendb.io/ https://martendb.io/ , just an API wrapper around PSQL. Named Marten because thats a natural enemy of ravens:)
- jf22 3y ago
  Funny, I also had to abandon Raven around that time for the same reasons and ended up using MartenDb years later. MartenDb is great and the community around it is excellent.
- HdS84 3y ago
  I've used ravemsb 3 and did not observe auch problems. If I remember correctly, sent was replaced around ravemsb 2 and licence is now optional
  - AlfeG 3y ago
    We have abandonded raven because server tends to go into some "repair" state for hours eating 16+ Gb of memory. Multiple times on production. It were easier to migrate to posgres, then try to fix ravendb
gigatexal 3y ago
I love when Jepsen's reports hit HN. I always learn a ton about databases from them. Kudos to the projects brave enough to put their claims to the literal test. Jepsen is the best in the biz.
Twirrim 3y ago
When it comes to databases, that's when I get the most conservative in tech choices. Stick with the tried and tested approaches. Data/Metadata integrity is generally the single most important thing for whatever I'm working on.
- danielovichdk 3y ago
  I rarely used a RDBMS these days. But when I do I use a SQL Server. For the same reasons you just made.
  - Cwizard 3y ago
    What do you use instead?
endisneigh 3y ago
I’m still waiting on their report of foundationdb, which Kyle claims would readily pass their test so they didn’t bother to do one.
- aphyr 3y ago
  Even an unpaid report like this involves weeks of work: reading docs, designing tests, executing and refining them, filing issues, writing the report, and editing it. Each report costs thousands of dollars in hardware and and editing. I'd love to do an FDB test! It's just that between contracting, volunteer work, and research, I can only do so much.
  - willvarfar 3y ago
    We'd all love you to be _paid_ to do an FDB test! What are your most tempting/daunting databases that you haven't got a chance to put through their paces? And, a second question, if you step back and think about the various APIs you've had to use, have you personally developed favourite styles of API to use?
    - aphyr 3y ago
      Same. Hi, Apple! :D I'd love to do more work with predicates in general. That's an open research problem I've been noodling on for years. Pretty much any SQL DB would be a good candidate for that work! I'm gonna be a weirdo and say I actually loved Fauna's FQL. A little Lisp-ish functional language for queries is a great way to interact with document-structured data. SQL is fantastic for sheer breadth, though its specification is a nightmare and actually writing portable SQL is real challenging. One of those places where a stronger spec and conformance tests would have really helped.
      - freels 3y ago
        Thanks for the shoutout. At some point if you find yourself with some spare time you can check out our new FQL version. It's closer to JS in terms of syntax now, but still a small, relatively functional language.
jjirsa 3y ago
It’s disappointing to me that the technologist desire to experiment with new DBs continually puts naive customers at correctness and durability risk they don’t (won’t) understand.
ukd1 3y ago
This aged well - https://github.com/ravendb/ravendb/issues/13218#issuecomment-987884177 https://github.com/ravendb/ravendb/issues/13218#issuecomment...
- ukd1 3y ago
  Tickets related to this report from Aphyr: - https://github.com/ravendb/ravendb/issues/17928 https://github.com/ravendb/ravendb/issues/17928 - https://github.com/ravendb/ravendb/issues/17927 https://github.com/ravendb/ravendb/issues/17927
  - nemothekid 3y ago
    Very amusing what marketing will come up with; RavenDB doesn't support actual transactions, but supports "business" transactions.
- progbits 3y ago
  They claim to run that test on any changes but in fact might not be testing anything at all according to this footnote in the Jepsen report: > RavenDB’s Jepsen test may not have measured anything at all: at least in the most recent revision, the generator included no client operations of any kind. Remember folks, if you can't get your test to fail by intentionally breaking the implementation, you don't have a test.
mjb 3y ago
As database builders and users, we’ve made talking about systems a lot harder on ourselves by conflating the ideas of replication, active-active, atomic commitment, and concurrency control. - Replication is a technique used to achieve higher availability and durability than a single node can offer, by making multiple copies of the data. Techniques include Paxos, Raft, chain replication, quorum protocols, etc. - Active-active means that transactions can run against multiple different replicas at the same time, while still achieving the desired level of isolation and consistency. - Atomic commitment is a technique used in sharded/partitioned databases (which themselves exist to scale throughput or size beyond the capabilities of a single machine) to allow transactions to be atomically (“all or nothing”) committed across multiple shards (and allow one or more shards to vote “nah, let’s not commit this”). 2 phase commit (2PC) is the classic technique. - Concurrency control is a set of techniques to implement isolation, which is needed in any database that allows concurrent sessions (single node or multi-node). Classic techniques include 2PL and OCC, but many exist. When vendors or projects answer concurrency control questions with replication answers (which appears to be the case here), it’s worth diving deeper into those answers. There are cases where “Paxos” or “Raft” might be answers to atomic commitment or even concurrency control questions, but at best they are very partial answers and building blocks of a larger protocol. Databases that only support “single shot”/predeclared transactions can get away without a lot of concurrency control, for example, and might be able to do the required work as part of their state machine replication protocol. In general, I'd see using words like "Paxos" and "Raft" in the marketing for a database as a negative sign. It's not a fully reliable one, but it's often the least interesting part of the implementation and the choices the database is making. To be extra clear, I’m not criticizing Aphyr here (the article clearly doesn’t conflate these concepts), but more pointing out what I think lies at the bottom of a lot of the issues we see with distributed database claims.
- jwr 3y ago
  I think Aphyr helped a lot by creating this useful resource: https://jepsen.io/consistency https://jepsen.io/consistency which presents a clear classification of consistency models. I am not sure if talking about anything else in the context of distributed databases is reasonable.
PreInternet01 3y ago
My fun RavenDB story: I briefly used it for an analytics (music royalty data, not advertising) solution somewhere late in the 1.x release series. Ayende (the initial RavenDB author) was/is an avid blogger, and made a really good case for their product in the .NET ecosystem. It did not... go exactly as planned. Initial tests looked OK, but when I did testing with actual users, there were huge issues right away. Like: OK, I just ran your ingestion pipeline. What do you see? And the answer was 'well, nothing', or 'ehhm, a lot less than I expected'. These issues turned out to be pretty much impossible to fix: there were no real errors, but the data just seemed to... disappear randomly, even in a simple single-node cluster. I got community support involved in a bunch of particular issues, but nothing really helped: the aggregate numbers we got never added up to what they should be. I then migrated the whole thing to a single SQLite database. That file is, as I write this, a good 2TB in size, and still performs as well as the day it was deployed and never had any unexplained-number issues, without any changes to the surrounding code. I did eventually move away from the .NET Entity Framework (as that did cause some rare, yet unexpected and hard-to-fix concurrency issues, but those were hard crashes and not silent data corruptions) to a hand-rolled entity mapper, but all has been good since then... TL;DR: databases are very hard, and fashionable choices are not necessarily desirable.
jwr 3y ago
Healthy reminder that a pretty website and warm fuzzies all over do not make a distributed database actually work. I witnessed RethinkDB losing to MongoDB in spite of being significantly better. I am now worried that FoundationDB isn't gaining popularity, even though it is arguably the best and most well-tested distributed database out there, with strict serializability (!) guarantees. But it doesn't have a shiny website and doesn't cause warm fuzzies, quite the opposite, it looks complex and intimidating. So it isn't popular. This is worrying, but perhaps neither new nor surprising: we have a history of picking inferior solutions because the good ones looked too complex or intimidating (betamax vs VHS in video formats, ATM vs Ethernet in WANs).
- redwood 3y ago
  I had understood FoundationDB to be more akin to a storage engine (e.g. a sub-component of a DBMS) than a full-on DBMS. Was I misunderstanding? If so I bet a lot of people have this understanding, going back to your point on the web site/general sentiment in the zeitgeist not necessarily reflecting what they are. Can you share any more detail? Are you saying there are companies that build software on top of FoundationDB as their primary data store? or are those companies building software around FoundationDB that in turn presents as more of a data store in the traditional sense?
  - saled 3y ago
    Snowflake uses foundationdb as their metadata store to find files in S3, though I believe the actual queries are run on separate nodes. https://www.snowflake.com/blog/how-foundationdb-powers-snowflake-metadata-forward/ https://www.snowflake.com/blog/how-foundationdb-powers-snowf... But yeah you're right for the most part. Turns out pretty much any database can be written in terms of transactions of KV pairs, which is what foundationdb gives you, so it means you can write your database query layer as a stateless, scalable service. There have been attempts to write a SQL RDMS layer for it but it isn't maintained.
  - jwr 3y ago
    It's more of a database-building toolkit than a storage engine. What you get is a distributed KV store with a strict serializable consistency model (see https://jepsen.io/consistency https://jepsen.io/consistency) and very interesting versionstamp functionality. What you do not get is a "query language" or indexing. Yes, there are companies that use FoundationDB as their primary data store. It makes a lot of sense to integrate it directly with the application rather than go through additional layers and a "query language". I am working on adapting my app to use it, and so far very happy with the results.
philipbjorge 3y ago
We used RavenDB 2-4 at Leafly. Won't go into battle scars here, but this report does not surprise me. We're much happier with Postgres and Elasticsearch.
JazCE 3y ago
Kelly Somers has been vindicated.
CyanLite2 3y ago
I gave up on RavenDB when Oren would post blog entries regarding interviews with candidates, bashing how they would fail his coding exams. I mean who posts that kinda stuff on their public website?
dramm 3y ago
I need a brain colonic after reading though just some of the mess of overhyped claims in RavenDB marketing and documentation. I appreciate Aphyr doing all this wonderful work and how some of the Raven claims triggered that work. I'd have hoped that anybody building a critical system would have read the mess of Raven documentation/claims/hype and run the other way.
oliverpk 3y ago
Genuinely one of my favourite posts here
neonsunset 3y ago
The unfortunate thing is .NET deserves to have a proper database written in pure C# because the language offers all the tools to achieve a really performant, safe and cross-platform implementation. But RavenDB does not do it justice and uses unsafe in catastrophic amounts in places where it is not necessary or in ways which are straight up UB despite the fact that JIT/ILC is much more strict than GCC/LLVM. There have been multiple bug reports submitted to dotnet/runtime by RavenDB which required extensive debugging effort only to end up being an issue on RavenDBs end due to explicit misuse of unsafe APIs (in ways, I must reiterate, that have safe alternatives to achieve the same performance). (if anyone's interested, I can later ask around/dig through issue history and give the references)

195 points by aphyr 3y ago

I love Jepsen, but it seriously worries me how bad software turns out to be, and how many outrageous claims companies make that turn out to be so easily proved false. Should there be more serious penalties when companies make claims which turn out to be false as soon as they are tested? I think there should be.

skyde 3y ago
Well some company try to implement Raft algorithm with good intentions but do a bad job a at so bugs make it so they fail Jensen test. You can’t force people to test their software correctly unless it’s a regulated field like aviation.
- okigan 3y ago
  [flagged]
  - dallasg3 3y ago
    Also is Jepsen, not Jensen.
    - myk9001 3y ago
      Named after Carly Rae Jepsen for her song Call Me Maybe.
  - lambda 3y ago
    Aviation has an extremely strong safety record, which has been getting better year by year. Yes, there are misses, but they have been happening increasingly less frequently over time. https://en.wikipedia.org/wiki/Aviation_safety#/media/File:Fatalities_per_revenue_passenger_kilometre_in_air_transport_since_1970.png https://en.wikipedia.org/wiki/Aviation_safety#/media/File:Fa... Aviation isn't perfect; nothing implemented by large groups of fallible humans with budget constraints will be. But it has one of the best safety track records of any industry. Now, why won't aviation style engineering be applied in other fields, like databases? Well, because no one really cares enough. No one dies if some random database used by some ad platform loses the occasional transaction. Yeah, it's frustrating to engineers who are trying to build reliable systems, but in the grand scheme of things losing a few percent of transactions isn't the end of the world for most businesses. You get safety cultures like that in aviation because there are real, substantial risks, so you need to have thorough engineering discipline, properly designed redundancy, etc. For databases that are used for the majority of the business world, efficiency is generally a bigger concern than correctness; they'd rather have cheap and fast databases that lose a few transactions occasionally than something that actually provides consistency. But of course everyone thinks they need consistency, so it's advertised as a selling point while not actually being provided in practice.
    - _a_a_a_ 3y ago
      > For databases that are used for the majority of the business world, efficiency is generally a bigger concern than correctness; they'd rather have cheap and fast databases that lose a few transactions occasionally than something that actually provides consistency 'majority' is a strong claim. Any figures to back it up?
      - lambda 3y ago
        No, no figures to back it up. This is just based on anecdotal experience, having never seen very many businesses that actually prioritize consistency or testing or validating it. The fact that Kyle keeps on finding these massive problems in popular databases is part of that anecdotal evidence. Obviously people want correctness; they would like their databases to not randomly lose data. Hence the fact that it's a highly advertised feature. But when it actually comes down to selecting a database, convenience and performance seem to be what people actually compare on, there are very few places that actually hire someone like Kyle to dig in and verify the the consistency claims about a database.
    - CJefferson 3y ago
      I'm positive people want their databases to be correct. There are databases which promise great speed in return for occasionally losing bits of your data, and they get very little use outside of special uses.
      - simplyinfinity 3y ago
        want != bet your life on them
        lambda 3y ago
        Exactly; people would like their databases to be correct and consistent, but generally don't care enough to actually do something like hiring someone like Kyle to verify it before buying a database. They just find something with the right features and performance, and go ahead and use it. You see a lot of them in this thread; people who used RavenDB for a while, and then had to migrate away because of issues. If the business actually cared enough about consistency, there would have been some kind of validation and verification before selecting a database.
candiddevmike 3y ago
I don't think any company explicitly claims their implementation handles X failure scenarios, certainly not in the EULA or license. You could possibly pick apart their documentation, but I think the warranty clause in most licenses would cover the product. This may not apply to regulated products dealing with health/safety/defense, and IANAL. Either way, best to assume most companies are lying through their teeth about any feature until you or someone you trust has validated it.

romanovcode 3y ago

RavenDB is expensive pay-to-use database. I do not understand why would one choose this over Postgres.

beoberha 3y ago
I had never heard of it until today, but a quick Google tells you it’s a Mongo-like JSON database with multi master capabilities. If you really really want those (I’m not sure it’s a good idea), then this seems way better than PG.
- wmfiv 3y ago
  It's not better because it doesn't actually work. If you're not interested in a science experiment, Cassandra (or Scylla) are the multi-master databases that are mainstream and proven to work and scale. They're not fun or sexy and their feature set is much smaller but they do what they say and they work. Or AWS/GCP/Azure will happily give you an API for one.
  - endisneigh 3y ago
    It’s funny you say this when they’ve both failed jepsen tests. And by fail, I mean there are scenarios where they do not behave as intended - this is not to suggest either are broken inherently.
    - wmfiv 3y ago
      For sure. Virtually every Jepsen test has found bugs even with extremely mature products like Postgres. But fundamentally Cassandra (and Scylla) and most other databases tested have fixed those bugs and improved their documentation with many incorporating Jepsen tests into their ongoing process. That's different from how Raven has just wildly mistated their capabilities.
    - jjirsa 3y ago
      Cassandra’s jepsen is over 10 years old now. That database spent 3-4 years primarily focused on correctness from 2018-2022. The industry moves fast, our memories are slow. But there are millions of instances of cassandra in production across most of the fortune 500, and half of this thread has never heard of RavenDB

RcouF1uZ4gsC 3y ago

> AP systems are known for availability, not safety; I think in 99.9% of cases, you don't want AP. The P only matters when the network is more prone to go down than the machines. For example, if every node goes down, your AP design won't be available. With the massive improvements in network and connectivity and increased redundancy, you should aim for CP. If you really, really need AP, then a ground up design based on CRDTs seems the best, most discipline approach. With CRDT, you can have availability because the operations can be entirely local, and you know you can sync to the other nodes when available without conflict.

mjb 3y ago
Completely agree. In-region (e.g. at the scale of a US state), CP seems like a clear winner. For more geo-distributed latency becomes challenging, and in environments like IoT and mobile unreliable connectivity becomes challenging. If you're there, you need a principled way of approaching AP (e.g. CRDTs or CALM https://arxiv.org/pdf/1901.01930.pdf https://arxiv.org/pdf/1901.01930.pdf).

hudo 3y ago

I was using Raven around ver 1-3. Even it was single node and simple app, we observed stale reads and lost writes so had to eventually migrate to SQL Server. It was really weird reading claims from Oren (expert in .NET space, famous from his great work on Nhibernate and few other frameworks), where his db didn't work as advertised at all (back then build with Esent key/value store + full text search for map/reduce, think it was lucene.net - obviously very broken tech for this purpose). Too bad, was really hoped things were fixed by now, Db has really good programming APIs. Interesting trivia: there's "raven db done right" - https://martendb.io/ https://martendb.io/ , just an API wrapper around PSQL. Named Marten because thats a natural enemy of ravens:)

jf22 3y ago
Funny, I also had to abandon Raven around that time for the same reasons and ended up using MartenDb years later. MartenDb is great and the community around it is excellent.
HdS84 3y ago
I've used ravemsb 3 and did not observe auch problems. If I remember correctly, sent was replaced around ravemsb 2 and licence is now optional
- AlfeG 3y ago
  We have abandonded raven because server tends to go into some "repair" state for hours eating 16+ Gb of memory. Multiple times on production. It were easier to migrate to posgres, then try to fix ravendb

gigatexal 3y ago

I love when Jepsen's reports hit HN. I always learn a ton about databases from them. Kudos to the projects brave enough to put their claims to the literal test. Jepsen is the best in the biz.

Twirrim 3y ago

When it comes to databases, that's when I get the most conservative in tech choices. Stick with the tried and tested approaches. Data/Metadata integrity is generally the single most important thing for whatever I'm working on.

danielovichdk 3y ago
I rarely used a RDBMS these days. But when I do I use a SQL Server. For the same reasons you just made.
- Cwizard 3y ago
  What do you use instead?

endisneigh 3y ago

I’m still waiting on their report of foundationdb, which Kyle claims would readily pass their test so they didn’t bother to do one.

aphyr 3y ago
Even an unpaid report like this involves weeks of work: reading docs, designing tests, executing and refining them, filing issues, writing the report, and editing it. Each report costs thousands of dollars in hardware and and editing. I'd love to do an FDB test! It's just that between contracting, volunteer work, and research, I can only do so much.
- willvarfar 3y ago
  We'd all love you to be _paid_ to do an FDB test! What are your most tempting/daunting databases that you haven't got a chance to put through their paces? And, a second question, if you step back and think about the various APIs you've had to use, have you personally developed favourite styles of API to use?
  - aphyr 3y ago
    Same. Hi, Apple! :D I'd love to do more work with predicates in general. That's an open research problem I've been noodling on for years. Pretty much any SQL DB would be a good candidate for that work! I'm gonna be a weirdo and say I actually loved Fauna's FQL. A little Lisp-ish functional language for queries is a great way to interact with document-structured data. SQL is fantastic for sheer breadth, though its specification is a nightmare and actually writing portable SQL is real challenging. One of those places where a stronger spec and conformance tests would have really helped.
    - freels 3y ago
      Thanks for the shoutout. At some point if you find yourself with some spare time you can check out our new FQL version. It's closer to JS in terms of syntax now, but still a small, relatively functional language.

jjirsa 3y ago

It’s disappointing to me that the technologist desire to experiment with new DBs continually puts naive customers at correctness and durability risk they don’t (won’t) understand.

ukd1 3y ago

This aged well - https://github.com/ravendb/ravendb/issues/13218#issuecomment-987884177 https://github.com/ravendb/ravendb/issues/13218#issuecomment...

ukd1 3y ago
Tickets related to this report from Aphyr: - https://github.com/ravendb/ravendb/issues/17928 https://github.com/ravendb/ravendb/issues/17928 - https://github.com/ravendb/ravendb/issues/17927 https://github.com/ravendb/ravendb/issues/17927
- nemothekid 3y ago
  Very amusing what marketing will come up with; RavenDB doesn't support actual transactions, but supports "business" transactions.
progbits 3y ago
They claim to run that test on any changes but in fact might not be testing anything at all according to this footnote in the Jepsen report: > RavenDB’s Jepsen test may not have measured anything at all: at least in the most recent revision, the generator included no client operations of any kind. Remember folks, if you can't get your test to fail by intentionally breaking the implementation, you don't have a test.

mjb 3y ago

As database builders and users, we’ve made talking about systems a lot harder on ourselves by conflating the ideas of replication, active-active, atomic commitment, and concurrency control. - Replication is a technique used to achieve higher availability and durability than a single node can offer, by making multiple copies of the data. Techniques include Paxos, Raft, chain replication, quorum protocols, etc. - Active-active means that transactions can run against multiple different replicas at the same time, while still achieving the desired level of isolation and consistency. - Atomic commitment is a technique used in sharded/partitioned databases (which themselves exist to scale throughput or size beyond the capabilities of a single machine) to allow transactions to be atomically (“all or nothing”) committed across multiple shards (and allow one or more shards to vote “nah, let’s not commit this”). 2 phase commit (2PC) is the classic technique. - Concurrency control is a set of techniques to implement isolation, which is needed in any database that allows concurrent sessions (single node or multi-node). Classic techniques include 2PL and OCC, but many exist. When vendors or projects answer concurrency control questions with replication answers (which appears to be the case here), it’s worth diving deeper into those answers. There are cases where “Paxos” or “Raft” might be answers to atomic commitment or even concurrency control questions, but at best they are very partial answers and building blocks of a larger protocol. Databases that only support “single shot”/predeclared transactions can get away without a lot of concurrency control, for example, and might be able to do the required work as part of their state machine replication protocol. In general, I'd see using words like "Paxos" and "Raft" in the marketing for a database as a negative sign. It's not a fully reliable one, but it's often the least interesting part of the implementation and the choices the database is making. To be extra clear, I’m not criticizing Aphyr here (the article clearly doesn’t conflate these concepts), but more pointing out what I think lies at the bottom of a lot of the issues we see with distributed database claims.

jwr 3y ago
I think Aphyr helped a lot by creating this useful resource: https://jepsen.io/consistency https://jepsen.io/consistency which presents a clear classification of consistency models. I am not sure if talking about anything else in the context of distributed databases is reasonable.

PreInternet01 3y ago

My fun RavenDB story: I briefly used it for an analytics (music royalty data, not advertising) solution somewhere late in the 1.x release series. Ayende (the initial RavenDB author) was/is an avid blogger, and made a really good case for their product in the .NET ecosystem. It did not... go exactly as planned. Initial tests looked OK, but when I did testing with actual users, there were huge issues right away. Like: OK, I just ran your ingestion pipeline. What do you see? And the answer was 'well, nothing', or 'ehhm, a lot less than I expected'. These issues turned out to be pretty much impossible to fix: there were no real errors, but the data just seemed to... disappear randomly, even in a simple single-node cluster. I got community support involved in a bunch of particular issues, but nothing really helped: the aggregate numbers we got never added up to what they should be. I then migrated the whole thing to a single SQLite database. That file is, as I write this, a good 2TB in size, and still performs as well as the day it was deployed and never had any unexplained-number issues, without any changes to the surrounding code. I did eventually move away from the .NET Entity Framework (as that did cause some rare, yet unexpected and hard-to-fix concurrency issues, but those were hard crashes and not silent data corruptions) to a hand-rolled entity mapper, but all has been good since then... TL;DR: databases are very hard, and fashionable choices are not necessarily desirable.

jwr 3y ago

Healthy reminder that a pretty website and warm fuzzies all over do not make a distributed database actually work. I witnessed RethinkDB losing to MongoDB in spite of being significantly better. I am now worried that FoundationDB isn't gaining popularity, even though it is arguably the best and most well-tested distributed database out there, with strict serializability (!) guarantees. But it doesn't have a shiny website and doesn't cause warm fuzzies, quite the opposite, it looks complex and intimidating. So it isn't popular. This is worrying, but perhaps neither new nor surprising: we have a history of picking inferior solutions because the good ones looked too complex or intimidating (betamax vs VHS in video formats, ATM vs Ethernet in WANs).

redwood 3y ago
I had understood FoundationDB to be more akin to a storage engine (e.g. a sub-component of a DBMS) than a full-on DBMS. Was I misunderstanding? If so I bet a lot of people have this understanding, going back to your point on the web site/general sentiment in the zeitgeist not necessarily reflecting what they are. Can you share any more detail? Are you saying there are companies that build software on top of FoundationDB as their primary data store? or are those companies building software around FoundationDB that in turn presents as more of a data store in the traditional sense?
- saled 3y ago
  Snowflake uses foundationdb as their metadata store to find files in S3, though I believe the actual queries are run on separate nodes. https://www.snowflake.com/blog/how-foundationdb-powers-snowflake-metadata-forward/ https://www.snowflake.com/blog/how-foundationdb-powers-snowf... But yeah you're right for the most part. Turns out pretty much any database can be written in terms of transactions of KV pairs, which is what foundationdb gives you, so it means you can write your database query layer as a stateless, scalable service. There have been attempts to write a SQL RDMS layer for it but it isn't maintained.
- jwr 3y ago
  It's more of a database-building toolkit than a storage engine. What you get is a distributed KV store with a strict serializable consistency model (see https://jepsen.io/consistency https://jepsen.io/consistency) and very interesting versionstamp functionality. What you do not get is a "query language" or indexing. Yes, there are companies that use FoundationDB as their primary data store. It makes a lot of sense to integrate it directly with the application rather than go through additional layers and a "query language". I am working on adapting my app to use it, and so far very happy with the results.

philipbjorge 3y ago

We used RavenDB 2-4 at Leafly. Won't go into battle scars here, but this report does not surprise me. We're much happier with Postgres and Elasticsearch.

JazCE 3y ago

Kelly Somers has been vindicated.

CyanLite2 3y ago

I gave up on RavenDB when Oren would post blog entries regarding interviews with candidates, bashing how they would fail his coding exams. I mean who posts that kinda stuff on their public website?

dramm 3y ago

I need a brain colonic after reading though just some of the mess of overhyped claims in RavenDB marketing and documentation. I appreciate Aphyr doing all this wonderful work and how some of the Raven claims triggered that work. I'd have hoped that anybody building a critical system would have read the mess of Raven documentation/claims/hype and run the other way.

oliverpk 3y ago

Genuinely one of my favourite posts here

neonsunset 3y ago

The unfortunate thing is .NET deserves to have a proper database written in pure C# because the language offers all the tools to achieve a really performant, safe and cross-platform implementation. But RavenDB does not do it justice and uses unsafe in catastrophic amounts in places where it is not necessary or in ways which are straight up UB despite the fact that JIT/ILC is much more strict than GCC/LLVM. There have been multiple bug reports submitted to dotnet/runtime by RavenDB which required extensive debugging effort only to end up being an issue on RavenDBs end due to explicit misuse of unsafe APIs (in ways, I must reiterate, that have safe alternatives to achieve the same performance). (if anyone's interested, I can later ask around/dig through issue history and give the references)