7 ms·

For Anyone with experience with FoundationDB! How does it compare to CockroachDB ?

by skyde 5y ago

eatonphil 5y ago
FoundationDB is just a key value store, so it's not comparable. There is a relational layer you can run though called Foundation Record Layer. But this doesn't support SQL AFAICT. The examples on Github just show a query builder interface in Java [0]. There does seem to be a Lucene interface but it doesn't seem to be used [1]. [0] https://github.com/FoundationDB/fdb-record-layer/blob/main/examples/src/main/java/com/apple/foundationdb/record/sample/Main.java https://github.com/FoundationDB/fdb-record-layer/blob/main/e... [1] https://forums.foundationdb.org/t/lucene-layer-on-foundationdb/1229/5 https://forums.foundationdb.org/t/lucene-layer-on-foundation...
- StreamBright 5y ago
  There used to be some SQL layer: https://forums.foundationdb.org/t/sql-layer-in-foundationdb/94 https://forums.foundationdb.org/t/sql-layer-in-foundationdb/...
  - eatonphil 5y ago
    Neat, here [0] is a 3rd party one at the end of the thread but this isn't the one the thread is about. Doesn't seem like that one is public. [0] https://github.com/rustdesk/opentick https://github.com/rustdesk/opentick
    - aseipp 5y ago
      It was open source and if you Google a bit you can find forks with the relevant code snapshots, here's one: https://github.com/AydinSakar/sql-layer https://github.com/AydinSakar/sql-layer
- tonyhb 5y ago
  It's comparable based off of the fundamentals and I have the same question. They both use KVs using LSMs, do range queries, support ACID transactions, handle resharding and clustering for you. They're both based off of Spanner fundamentals. They both actually have an SQL layer - foundationdb's is the record layer. Just because one has a primary SQL interface doesn't mean we can't compare. I'd really like to know a comparison of the fundamentals including hotspot management, resharding, read/write performance, resharding etc. also and have been looking for this.
  - eatonphil 5y ago
    > They both actually have an SQL layer - foundationdb's is the record layer. Can you show me their SQL layer? I was looking for it.
    - tonyhb 5y ago
      https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer is the SQL/orm like interface. This handles things in a DB like way - joins, indexes. I misspoke when I said SQL regarding this - sorry. There's a go and rust library floating around that's not as good. I've tried em :)
      - eatonphil 5y ago
        The readme says > Queries - The Record Layer does not provide a query language, however it provides query APIs with the ability to scan, filter, and sort across one or more record types, and a query planner capable of automatic selection of indexes. So I'm not sure how you can call that a SQL layer when SQL is a query language and FDB Record Layer says it doesn't have a query language?
        tonyhb 5y ago
        Yeah, I did say "I misspoke when I said SQL regarding this - sorry.". I should have said "SQL-like", or at least relation-like. There were a couple SQL-ish things I found in other languages and got them mixed up. Sorry about that :)
ledgerdev 5y ago
I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltactivedata.com/blog/2015/04/foundationdbs-le... Also this, excellent comparison of distributed database consistency methods, mentions cockroach, but not foundation. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst... This brings up the question, what method/protocol does foundation use for distributed consistency?
- richieartoul 5y ago
  "What method/protocol does foundation use for distributed consistency?" This is a really tough question to answer succinctly. It took me a long time to wrap my head around it, but the TLDR is: 1. It uses a quorum of Coordinator nodes to elect another node (via Paxos) whose job it is to then assigns roles to other nodes. 2. Some of those roles are replicated (Like TLogs) and some of those roles are singletons (like the thing that advances the clock and hands out versionstamps). 3. If any node in the "transaction subsystem" ever fails, the cluster goes through a recovery where it becomes unavailable for ~3s during which it reassigns the roles and keeps chugging. In terms of what it looks like for a transaction to actually commit, thats another long conversation. Its hard to draw analogies to existing systems because it really is kind of its own thing.
  - ledgerdev 5y ago
    Awesome answer, thanks! Foundation seems like magic almost, from just choosing very smart trade offs and co-ordination.
    - richieartoul 5y ago
      Yeah that’s basically it. It really is a wonderful piece of free software that was (and still is) way ahead of its time in many ways. They basically made a ton of good architectural decisions really early on and then just nailed down the details super tight over many years. Their technique of running cluster-wide recoveries to handle failures is a great example. Most distributed systems engineers find the idea of designing a database that becomes unavailable (even for a few seconds) when a single node is lost completely appalling, but it turns out this small trade off leads to massive simplifications elsewhere and the resulting system is net/net much more reliable than anything else I’ve ever seen.
manigandham 5y ago
FoundationDB is a key/value store, and well-known for being fast, scalable and incredibly reliable. It passed many of the distributed database tests unlike many competitors. Here's a good HN thread when Apple open-sourced it a few years ago: https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395 CockroachDB implements postgres-compatible SQL layer on top of their own key/value store called Pebble. This is similar to TiDB which implements mysql on top of their TiKV key/value store. The latter databases offer easier programming with SQL interfaces, but since you can model just about anything on top of key/value store, there are already "layers" that can implement similar abstractions on top of FDB like the Record Layer[1], Document Layer [2], and even a SQL layer [3] at one point. 1) https://www.foundationdb.org/blog/announcing-record-layer/ https://www.foundationdb.org/blog/announcing-record-layer/ 2) https://foundationdb.github.io/fdb-document-layer/ https://foundationdb.github.io/fdb-document-layer/ 3) https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer
- EGreg 5y ago
  What is better, FoundationDB, CockroachDB, or Hypercore?
  - vvern 5y ago
    One of these is not like the others.
- atombender 5y ago
  An important difference between CockroachDB and TiDB is that the former has an embedded key-value store; everything runs in the same process. TiDB is split into multiple parts: TiDB proper (stateless SQL/query plan layer), TiKV (the key-value store), and PD (placement driver, the main control plane and metadata layer). TiKV also isn't a pure key-value store, as it handles some particular query operations that can be "pushed down" from TiDB to reduce network round trips. FoundationDB's role similar to TiKV. It's a pure key-value store with replication built in.
- jandrewrogers 5y ago
  There is one caveat I would add. While you can always build just about any kind of database or data model on top of a logical KV store, the scalability and performance is almost always going to be qualitatively worse for queries than using a storage engine purpose-built for the intended type of database or data model. It is simpler to implement, which is why people do it, but you give up a lot of selectivity and locality on the query side. The target design case for KV stores is when all your queries are essentially trivial. Layering SQL on top of it implies the expected use case is complex queries of the type where a KV store won't perform well. Good for a compatibility interface, but not performant.
richieartoul 5y ago
I've never used CockroachDB, but FoundationDB has been an absolute dream to work with. There are a few downsides to using it: - There is (currently) no hosted solution you can use, so you have to run it yourself. - There is no great "layer" that provides a nice document/SQL layer so you kind of have to work with the raw sorted KV abstraction. - You have to structure your workload to avoid a high rate of conflicts on individual keys. - It won't scale to extremely high write volume workloads like Cassandra/Scylla will. That said, in exchange you get: - Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. Jepsen refuses to test FDB because FDB's simulation testing is much more rigorous than anything Jepsen does, but AFAIK the most recent releases of Cockroach/Mongo/Cassandra/Fauna/etc have never fully passed a Jepsen test. - Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput. - A strongly consistent store you can actually use in anger and that you don't have to baby. We've never managed to actually break a FoundationDB cluster, the worst case scenario we've ever seen is that we slam it with too many writes/reads and it starts to apply backpressure by increasing latency. - Write/Read throughput scales ~linearly as nodes are added. - Built in backup/restore from blob storage. - Strongly consistent disaster recovery. This one is amazing and its hard to go back to other databases that don't support it, but basically you can run 2 FDB clusters in DR mode where one is the leader (accepting writes + reads) and the other is passively replicating from it. You can then fail over from the primary FDB to the secondary *consistently* in such a way that the two clusters coordinate to lock the primary and then unlock the secondary so that your application is guaranteed to not see any inconsistency or data loss with no participation from the application required (except teaching it how to talk to 2 different FDB clusters at once). Basically its lacking some niceties for building traditional applications on top of it, but it is an amazing primitive for building rock solid distributed systems and will almost never let you down once you've developed some operational experience with it. Its the closest thing to "magic" that I've ever worked with in the distributed storage space.
- sigstoat 5y ago
  to the folks asking why you'd use FDB, i'd highlight the main points as: > Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. the transaction guarantees on FDB are just the best there is in any product you can get for free. not to fault the rest, FDB just went past what ~98% of folks will ever need. but if you're in that last 2%, god speed trying to get anything else to offer you the same semantics. > A strongly consistent store you can actually use in anger and that you don't have to baby. i've seen some folks trying to run a cluster without proper monitoring, who were nearly driving the poor thing into the ground, with hosts dying, and having so much pressure put on them they failed out, but it never lost anything. i don't think it ever even became unavailable. > Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput eh, in defense of everything else, FDB is a key value store, and the keys are byte strings, so the situation is a bit easier for them.

For Anyone with experience with FoundationDB! How does it compare to CockroachDB ?

by skyde 5y ago

For Anyone with experience with FoundationDB! How does it compare to CockroachDB ?

FoundationDB is just a key value store, so it's not comparable. There is a relational layer you can run though called Foundation Record Layer. But this doesn't support SQL AFAICT. The examples on Github just show a query builder interface in Java [0]. There does seem to be a Lucene interface but it doesn't seem to be used [1]. [0] https://github.com/FoundationDB/fdb-record-layer/blob/main/examples/src/main/java/com/apple/foundationdb/record/sample/Main.java https://github.com/FoundationDB/fdb-record-layer/blob/main/e... [1] https://forums.foundationdb.org/t/lucene-layer-on-foundationdb/1229/5 https://forums.foundationdb.org/t/lucene-layer-on-foundation...

StreamBright 5y ago
There used to be some SQL layer: https://forums.foundationdb.org/t/sql-layer-in-foundationdb/94 https://forums.foundationdb.org/t/sql-layer-in-foundationdb/...
- eatonphil 5y ago
  Neat, here [0] is a 3rd party one at the end of the thread but this isn't the one the thread is about. Doesn't seem like that one is public. [0] https://github.com/rustdesk/opentick https://github.com/rustdesk/opentick
  - aseipp 5y ago
    It was open source and if you Google a bit you can find forks with the relevant code snapshots, here's one: https://github.com/AydinSakar/sql-layer https://github.com/AydinSakar/sql-layer
tonyhb 5y ago
It's comparable based off of the fundamentals and I have the same question. They both use KVs using LSMs, do range queries, support ACID transactions, handle resharding and clustering for you. They're both based off of Spanner fundamentals. They both actually have an SQL layer - foundationdb's is the record layer. Just because one has a primary SQL interface doesn't mean we can't compare. I'd really like to know a comparison of the fundamentals including hotspot management, resharding, read/write performance, resharding etc. also and have been looking for this.
- eatonphil 5y ago
  > They both actually have an SQL layer - foundationdb's is the record layer. Can you show me their SQL layer? I was looking for it.
  - tonyhb 5y ago
    https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer is the SQL/orm like interface. This handles things in a DB like way - joins, indexes. I misspoke when I said SQL regarding this - sorry. There's a go and rust library floating around that's not as good. I've tried em :)
    - eatonphil 5y ago
      The readme says > Queries - The Record Layer does not provide a query language, however it provides query APIs with the ability to scan, filter, and sort across one or more record types, and a query planner capable of automatic selection of indexes. So I'm not sure how you can call that a SQL layer when SQL is a query language and FDB Record Layer says it doesn't have a query language?
      - tonyhb 5y ago
        Yeah, I did say "I misspoke when I said SQL regarding this - sorry.". I should have said "SQL-like", or at least relation-like. There were a couple SQL-ish things I found in other languages and got them mixed up. Sorry about that :)

ledgerdev 5y ago

I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltactivedata.com/blog/2015/04/foundationdbs-le... Also this, excellent comparison of distributed database consistency methods, mentions cockroach, but not foundation. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst... This brings up the question, what method/protocol does foundation use for distributed consistency?

richieartoul 5y ago
"What method/protocol does foundation use for distributed consistency?" This is a really tough question to answer succinctly. It took me a long time to wrap my head around it, but the TLDR is: 1. It uses a quorum of Coordinator nodes to elect another node (via Paxos) whose job it is to then assigns roles to other nodes. 2. Some of those roles are replicated (Like TLogs) and some of those roles are singletons (like the thing that advances the clock and hands out versionstamps). 3. If any node in the "transaction subsystem" ever fails, the cluster goes through a recovery where it becomes unavailable for ~3s during which it reassigns the roles and keeps chugging. In terms of what it looks like for a transaction to actually commit, thats another long conversation. Its hard to draw analogies to existing systems because it really is kind of its own thing.
- ledgerdev 5y ago
  Awesome answer, thanks! Foundation seems like magic almost, from just choosing very smart trade offs and co-ordination.
  - richieartoul 5y ago
    Yeah that’s basically it. It really is a wonderful piece of free software that was (and still is) way ahead of its time in many ways. They basically made a ton of good architectural decisions really early on and then just nailed down the details super tight over many years. Their technique of running cluster-wide recoveries to handle failures is a great example. Most distributed systems engineers find the idea of designing a database that becomes unavailable (even for a few seconds) when a single node is lost completely appalling, but it turns out this small trade off leads to massive simplifications elsewhere and the resulting system is net/net much more reliable than anything else I’ve ever seen.

manigandham 5y ago

FoundationDB is a key/value store, and well-known for being fast, scalable and incredibly reliable. It passed many of the distributed database tests unlike many competitors. Here's a good HN thread when Apple open-sourced it a few years ago: https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395 CockroachDB implements postgres-compatible SQL layer on top of their own key/value store called Pebble. This is similar to TiDB which implements mysql on top of their TiKV key/value store. The latter databases offer easier programming with SQL interfaces, but since you can model just about anything on top of key/value store, there are already "layers" that can implement similar abstractions on top of FDB like the Record Layer[1], Document Layer [2], and even a SQL layer [3] at one point. 1) https://www.foundationdb.org/blog/announcing-record-layer/ https://www.foundationdb.org/blog/announcing-record-layer/ 2) https://foundationdb.github.io/fdb-document-layer/ https://foundationdb.github.io/fdb-document-layer/ 3) https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer

EGreg 5y ago
What is better, FoundationDB, CockroachDB, or Hypercore?
- vvern 5y ago
  One of these is not like the others.
atombender 5y ago
An important difference between CockroachDB and TiDB is that the former has an embedded key-value store; everything runs in the same process. TiDB is split into multiple parts: TiDB proper (stateless SQL/query plan layer), TiKV (the key-value store), and PD (placement driver, the main control plane and metadata layer). TiKV also isn't a pure key-value store, as it handles some particular query operations that can be "pushed down" from TiDB to reduce network round trips. FoundationDB's role similar to TiKV. It's a pure key-value store with replication built in.
jandrewrogers 5y ago
There is one caveat I would add. While you can always build just about any kind of database or data model on top of a logical KV store, the scalability and performance is almost always going to be qualitatively worse for queries than using a storage engine purpose-built for the intended type of database or data model. It is simpler to implement, which is why people do it, but you give up a lot of selectivity and locality on the query side. The target design case for KV stores is when all your queries are essentially trivial. Layering SQL on top of it implies the expected use case is complex queries of the type where a KV store won't perform well. Good for a compatibility interface, but not performant.

richieartoul 5y ago

I've never used CockroachDB, but FoundationDB has been an absolute dream to work with. There are a few downsides to using it: - There is (currently) no hosted solution you can use, so you have to run it yourself. - There is no great "layer" that provides a nice document/SQL layer so you kind of have to work with the raw sorted KV abstraction. - You have to structure your workload to avoid a high rate of conflicts on individual keys. - It won't scale to extremely high write volume workloads like Cassandra/Scylla will. That said, in exchange you get: - Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. Jepsen refuses to test FDB because FDB's simulation testing is much more rigorous than anything Jepsen does, but AFAIK the most recent releases of Cockroach/Mongo/Cassandra/Fauna/etc have never fully passed a Jepsen test. - Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput. - A strongly consistent store you can actually use in anger and that you don't have to baby. We've never managed to actually break a FoundationDB cluster, the worst case scenario we've ever seen is that we slam it with too many writes/reads and it starts to apply backpressure by increasing latency. - Write/Read throughput scales ~linearly as nodes are added. - Built in backup/restore from blob storage. - Strongly consistent disaster recovery. This one is amazing and its hard to go back to other databases that don't support it, but basically you can run 2 FDB clusters in DR mode where one is the leader (accepting writes + reads) and the other is passively replicating from it. You can then fail over from the primary FDB to the secondary *consistently* in such a way that the two clusters coordinate to lock the primary and then unlock the secondary so that your application is guaranteed to not see any inconsistency or data loss with no participation from the application required (except teaching it how to talk to 2 different FDB clusters at once). Basically its lacking some niceties for building traditional applications on top of it, but it is an amazing primitive for building rock solid distributed systems and will almost never let you down once you've developed some operational experience with it. Its the closest thing to "magic" that I've ever worked with in the distributed storage space.

sigstoat 5y ago
to the folks asking why you'd use FDB, i'd highlight the main points as: > Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. the transaction guarantees on FDB are just the best there is in any product you can get for free. not to fault the rest, FDB just went past what ~98% of folks will ever need. but if you're in that last 2%, god speed trying to get anything else to offer you the same semantics. > A strongly consistent store you can actually use in anger and that you don't have to baby. i've seen some folks trying to run a cluster without proper monitoring, who were nearly driving the poor thing into the ground, with hosts dying, and having so much pressure put on them they failed out, but it never lost anything. i don't think it ever even became unavailable. > Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput eh, in defense of everything else, FDB is a key value store, and the keys are byte strings, so the situation is a bit easier for them.