10 ms·

FoundationDB: A distributed, unbundled, transactional key value store [pdf]

283 points by wwilson 5y ago

ryanworl 5y ago
Two quotes from the paper that I think will motivate people to read it: "Rigorous correctness testing via simulation makes FDB extremely reliable. In the past several years, CloudKit [59] has deployed FDB for more than 0.5M disk years without a single data corruption event. Additionally, we constantly perform data consistency checks by comparing replicas of data records and making sure they are the same. To this date, no inconsistent data replicas have ever been found in our production clusters." "For example, early versions of FDB depended on Apache Zookeeper for coordination, which was deleted after real-world fault injection found two independent bugs in Zookeeper (circa 2010) and was replaced by a de novo Paxos implementation written in Flow. No production bugs have ever been reported since."
- jeffbee 5y ago
  Ehhhh, doesn't align with my experience. I think FDB is actually really poorly tested. When I was evaluating it for replacement of the metadata key-value store at a major, public web services company we found that injecting faults into virtual NVMe devices on individual replicas would cause corrupt results returned to clients. We also found that it would just crash-loop on Linux systems with huge pages, because although someone from the project had written a huge-page-aware C++ allocator "for performance", evidently nobody had ever actually tried to use it, including the author. It's also really, really weird that their non-scalable architecture hits a brick wall at 25 machines. Ignoring the correctness flaws, it only works if you can either design around that limit by sharding, and never off cross-shard transactions, or if you can assure yourself that your use case will never outgrow half a rack of equipment.
  - bpicolo 5y ago
    What were the strong contenders?
  - rbranson 5y ago
    Were there other distributed databases that did pass the fault injection testing?
    - jeffbee 5y ago
      There weren't any, which is why that particular shop elected to roll their own distributed system on top of rocks. In general I think people who think they want to do FoundationDB owe themselves a serious contemplation of the cost/benefit of using Cloud Spanner instead. Obviously you cannot do your own fault injection testing of Spanner, but it does have end-to-end checksums.
      - sigstoat 5y ago
        > There weren't any, which is why that particular shop elected to roll their own distributed system on top of rocks. that's nuts. rocks could've been added as a storage engine to fdb far more easily.
        ryanworl 5y ago
        This is currently in progress right now. https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea21aeb94972fc79cd7b5edbab/fdbserver/KeyValueStoreRocksDB.actor.cpp https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea2...
        jeffbee 5y ago
        For the record, I said the same thing. But it's a management problem because on the one hand you have a known open project with demonstrable flaws, and on the other you have your own in-house developers and you will tend to discount the bugs they haven't written yet. But, also for the same record, thinking you can implement a reliable, globally-replicated key-value store on top of FoundationDB that is cheaper and better than Cloud Spanner may be evidence of the same cognitive bias.
        sigstoat 5y ago
        > But, also for the same record, thinking you can implement a reliable, globally-replicated key-value store on top of FoundationDB that is cheaper and better than Cloud Spanner may be evidence of the same cognitive bias. man, good thing nobody made any claim like that.
  - fnordpiglet 5y ago
    Can you fix a point in time? Software evolves and I think a point I saw is that it wasn’t well tested then they changed once production workloads told them it needs to change.
- sandinmyjoints 5y ago
  What is the Flow referred to here?
  - oconnor663 5y ago
    It's an async/await framework for C++. I'm not sure what the best source on this is, but here's a discussion: https://forums.foundationdb.org/t/why-was-flow-developed/1711/3 https://forums.foundationdb.org/t/why-was-flow-developed/171... My understanding is that FDB relies heavily on deterministic simulations for testing, and that their async/await model is a big part of how they make sure they cover different possible interleavings in a deterministic way.
- fizwhiz 5y ago
  > de novo Paxos implementation written in Flow That's... brave. Flow is a DSL built on top of C++?
  - alistairw 5y ago
    Yeah it's their own language on top of c++ to help them with testing distributed systems with deterministic simulation. Their talk from a while ago about it was something that really blew me away at the time [0] [0] https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
- jorangreef 5y ago
  Thanks for the quotes, I've been wanting to read this paper for some time. Great to see they went through the consensus literature and made a decision to go with Active Disk Paxos, instead of stopping short and not fully understanding the consensus they're building on. The consensus and replication protocol is such a huge part of building a distributed database.
jtdev 5y ago
I’d love to see a good primer on data models and scenarios that are well suited to FDB.
- selljamhere 5y ago
  Their docs might be a good place to start. https://apple.github.io/foundationdb/developer-guide.html#data-model https://apple.github.io/foundationdb/developer-guide.html#da...
- sigstoat 5y ago
  this is limited by your creativity and willingness to make tradeoffs. the only really general statement i can think of is that the "larger"/"longer" your transactions are, the harder a time you'll have getting it to cooperate with FDB. "small"/"fast" transactions will be easier to fit into its model. (to likely replies: this isn't an absolute, see all the quotes. yes things like redwood will alleviate some of this, but not all.)
  - vvern 5y ago
    IIRC fdb is fully optimistic concurrency control. It doesn't do any locking. If you have workloads which are highly contended, you'll need to do something in the layer above to coordinate. Otherwise, performance will be unbearable. This may be out-dated, please let me know if the story has evolved here.
    - sigstoat 5y ago
      if your transactions are conflicting heavily with each other, yeah, you'll have a bad time. and if everything synchronizes on some small set of keys, you'll have a really bad time. monitoring the transaction conflict rate on your cluster is important.
georgelyon 5y ago
FDB is an awesome and unique piece of software (I attribute quite a bit of Snowflake's success to FDB). I've also had the pleasure of meeting some folks from the original team and they are true engineers. Does anyone know if/when Redwood (the new storage engine) has landed / will land?
- victor106 5y ago
  > I attribute quite a bit of Snowflake's success to FDB How so?
  - foobiekr 5y ago
    Snowflake is the biggest deployment of fdb in the world after iCloud.
- kendallgclark 5y ago
  Founders are building a distributed systems simulation product now called Antithesis. My data fabric startup, Stardog, is a happy Antithesis early adopter customer. It’s helping us reproduce and fix non-deterministic bugs deterministically. Good stuff.
  - maxmcd 5y ago
    Fascinating: https://antithesis.com/ https://antithesis.com/
jorangreef 5y ago
Markus Pilman from Snowflake did an awesome talk on FoundationDB's testing at CMU's Quarantine Tech Talks (2020), How I Learned to Stop Worrying and Trust the Database: https://www.youtube.com/watch?v=OJb8A6h9jQQ https://www.youtube.com/watch?v=OJb8A6h9jQQ
- sgk284 5y ago
  Here's another excellent talk at Strangeloop on FoundationDB's simulation testing by Will Wilson in 2014: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
e12e 5y ago
This seems like a good place to ask - are there any new and exiting FOSS "application" worth checking out? I recall from the initial publication of the source - there was references to a great sql layer? I don't know if a FOSS work-a-like ever materialized? Other things I'd hoped for was a network filesystem/blob layer, like maybe s3/nfs/webdavfs compatible? What are people building on top of foundationdb today? Ed: i suppose various document/db applications - like IMAP might be a good fit too?
- sigstoat 5y ago
  peruse the fdb forum. they produce document and record layers now. there are community layers of varying quality for a network block device, a filesystem, and a few other things.
- agency 5y ago
  I’m curious about this as well. Is anyone working on building text search on top of FDB? It’s kind of astounding to me that last time I checked Elasticsearch was still essentially the only game in town.
  - jFriedensreich 5y ago
    its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for the forseeable future.
    - jng 5y ago
      Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...
deleted 5y ago
[deleted]
rubyn00bie 5y ago
Here's one of my favorite articles on FoundationDB, where it (FDB) passes Jepsen first try: https://web.archive.org/web/20150312112556/http://blog.foundationdb.com/foundationdb-vs-the-new-jepsen-and-why-you-should-care https://web.archive.org/web/20150312112556/http://blog.found... > I ran FoundationDB Key-Value Store through every nemesis in Jepsen - including those that found failures in other databases - and FoundationDB passed all of them with flying colors. FoundationDB is one of the coolest pieces of technology I've used in the past decade. The tuple keyspace is incredibly useful, so are the multi-key transactions. I've physically killed the power on an FDB node and FDB cluster; multiple times (heh, home servers)... and every time the cluster or node just comes back.
- gregwebs 5y ago
  That's great that you are doing your own resiliency testing. Having someone other than those officially on the Jepsen project run the Jepsen test is a good start. However, many databases have claimed to run the Jepsen tests themselves and pass, but when there is an actual paid engagement for a distributed database there are always issues that are found. That's generally true even for unpaid official runs as well although Zookeeper did pass existing tests. Every database is different and the paid engagement will design specific tests designed to break the database in question.
  - deleted 5y ago
    [deleted]
- kendallgclark 5y ago
  This was the FDB team’s stock demo in the early days. It’s a killer move.
AtlasBarfed 5y ago
They got acquihired by apple, didn't they? Was. Fdb ever oss'd? Is it CP or AP? Comments seem to imply AP
- ssgao 5y ago
  FoundationDB is Apache 2.0 https://github.com/apple/foundationdb/blob/master/LICENSE https://github.com/apple/foundationdb/blob/master/LICENSE It is CP per https://apple.github.io/foundationdb/cap-theorem.html https://apple.github.io/foundationdb/cap-theorem.html
- kendallgclark 5y ago
  It wasn’t an acquihire. Apple paid a lot of $$ for FDB.
polskibus 5y ago
What is the backup / restore story in FoundationDB? How does it compare to postgresql?
- ex3ndr 5y ago
  Much much better. Single line backup/restore and Disaster Recovery mode that syncs second DC and able too switch on the fly with barely any configs (except one file).
jbverschoor 5y ago
It's unfortunate that they went silent for years after the Apple acquisition. That period was key for database adoption. I have the feeling everybody kind of settled for pgsql.
- eloff 5y ago
  Those are two completely non overlapping use cases. If you can use pgsql for your problem, you have no business trying to use a distributed key value store instead. That would be at least as dumb as driving screws with a hammer.
  - cwp 5y ago
    Yeah, but there are quite a few efforts out there to extend PG into a distributed DB of one flavor or another. Some examples are YugabyteDB, CockroachDB, Aurora and Citus. It's a reasonable approach, but it's also reasonable to come at it from the other direction - build a SQL engine on top of a solid distributed key-value store. Contrafactuals are always dicey, but FDB vanishing behind the Apple wall of silence sure didn't help.
    - eloff 5y ago
      > Some examples are YugabyteDB, CockroachDB, Aurora and Citus. Of those the first two are not PG, they just share the wire protocol and try to be compatible at a SQL level. Aurora is not really distributed, it's replicated for availability and durability six ways at the block storage level. Citus is distributed as I understand it though. > but it's also reasonable to come at it from the other direction - build a SQL engine on top of a solid distributed key-value store. Sure, that's possible. I'm not talking about the wisdom of building your own relational database as the end goal, just that a distributed key value database and a SQL database don't have overlapping problem sets.
      - cwp 5y ago
        Sure quibble about the details. The point is, these are all attempts to make PG more scalable. Around the time these projects got started Foundation looked like abandonware. If it hadn't, it's possible that "how can we have really scalable SQL databases?" might have had Foundation as part of the answer.
monstrado 5y ago
Have nothing but praise for FoundationDB. It has been by far the most rock solid distributed database I have ever had the pleasure of using. I used to manage HBase clusters, and the fact that I have never once had to worry about manually splitting "regions" is such a boon for administration...let alone JVM GC tuning. We run several FDB clusters using 3-DC replication and have never once lost data. I remember when we wanted to replace all of the FDB hardware (one cluster) in AWS, and so we just doubled the cluster size, waited for data shuffling to calm down, and just started axing the original hardware. We did this all while performing over 100K production TPS. One thing that makes the above seamless for all existing connections is that clients automatically update their "cluster file" in the event that new coordinators join or are reassigned. That alone is amazing...as you don't have to track down every single client and change / re-roll with new connection parameters. Anyway, I talk this database up every chance I get. Keep up the awesome work. - A very happy user.
- maxpert 5y ago
  How would it compare to say something like hosted Redis, or if you wanna be more fancy ElasticSearch. I have been looking into FDB for pretty long time and have been looking for a perfect opportunity to use it. Would be helpful if you can describe your usage scenario (kind of data you are storing).
  - monstrado 5y ago
    The key takeaways of FoundationDB is that it is a strongly consistent KeyValue store that preserves ordering (lexicographical). Although you might consider FDB's APIs to be quite primitive (get/set/scan), the payoff is how it seamlessly handles multi-key transactions without requiring you to write or manage some client side two-phase-commit process. Given these primitives, you or other engineers can write higher level APIs on top (e.g. sql, search, etc). In fact, we make extensive use of their RecordLayer [1] library which provides a strongly consistent schema based write process using Protobuf. This includes on-write-consistent indexes. Apple has also open-sourced a MongoDB API [2] compliant interface that allows you to get all the consistency guarantees from FDB, but with an API interaction of MongoDB. The beauty of FDB is that the primitives are done in such a rock solid fashion, you can write higher level APIs without having to worry about the really hard stuff (transactions, failures, config errors, testing, simulation, etc). Another example is that CouchDB is switching their back-end to use FDB for their 4.0 release. Given that the database sorts data lexicographical...to us, it became a natural fit for an online (always-mutable) time-series database. If you want more insight in how we use it, I go over it in some detail towards the end of my keynote [3] from last August. [1] https://www.foundationdb.org/blog/announcing-record-layer/ https://www.foundationdb.org/blog/announcing-record-layer/ [2] https://github.com/FoundationDB/fdb-document-layer https://github.com/FoundationDB/fdb-document-layer [3] https://www.youtube.com/watch?v=93b--lTq2ng https://www.youtube.com/watch?v=93b--lTq2ng
eyelovewe 5y ago
CouchDB 4 is built upon Foundation FWIW
- jbverschoor 5y ago
  Didn’t know, very happy to hear
jwr 5y ago
I just implemented a database with changefeeds using FoundationDB (in Clojure), to eventually replace RethinkDB in my system. Very impressed so far.
- adamfeldman 5y ago
  That’s awesome! I’m interested specifically in using FDB with Clojure. Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).
  - krn 5y ago
    Also, would it make any sense to use FoundationDB instead of Kafka as Document Store for Crux?
    - adamfeldman 5y ago
      This is actively being discussed in #crux on Clojurians Slack [1]. It ought to show up within the archive [2] in a couple of days. Should already be in the Clojurians Zulip mirror as well. [1]: https://clojurians.slack.com/archives/CG3AM2F7V/p1623102513147400 https://clojurians.slack.com/archives/CG3AM2F7V/p16231025131... [2]: https://clojurians-log.clojureverse.org/crux https://clojurians-log.clojureverse.org/crux
  - jwr 5y ago
    Yes, I did. But I had specific requirements, and the main one was that I need a fully distributed database, where one of the nodes can disappear for any reason at any time and things would just continue as if nothing happened.
    - refset 5y ago
      Hi, I work on the Crux team. I think "fully distributed" has a few possible meanings, but is it essentially a case of wanting something with a dead-simple clustering story? Or is it more about multi-region distribution & availability? Whilst Kafka itself is almost certainly not as simple to operate as FDB (although I can't speak from experience), it does in turn provide Crux with dead-simple clustering, because each Crux node acts as an isolated replica. FDB could be used equally instead of Kafka, but the maximum write throughput would then be somewhat lower. The only key part of the story Crux doesn't currently supply out-of-the-box is a load-balancing layer atop a cluster of nodes. I expect the real drawback of Crux's current design, by comparison, is that each node is an ~expensive full replica, whereas your system (built on FDB) will benefit from fully sharded indexes running across a cluster of (smaller) machines. The main trade-off then is the low-latency query performance of a fully local KV store.
twoodfin 5y ago
Did they ever implement a SQL layer? They seemed like one of the only NoSQL products with the architecture to make it plausible to do so.
- funny_falcon 5y ago
  iirc, they did. But it wasn't opensourced.
- mping 5y ago
  There was an SQL layer but performance was sub-par IIRC. There was also a blog post somewhere explaining why it's probably not a good idea to build an SQL layer on top of a KV store, devil in details, etc. Edit: found it https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fas... Not sure what to think of it, I'm not a DB expert by any means but the post sounds plausible enough and the SQL layer is discontinued AFAIK. I guess with each new abstraction layer you leave some perf on the table.
  - voidmain 5y ago
    It's not a very good analysis. The only reason there isn't a good OLTP SQL engine on top of FDB is that no one has invested the considerable resources and expertise to build one. The FDB SQL Layer was on its way, but Apple prioritized other things. Snowflake built a world class analytic SQL database on FDB and the door is still open for someone to do OLTP. There might even be room for two different designs (one more like Akiban/SQL Layer or F1, competing with Spanner, and one more like Aurora, with a more familiar performance envelope but slightly less scalability)
jFriedensreich 5y ago
I am pretty sure that the new cloudant transaction/storage engine is also based on foundationDB, which powers a lot of things behind the scenes at ibm. And couchdb 4 with foundationDB storage engine is hopefully not too far out either. Lets see how long this whole transition takes, but i am still hopeful that the mindshare and motivation of apple, snowflake, ibm and apache community will lead to something great.
Meai 5y ago
Personally I don't understand how you can call a database robust if it can't scale down nodes after you scaled them up once. What am I supposed to do if I ever deploy to 50 nodes and then it turns out that I only need 5. Shut the business down? Pay to run database servers forever that I don't even need anymore? Also the database configuration has a lot of gotchas and is very opaque. You might be waiting for 30sec for your CLI to connect to your localhost cluster of two processes and you have no idea what is happening or why it is taking that long. It just never felt so safe and robust as people claim it to be. I don't know, these were just my findings on the brief tests I did with it. Also you better get familiar with a whole bunch of hidden "knobs" that are apparently configurable and very important somewhere and then get printed out into xml logs but of course there is no log viewer so you have to write your own. Maybe this isn't a problem for large companies but I'm providing feedback as a single user here. I also don't understand how people can praise the c++ DSL. They should rewrite that into standard c++ coroutines as soon as possible so their entire build and dev environment isn't so hard to understand. As a user of open source software I generally like to be able to debug through the projects I use and figure out problems I have. It's much harder when a project uses their own custom language. I certainly tried to set it all up correctly but there always seemed to be some problems in regards to Intellisense within the IDE.
- __float 5y ago
  https://apple.github.io/foundationdb/administration.html#removing-machines-from-a-cluster https://apple.github.io/foundationdb/administration.html#rem... What is this process describing, and how does it differ from what you were trying to do?
  - Meai 5y ago
    They are using the term "machine" and "process" in multiple conflicting meanings I think. I mean maybe they improved this since I used it 1.5 years ago but I kind of doubt it. If I remember correctly there is a definite problem if you remove one of the processes that end up guaranteeing your configured redundancy mode. So then suddenly your entire cluster is inoperable. Yes really, I think it doesn't even properly respond to cli commands anymore and shows nodes as simply missing. Oh and suddenly those long wait times for your cli commands are really starting to bother you... I don't really want to talk more about it because it's been a while but I just wanted to make the point that the user experience was quite bad.
strangattractor 5y ago
With little or no admin and monitoring tools.
grouphugs 5y ago
was slightly excited about this, but the whole apple thing just ruined that
davgoldin 5y ago
Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HDFS, MinIO and some others, and eventually decided on a custom FDB solution. It's a breeze to setup, and seems to eclipse others in performance [0] and reliability - Kyle (aphyr) the author of Jepsen series on distributed systems correctness, said: "haven't tested foundation in part because their testing appears to be waaaay more rigorous than mine." [1] The way we use FDB, if anyone is interested, is we simply split files into small chunks (per FDB data design recommendations), and store all file's & folder's meta data in FDB such as byte count, create/access/write times, permissions, and a lot more. Folders are handled by the builtin Directory layer [2]. [0] https://apple.github.io/foundationdb/performance.html https://apple.github.io/foundationdb/performance.html [1] https://web.archive.org/web/20150312112552/http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-jepsen https://web.archive.org/web/20150312112552/http://blog.found... [2] https://forums.foundationdb.org/t/whats-the-purpose-of-the-directory-layer/677 https://forums.foundationdb.org/t/whats-the-purpose-of-the-d...
- tpetry 5y ago
  That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.
  - davgoldin 5y ago
    FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte count, you could use FDB's atomic increment operation. Here's a good starting point (not mine): https://forums.foundationdb.org/t/object-store-on-foundationdb/387/3 https://forums.foundationdb.org/t/object-store-on-foundation... I'll be happy to answer specific question if you got any.
gigatexal 5y ago
Didn't couchbase move to FDB for their underlying engine?
acjohnson55 5y ago
Am I right that this is like a distributed form of something like LevelDB or RocksDB, which would be the underlying storage engine for a full database product? And/or would it be comparable to DynamoDB?
dinedal 5y ago
Does anyone know of the"sqlite connector" mentioned in this post ? https://opensourceconnections.com/blog/2013/05/06/does-foundationdb-beat-the-cap-conjecture-hackathon-on-friday/ https://opensourceconnections.com/blog/2013/05/06/does-found... It would be really cool to find it, if it's still out there.
one2three4 5y ago
>> In its newest release, CouchDB [2] (arguably the firstNoSQL system) is being re-built as a layer on top of FoundationDB. That is impressive. Like a framework for implementing noSQL DBs.
- __float 5y ago
  This is similar to WiredTiger, and its role in MongoDB.
z77dj3kl 5y ago
This is a really good document: https://apple.github.io/foundationdb/data-modeling.html https://apple.github.io/foundationdb/data-modeling.html I have been studying these key-value stores with efficient range iteration lately (such as LevelDB, RocksDB, BigTable, FoundationDB, etc). This is a great reference on how to make such a simple abstraction do a lot of useful things.

283 points by wwilson 5y ago

Two quotes from the paper that I think will motivate people to read it: "Rigorous correctness testing via simulation makes FDB extremely reliable. In the past several years, CloudKit [59] has deployed FDB for more than 0.5M disk years without a single data corruption event. Additionally, we constantly perform data consistency checks by comparing replicas of data records and making sure they are the same. To this date, no inconsistent data replicas have ever been found in our production clusters." "For example, early versions of FDB depended on Apache Zookeeper for coordination, which was deleted after real-world fault injection found two independent bugs in Zookeeper (circa 2010) and was replaced by a de novo Paxos implementation written in Flow. No production bugs have ever been reported since."

jeffbee 5y ago
Ehhhh, doesn't align with my experience. I think FDB is actually really poorly tested. When I was evaluating it for replacement of the metadata key-value store at a major, public web services company we found that injecting faults into virtual NVMe devices on individual replicas would cause corrupt results returned to clients. We also found that it would just crash-loop on Linux systems with huge pages, because although someone from the project had written a huge-page-aware C++ allocator "for performance", evidently nobody had ever actually tried to use it, including the author. It's also really, really weird that their non-scalable architecture hits a brick wall at 25 machines. Ignoring the correctness flaws, it only works if you can either design around that limit by sharding, and never off cross-shard transactions, or if you can assure yourself that your use case will never outgrow half a rack of equipment.
- bpicolo 5y ago
  What were the strong contenders?
- rbranson 5y ago
  Were there other distributed databases that did pass the fault injection testing?
  - jeffbee 5y ago
    There weren't any, which is why that particular shop elected to roll their own distributed system on top of rocks. In general I think people who think they want to do FoundationDB owe themselves a serious contemplation of the cost/benefit of using Cloud Spanner instead. Obviously you cannot do your own fault injection testing of Spanner, but it does have end-to-end checksums.
    - sigstoat 5y ago
      > There weren't any, which is why that particular shop elected to roll their own distributed system on top of rocks. that's nuts. rocks could've been added as a storage engine to fdb far more easily.
      - ryanworl 5y ago
        This is currently in progress right now. https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea21aeb94972fc79cd7b5edbab/fdbserver/KeyValueStoreRocksDB.actor.cpp https://github.com/apple/foundationdb/blob/e7d7b39f12afa8ea2...
      - jeffbee 5y ago
        For the record, I said the same thing. But it's a management problem because on the one hand you have a known open project with demonstrable flaws, and on the other you have your own in-house developers and you will tend to discount the bugs they haven't written yet. But, also for the same record, thinking you can implement a reliable, globally-replicated key-value store on top of FoundationDB that is cheaper and better than Cloud Spanner may be evidence of the same cognitive bias.
        sigstoat 5y ago
        > But, also for the same record, thinking you can implement a reliable, globally-replicated key-value store on top of FoundationDB that is cheaper and better than Cloud Spanner may be evidence of the same cognitive bias. man, good thing nobody made any claim like that.
- fnordpiglet 5y ago
  Can you fix a point in time? Software evolves and I think a point I saw is that it wasn’t well tested then they changed once production workloads told them it needs to change.
sandinmyjoints 5y ago
What is the Flow referred to here?
- oconnor663 5y ago
  It's an async/await framework for C++. I'm not sure what the best source on this is, but here's a discussion: https://forums.foundationdb.org/t/why-was-flow-developed/1711/3 https://forums.foundationdb.org/t/why-was-flow-developed/171... My understanding is that FDB relies heavily on deterministic simulations for testing, and that their async/await model is a big part of how they make sure they cover different possible interleavings in a deterministic way.
fizwhiz 5y ago
> de novo Paxos implementation written in Flow That's... brave. Flow is a DSL built on top of C++?
- alistairw 5y ago
  Yeah it's their own language on top of c++ to help them with testing distributed systems with deterministic simulation. Their talk from a while ago about it was something that really blew me away at the time [0] [0] https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
jorangreef 5y ago
Thanks for the quotes, I've been wanting to read this paper for some time. Great to see they went through the consensus literature and made a decision to go with Active Disk Paxos, instead of stopping short and not fully understanding the consensus they're building on. The consensus and replication protocol is such a huge part of building a distributed database.

jtdev 5y ago

I’d love to see a good primer on data models and scenarios that are well suited to FDB.

selljamhere 5y ago
Their docs might be a good place to start. https://apple.github.io/foundationdb/developer-guide.html#data-model https://apple.github.io/foundationdb/developer-guide.html#da...
sigstoat 5y ago
this is limited by your creativity and willingness to make tradeoffs. the only really general statement i can think of is that the "larger"/"longer" your transactions are, the harder a time you'll have getting it to cooperate with FDB. "small"/"fast" transactions will be easier to fit into its model. (to likely replies: this isn't an absolute, see all the quotes. yes things like redwood will alleviate some of this, but not all.)
- vvern 5y ago
  IIRC fdb is fully optimistic concurrency control. It doesn't do any locking. If you have workloads which are highly contended, you'll need to do something in the layer above to coordinate. Otherwise, performance will be unbearable. This may be out-dated, please let me know if the story has evolved here.
  - sigstoat 5y ago
    if your transactions are conflicting heavily with each other, yeah, you'll have a bad time. and if everything synchronizes on some small set of keys, you'll have a really bad time. monitoring the transaction conflict rate on your cluster is important.

georgelyon 5y ago

FDB is an awesome and unique piece of software (I attribute quite a bit of Snowflake's success to FDB). I've also had the pleasure of meeting some folks from the original team and they are true engineers. Does anyone know if/when Redwood (the new storage engine) has landed / will land?

victor106 5y ago
> I attribute quite a bit of Snowflake's success to FDB How so?
- foobiekr 5y ago
  Snowflake is the biggest deployment of fdb in the world after iCloud.
kendallgclark 5y ago
Founders are building a distributed systems simulation product now called Antithesis. My data fabric startup, Stardog, is a happy Antithesis early adopter customer. It’s helping us reproduce and fix non-deterministic bugs deterministically. Good stuff.
- maxmcd 5y ago
  Fascinating: https://antithesis.com/ https://antithesis.com/

jorangreef 5y ago

Markus Pilman from Snowflake did an awesome talk on FoundationDB's testing at CMU's Quarantine Tech Talks (2020), How I Learned to Stop Worrying and Trust the Database: https://www.youtube.com/watch?v=OJb8A6h9jQQ https://www.youtube.com/watch?v=OJb8A6h9jQQ

sgk284 5y ago
Here's another excellent talk at Strangeloop on FoundationDB's simulation testing by Will Wilson in 2014: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc

e12e 5y ago

This seems like a good place to ask - are there any new and exiting FOSS "application" worth checking out? I recall from the initial publication of the source - there was references to a great sql layer? I don't know if a FOSS work-a-like ever materialized? Other things I'd hoped for was a network filesystem/blob layer, like maybe s3/nfs/webdavfs compatible? What are people building on top of foundationdb today? Ed: i suppose various document/db applications - like IMAP might be a good fit too?

sigstoat 5y ago
peruse the fdb forum. they produce document and record layers now. there are community layers of varying quality for a network block device, a filesystem, and a few other things.
agency 5y ago
I’m curious about this as well. Is anyone working on building text search on top of FDB? It’s kind of astounding to me that last time I checked Elasticsearch was still essentially the only game in town.
- jFriedensreich 5y ago
  its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for the forseeable future.
  - jng 5y ago
    Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

deleted 5y ago

[deleted]

rubyn00bie 5y ago

Here's one of my favorite articles on FoundationDB, where it (FDB) passes Jepsen first try: https://web.archive.org/web/20150312112556/http://blog.foundationdb.com/foundationdb-vs-the-new-jepsen-and-why-you-should-care https://web.archive.org/web/20150312112556/http://blog.found... > I ran FoundationDB Key-Value Store through every nemesis in Jepsen - including those that found failures in other databases - and FoundationDB passed all of them with flying colors. FoundationDB is one of the coolest pieces of technology I've used in the past decade. The tuple keyspace is incredibly useful, so are the multi-key transactions. I've physically killed the power on an FDB node and FDB cluster; multiple times (heh, home servers)... and every time the cluster or node just comes back.

gregwebs 5y ago
That's great that you are doing your own resiliency testing. Having someone other than those officially on the Jepsen project run the Jepsen test is a good start. However, many databases have claimed to run the Jepsen tests themselves and pass, but when there is an actual paid engagement for a distributed database there are always issues that are found. That's generally true even for unpaid official runs as well although Zookeeper did pass existing tests. Every database is different and the paid engagement will design specific tests designed to break the database in question.
- deleted 5y ago
  [deleted]
kendallgclark 5y ago
This was the FDB team’s stock demo in the early days. It’s a killer move.

AtlasBarfed 5y ago

They got acquihired by apple, didn't they? Was. Fdb ever oss'd? Is it CP or AP? Comments seem to imply AP

ssgao 5y ago
FoundationDB is Apache 2.0 https://github.com/apple/foundationdb/blob/master/LICENSE https://github.com/apple/foundationdb/blob/master/LICENSE It is CP per https://apple.github.io/foundationdb/cap-theorem.html https://apple.github.io/foundationdb/cap-theorem.html
kendallgclark 5y ago
It wasn’t an acquihire. Apple paid a lot of $$ for FDB.

polskibus 5y ago

What is the backup / restore story in FoundationDB? How does it compare to postgresql?

ex3ndr 5y ago
Much much better. Single line backup/restore and Disaster Recovery mode that syncs second DC and able too switch on the fly with barely any configs (except one file).

jbverschoor 5y ago

It's unfortunate that they went silent for years after the Apple acquisition. That period was key for database adoption. I have the feeling everybody kind of settled for pgsql.

eloff 5y ago
Those are two completely non overlapping use cases. If you can use pgsql for your problem, you have no business trying to use a distributed key value store instead. That would be at least as dumb as driving screws with a hammer.
- cwp 5y ago
  Yeah, but there are quite a few efforts out there to extend PG into a distributed DB of one flavor or another. Some examples are YugabyteDB, CockroachDB, Aurora and Citus. It's a reasonable approach, but it's also reasonable to come at it from the other direction - build a SQL engine on top of a solid distributed key-value store. Contrafactuals are always dicey, but FDB vanishing behind the Apple wall of silence sure didn't help.
  - eloff 5y ago
    > Some examples are YugabyteDB, CockroachDB, Aurora and Citus. Of those the first two are not PG, they just share the wire protocol and try to be compatible at a SQL level. Aurora is not really distributed, it's replicated for availability and durability six ways at the block storage level. Citus is distributed as I understand it though. > but it's also reasonable to come at it from the other direction - build a SQL engine on top of a solid distributed key-value store. Sure, that's possible. I'm not talking about the wisdom of building your own relational database as the end goal, just that a distributed key value database and a SQL database don't have overlapping problem sets.
    - cwp 5y ago
      Sure quibble about the details. The point is, these are all attempts to make PG more scalable. Around the time these projects got started Foundation looked like abandonware. If it hadn't, it's possible that "how can we have really scalable SQL databases?" might have had Foundation as part of the answer.

monstrado 5y ago

Have nothing but praise for FoundationDB. It has been by far the most rock solid distributed database I have ever had the pleasure of using. I used to manage HBase clusters, and the fact that I have never once had to worry about manually splitting "regions" is such a boon for administration...let alone JVM GC tuning. We run several FDB clusters using 3-DC replication and have never once lost data. I remember when we wanted to replace all of the FDB hardware (one cluster) in AWS, and so we just doubled the cluster size, waited for data shuffling to calm down, and just started axing the original hardware. We did this all while performing over 100K production TPS. One thing that makes the above seamless for all existing connections is that clients automatically update their "cluster file" in the event that new coordinators join or are reassigned. That alone is amazing...as you don't have to track down every single client and change / re-roll with new connection parameters. Anyway, I talk this database up every chance I get. Keep up the awesome work. - A very happy user.

maxpert 5y ago
How would it compare to say something like hosted Redis, or if you wanna be more fancy ElasticSearch. I have been looking into FDB for pretty long time and have been looking for a perfect opportunity to use it. Would be helpful if you can describe your usage scenario (kind of data you are storing).
- monstrado 5y ago
  The key takeaways of FoundationDB is that it is a strongly consistent KeyValue store that preserves ordering (lexicographical). Although you might consider FDB's APIs to be quite primitive (get/set/scan), the payoff is how it seamlessly handles multi-key transactions without requiring you to write or manage some client side two-phase-commit process. Given these primitives, you or other engineers can write higher level APIs on top (e.g. sql, search, etc). In fact, we make extensive use of their RecordLayer [1] library which provides a strongly consistent schema based write process using Protobuf. This includes on-write-consistent indexes. Apple has also open-sourced a MongoDB API [2] compliant interface that allows you to get all the consistency guarantees from FDB, but with an API interaction of MongoDB. The beauty of FDB is that the primitives are done in such a rock solid fashion, you can write higher level APIs without having to worry about the really hard stuff (transactions, failures, config errors, testing, simulation, etc). Another example is that CouchDB is switching their back-end to use FDB for their 4.0 release. Given that the database sorts data lexicographical...to us, it became a natural fit for an online (always-mutable) time-series database. If you want more insight in how we use it, I go over it in some detail towards the end of my keynote [3] from last August. [1] https://www.foundationdb.org/blog/announcing-record-layer/ https://www.foundationdb.org/blog/announcing-record-layer/ [2] https://github.com/FoundationDB/fdb-document-layer https://github.com/FoundationDB/fdb-document-layer [3] https://www.youtube.com/watch?v=93b--lTq2ng https://www.youtube.com/watch?v=93b--lTq2ng

eyelovewe 5y ago

CouchDB 4 is built upon Foundation FWIW

jbverschoor 5y ago
Didn’t know, very happy to hear

jwr 5y ago

I just implemented a database with changefeeds using FoundationDB (in Clojure), to eventually replace RethinkDB in my system. Very impressed so far.

adamfeldman 5y ago
That’s awesome! I’m interested specifically in using FDB with Clojure. Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).
- krn 5y ago
  Also, would it make any sense to use FoundationDB instead of Kafka as Document Store for Crux?
  - adamfeldman 5y ago
    This is actively being discussed in #crux on Clojurians Slack [1]. It ought to show up within the archive [2] in a couple of days. Should already be in the Clojurians Zulip mirror as well. [1]: https://clojurians.slack.com/archives/CG3AM2F7V/p1623102513147400 https://clojurians.slack.com/archives/CG3AM2F7V/p16231025131... [2]: https://clojurians-log.clojureverse.org/crux https://clojurians-log.clojureverse.org/crux
- jwr 5y ago
  Yes, I did. But I had specific requirements, and the main one was that I need a fully distributed database, where one of the nodes can disappear for any reason at any time and things would just continue as if nothing happened.
  - refset 5y ago
    Hi, I work on the Crux team. I think "fully distributed" has a few possible meanings, but is it essentially a case of wanting something with a dead-simple clustering story? Or is it more about multi-region distribution & availability? Whilst Kafka itself is almost certainly not as simple to operate as FDB (although I can't speak from experience), it does in turn provide Crux with dead-simple clustering, because each Crux node acts as an isolated replica. FDB could be used equally instead of Kafka, but the maximum write throughput would then be somewhat lower. The only key part of the story Crux doesn't currently supply out-of-the-box is a load-balancing layer atop a cluster of nodes. I expect the real drawback of Crux's current design, by comparison, is that each node is an ~expensive full replica, whereas your system (built on FDB) will benefit from fully sharded indexes running across a cluster of (smaller) machines. The main trade-off then is the low-latency query performance of a fully local KV store.

twoodfin 5y ago

Did they ever implement a SQL layer? They seemed like one of the only NoSQL products with the architecture to make it plausible to do so.

funny_falcon 5y ago
iirc, they did. But it wasn't opensourced.
mping 5y ago
There was an SQL layer but performance was sub-par IIRC. There was also a blog post somewhere explaining why it's probably not a good idea to build an SQL layer on top of a KV store, devil in details, etc. Edit: found it https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fas... Not sure what to think of it, I'm not a DB expert by any means but the post sounds plausible enough and the SQL layer is discontinued AFAIK. I guess with each new abstraction layer you leave some perf on the table.
- voidmain 5y ago
  It's not a very good analysis. The only reason there isn't a good OLTP SQL engine on top of FDB is that no one has invested the considerable resources and expertise to build one. The FDB SQL Layer was on its way, but Apple prioritized other things. Snowflake built a world class analytic SQL database on FDB and the door is still open for someone to do OLTP. There might even be room for two different designs (one more like Akiban/SQL Layer or F1, competing with Spanner, and one more like Aurora, with a more familiar performance envelope but slightly less scalability)

jFriedensreich 5y ago

I am pretty sure that the new cloudant transaction/storage engine is also based on foundationDB, which powers a lot of things behind the scenes at ibm. And couchdb 4 with foundationDB storage engine is hopefully not too far out either. Lets see how long this whole transition takes, but i am still hopeful that the mindshare and motivation of apple, snowflake, ibm and apache community will lead to something great.

Meai 5y ago

Personally I don't understand how you can call a database robust if it can't scale down nodes after you scaled them up once. What am I supposed to do if I ever deploy to 50 nodes and then it turns out that I only need 5. Shut the business down? Pay to run database servers forever that I don't even need anymore? Also the database configuration has a lot of gotchas and is very opaque. You might be waiting for 30sec for your CLI to connect to your localhost cluster of two processes and you have no idea what is happening or why it is taking that long. It just never felt so safe and robust as people claim it to be. I don't know, these were just my findings on the brief tests I did with it. Also you better get familiar with a whole bunch of hidden "knobs" that are apparently configurable and very important somewhere and then get printed out into xml logs but of course there is no log viewer so you have to write your own. Maybe this isn't a problem for large companies but I'm providing feedback as a single user here. I also don't understand how people can praise the c++ DSL. They should rewrite that into standard c++ coroutines as soon as possible so their entire build and dev environment isn't so hard to understand. As a user of open source software I generally like to be able to debug through the projects I use and figure out problems I have. It's much harder when a project uses their own custom language. I certainly tried to set it all up correctly but there always seemed to be some problems in regards to Intellisense within the IDE.

__float 5y ago
https://apple.github.io/foundationdb/administration.html#removing-machines-from-a-cluster https://apple.github.io/foundationdb/administration.html#rem... What is this process describing, and how does it differ from what you were trying to do?
- Meai 5y ago
  They are using the term "machine" and "process" in multiple conflicting meanings I think. I mean maybe they improved this since I used it 1.5 years ago but I kind of doubt it. If I remember correctly there is a definite problem if you remove one of the processes that end up guaranteeing your configured redundancy mode. So then suddenly your entire cluster is inoperable. Yes really, I think it doesn't even properly respond to cli commands anymore and shows nodes as simply missing. Oh and suddenly those long wait times for your cli commands are really starting to bother you... I don't really want to talk more about it because it's been a while but I just wanted to make the point that the user experience was quite bad.

strangattractor 5y ago

With little or no admin and monitoring tools.

grouphugs 5y ago

was slightly excited about this, but the whole apple thing just ruined that

davgoldin 5y ago

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HDFS, MinIO and some others, and eventually decided on a custom FDB solution. It's a breeze to setup, and seems to eclipse others in performance [0] and reliability - Kyle (aphyr) the author of Jepsen series on distributed systems correctness, said: "haven't tested foundation in part because their testing appears to be waaaay more rigorous than mine." [1] The way we use FDB, if anyone is interested, is we simply split files into small chunks (per FDB data design recommendations), and store all file's & folder's meta data in FDB such as byte count, create/access/write times, permissions, and a lot more. Folders are handled by the builtin Directory layer [2]. [0] https://apple.github.io/foundationdb/performance.html https://apple.github.io/foundationdb/performance.html [1] https://web.archive.org/web/20150312112552/http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-jepsen https://web.archive.org/web/20150312112552/http://blog.found... [2] https://forums.foundationdb.org/t/whats-the-purpose-of-the-directory-layer/677 https://forums.foundationdb.org/t/whats-the-purpose-of-the-d...

tpetry 5y ago
That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.
- davgoldin 5y ago
  FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte count, you could use FDB's atomic increment operation. Here's a good starting point (not mine): https://forums.foundationdb.org/t/object-store-on-foundationdb/387/3 https://forums.foundationdb.org/t/object-store-on-foundation... I'll be happy to answer specific question if you got any.

gigatexal 5y ago

Didn't couchbase move to FDB for their underlying engine?

acjohnson55 5y ago

Am I right that this is like a distributed form of something like LevelDB or RocksDB, which would be the underlying storage engine for a full database product? And/or would it be comparable to DynamoDB?

dinedal 5y ago

Does anyone know of the"sqlite connector" mentioned in this post ? https://opensourceconnections.com/blog/2013/05/06/does-foundationdb-beat-the-cap-conjecture-hackathon-on-friday/ https://opensourceconnections.com/blog/2013/05/06/does-found... It would be really cool to find it, if it's still out there.

one2three4 5y ago

>> In its newest release, CouchDB [2] (arguably the firstNoSQL system) is being re-built as a layer on top of FoundationDB. That is impressive. Like a framework for implementing noSQL DBs.

__float 5y ago
This is similar to WiredTiger, and its role in MongoDB.

z77dj3kl 5y ago

This is a really good document: https://apple.github.io/foundationdb/data-modeling.html https://apple.github.io/foundationdb/data-modeling.html I have been studying these key-value stores with efficient range iteration lately (such as LevelDB, RocksDB, BigTable, FoundationDB, etc). This is a great reference on how to make such a simple abstraction do a lot of useful things.