8 ms·
FoundationDB: A distributed unbundled transactional key value store
- aptxkid 5y agoI wrote a post explaining FDB as well, which got retweeted by @FoundationDB https://mobile.twitter.com/aptxk1d/status/1417869766577082377 https://mobile.twitter.com/aptxk1d/status/141786976657708237... My post focuses on the correctness, proof of the correctness of FDB’s failure recovery, some of which is missing even from the paper itself. Some of the paper authors reached out; and I corrected one issue on my blog post pointed out by one of the authors.
- noncoml 5y agoIsn't this the one that disappeared overnight? Good luck getting the people's trust again
- lokar 5y agoIt was a private company, was bought by Apple, and is now Apache 2.0 with major contributions from outside Apple.
- deleted 5y ago[deleted]
- Operyl 5y agohttps://github.com/apple/foundationdb https://github.com/apple/foundationdb https://github.com/foundationdb https://github.com/foundationdb EDIT: I forgot they were a commercial product before Apple open-sourced it after acquisition.
- deleted 5y ago[deleted]
- commoner 5y agoFoundationDB was pulled from market when Apple acquired it in 2015.[1] Apple re-released it as open source in 2018.[2] Since FoundationDB is FOSS this time around, it would be possible to fork it in case it becomes unavailable again. [1] https://news.ycombinator.com/item?id=9259986 https://news.ycombinator.com/item?id=9259986 [2] https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395
- Operyl 5y agoI completely forgot it was initially closed source, and only seriously considered it when it was opensourced in 2018. For the sake of the GP argument, though, I'd say that rebuilt an immense amount of trust in this case. My bad!
- srott 5y agoPerhaps you were referring to DabbleDB?
- jiggawatts 5y agoA vaguely similar project that might be of interest is: https://github.com/microsoft/FASTER https://github.com/microsoft/FASTER It's also an "unbundled" low-level component that one could use as the foundation for a database engine or whatever. According to Microsoft, FASTER is not just "fast", but significantly faster than even some basic in-memory data structures that ship in the .NET standard library! The downside is that it doesn't (yet) support some more advanced features like multi-server distributed mode. However, that relative simplicity may be preferred in some scenarios...
- pas 5y agoIs it a LevelDB-like log structured merge tree? Their distributed checkpointing/recovery thing (built on FASTER) is very interesting! The graph in the upper right corner of page 9 is certainly impressive: https://tli2.github.io/assets/pdf/dpr-sigmod2021.pdf https://tli2.github.io/assets/pdf/dpr-sigmod2021.pdf (it is indexed with number of virtual machines)
- SteadBytes 5y agoThe CMU Vaccination Database Talks series includes an excellent talk on FASTER from one of the authors: https://youtu.be/4Y0oNFud-VA https://youtu.be/4Y0oNFud-VA
- jiggawatts 5y agoThanks for that link.
- chalcolithic 5y agoIf anyone can use it for anything other than session storage I'd love to hear about that. Their API is so limited that I just don't understand how can I possibly use it.
- throwaway984393 5y agoIn theory, their design makes sense: separating the component pieces makes it easier to adapt to different systems/scenarios, it allows you to more succinctly map out the architecture (and hopefully simplify interfaces), you have that whole testing framework you can use on the individual pieces, etc. The problem is with implementation. The more you separate components, the more code you have to add to allow them to interoperate. The more code, the more complexity. The more complexity, the more bugs. In addition, when you continue to separate components out into new failure domains that have a higher probability of failure ("3 components running on 1 node" -> "3 components running on 3 separate nodes") you increase the chances of problems even more. So, while the design looks nice in theory, and I'm sure they have some wonderful reports about how thorough their test framework is, in practice it might be a tire fire (depending on how it's actually used). Some people are going to go, "I've been using this in production for a year, it's rock solid!" Well, people said that about Riak, until they wanted to set it on fire when they found out how buggy the implementation was at actually handling various errors and edge cases, and that they each had to hire dedicated Riak programmers just to fix all its problems. You can make large-scale Riak clusters work "at scale" if you hire enough on-call to constantly fight fires and monkey-patch the bugs and juggle clusters and nodes and repair indexes, and hand-wave outages and flaws as due to some other issue ("the cluster was completely hosed and had to be recovered from backup because the disks got corrupted and a node went down and a network partitioned all at once, it's not Riak's fault"). (looking at this HN thread, it seems like my assumption may be correct: https://news.ycombinator.com/item?id=27424605 https://news.ycombinator.com/item?id=27424605)
- j16sdiz 5y agoIf this paper is to be trusted. FDB is tested in many failure mode and scenceio. I expect it may fail, but should fail in predictable way.
- throwaway984393 5y agoMy point is that even if it fails in a predictable way in test scenarios, it won't necessarily in real life. The paper & test framework are effectively going to trick people into trusting that the thing will work well, rather than going by actual observations of how it works in varied production environments. For the casual observer, who cares? But for the people about to spend millions of dollars on this tech to run it in production, I hope they aren't swayed purely by theoreticals. Remember a decade ago when NoSQL came out, and everybody was hooping and hollering about how amazing the concept was, and people like me would go "Well, wait a minute, how well does it actually run in production?" And people on HN would shout us down because the cool new toy is always awesome. And lo and behold, most NoSQL databases are no better in production than Postgres, if not a tire fire. People who have fought these fires before can smell the smoke miles away.
- fiddlerwoaroof 5y agoOne thing I’ve never been able to figure out with FoundationDB is how to deploy it and a good list of layers for various use cases (SQL, document store, etc.)
- davgoldin 5y agoThese two resources helped me a lot in understanding how to setup FoundationDB processes correctly: https://nikita.melkozerov.dev/posts/2019/06/building-a-foundationdb-cluster-roles-classes-and-processes/ https://nikita.melkozerov.dev/posts/2019/06/building-a-found... https://forums.foundationdb.org/t/roles-classes-matrix/1340/2 https://forums.foundationdb.org/t/roles-classes-matrix/1340/...
- didip 5y agoWouldn't the "unbundled" cause a lot of headache when trying to deploy it in production?
- dilyevsky 5y agoAnyone has real experience running this in prod vs tikv? Latter seems like a more familiar design but missing watches which fdb seems to support
- OctoberTime 5y agoI've only ever tinkered with tikv, but one advantage that it currently has over fdb is its support of co-processors --the ability for each node in the cluster to run some piece of query logic locally on its own shard of data instead of having all of the data shipped to one node (or even all the way up to the application level) for processing, which is quite bandwidth intensive.
- taywrobel 5y agoFormer Apple employee here. They’ve been using it extensively in production for years. Pretty much all of CloudKit is run using FDB for storage. Public source for that info, so I don’t get accused of NDA violations - https://youtu.be/HLE8chgw6LI https://youtu.be/HLE8chgw6LI Timestamp 14:15 touches on CloudKit’s extensive FDB usage.
- davgoldin 5y agoOne of PingCap's (company behind TiKV and TiDB) founders once did his own FoundationDB review [1] - he concluded: "the performance is cool". I'm assuming at that time FDB's performance was superior to TiKV. [1] https://medium.com/@siddontang/benchmark-foundationdb-with-go-ycsb-f7282509dd58 https://medium.com/@siddontang/benchmark-foundationdb-with-g...
- OctoberTime 5y agoI'd love to know how Apple distributes its CloudKit data across all of (what I assume are) its dozens of globally distributed FoundationDB clusters. As of version 6.0, a single FoundationDB cluster currently only supports one active, writable 'master' region + one hot failover region. Transactions are all essentially LAN so performance characteristics are great. It doesn't support the slower TiDB/CRDB style active/active configuration where all nodes are writable, data is globally distributed and transactions can span across all nodes in the cluster. Do they have some kind of scheme whereby ACID is preserved for local transactions only, while far away regions are asynchronously replicated to with less guarantees on data consistency?
- deleted 5y ago[deleted]
- adamfeldman 5y agoThere were 2 FDB papers published recently – this one discusses some of that: https://www.foundationdb.org/files/QuiCK.pdf https://www.foundationdb.org/files/QuiCK.pdf
- Operyl 5y agoI suspect it’s because they’re not just running a single cluster, but many many clusters with customers distributed globally. They had a talk about how they do it with other systems some time ago that I can’t find at the moment.
- foobiekr 5y agoMaximum fdb cluster size is actually not that large, in host count terms, and the maximum practical size per host is similarly not large for the Apple scale, so they are almost certainly sharding objects across hundreds or thousands of clusters. This has nice properties for blast containment.
- endisneigh 5y agoOnce CouchDB switches to using FoundationDB I think it'll surpass most NoSQL stores. Can't wait. The main issue right now with FoundationDB is that it's very nontrivial to setup and interface with IMHO.
- option_greek 5y agoAny announced date for when this switch might happen? Overall a good decision by Couchdb community.
- tehbeard 5y agoI'm not sure when FDB will get easier soon, if ever. Certainly had a devil of a time trying to get it to work on windows.
- speedgoose 5y agoI also look forward to CouchDB 4. But CouchDB 2 and 3 has been very easy to setup and use over the years. Hopefully it stays the same.
- drenvuk 5y agoKeep in mind that while Foundationdb has its place and it's really good at what it does it's ssd only. If you're running on HDDs for cheaper storage you're better off with something else.
- PostThisTooFast 5y agoIt's "key/value store.' It's sad that the slash has become too hard for people.
- CodesInChaos 5y agoFoundationDB has rather severe limitations: > Transaction size cannot exceed 10,000,000 bytes of affected data In my experience almost all transactions are smaller than that, but the there is a long tail of large ones even for the same operation (because some 1:n link usually has a small n, but occasionally has an n which is orders of magnitude larger). > FoundationDB currently does not support transactions running for over five seconds. Limiting the duration of write-transactions isn't too bad. But being able to use a longer lived readonly snapshot is very convenient. Do higher level databases building on FoundationDB (e.g. the SQL layer) typically work around some of these limitations, or do they leave it to the application to avoid these?
- maxmcd 5y agoThe Record Layer provides mechanisms to get around the key size limitations and the the transaction time/size limitations. See section 4: https://www.foundationdb.org/files/record-layer-paper.pdf https://www.foundationdb.org/files/record-layer-paper.pdf
- qaq 5y agoIf that is severe limitation how would you characterize DynamoDB?
- Diggsey 5y agoThe size limit makes a lot of sense when you look at the transaction model. When you write to FDB, none of the writes are even sent to the server until you attempt to commit the transaction, so this limit is actually a "maximum request size limit" on the final commit request. AIUI, the transaction size does not include values that are read, and for snapshot reads does not include the keys either, so this 10 MB limit only really constrains transactions that are writing a large volume of data. For those transactions, the documentation suggests writing the data first (using manny smaller transactions) and then only updating a pointer to that data in the final transaction. I'm currently building a simple database I'm calling AgentDB on top of FDB. It implements a message-passing system within the database, with guaranteed exactly-once semantics. It's designed to allow business logic to be more easily expressed without having to worry about all the failure modes typically present in a distributed system. For this, I process messages in batches, with one transaction per batch. If a transaction fails, I retry with a smaller batch size, so in my case the answer would be "yes, to an extent". The user of AgentDB still has to ensure that processing a single message doesn't exceed the transaction size, but they don't have to worry that a batch of messages would exceed that.
- jwr 5y agoI'm migrating from RethinkDB to FoundationDB right now. One thing which is easy to miss when you look at FoundationDB is that it is more of a toolkit for building databases than a database in itself. So, if you're looking for a faster horse, say a "better PostgreSQL", or a MongoDB that doesn't lose your data, it will seem disappointing ("where are all the features I expected?"). Where I think it really shines is when you couple it tightly with your application. Suddenly you get a fully transactional distributed database with clearly understandable semantics, and if you can show some flexibility in how you map your data onto the underlying primitives, you can get fantastic performance, as well as correctness.