9 ms·

FoundationDB: A Distributed Key-Value Store

298 points by eatonphil 3y ago

qaq 3y ago
Suprised none used it as a foundation for a NewSQL DB, the thing is battle tested and actively developed by Apple and Snowflake.
- endisneigh 3y ago
  There’s https://www.tigrisdata.com/ https://www.tigrisdata.com/ It’s similar to mongo (it’s nosql)
  - eatonphil 3y ago
    Not a NewSQL database though as GP mentioned. I don't think Tigris has a SQL layer.
    - endisneigh 3y ago
      Yes I know. I explicitly said it was similar to mongo. Just responding to the bit that it’s battle tested and used as a foundation (no pun intended) for another db. As far as I know it’s the only database that has a company around it that is using FDB
  - qaq 3y ago
    there was poc of sqlite on top of FDB. There is also sql layer that Apple did not open source that they use at scale. Just seems a wasted opportunity.
    - endisneigh 3y ago
      It’s because you introduce a lot of latency. Cockroachdb for example (which is a great db) has a lot of latency compared to Postgres. At the time of its release it was probably hard to justify having an order of magnitude more latency than competitors (of course they were not fault tolerant, but still).
      - riku_iki 3y ago
        hypothetically, you can run cocroach with replication factor 1, and have also low latency and apples to apples comparison.
- canadiantim 3y ago
  I know some people have had success using FoundationDB as a KV store with SurrealDB[1] [1] https://github.com/orgs/surrealdb/discussions/25 https://github.com/orgs/surrealdb/discussions/25
  - qaq 3y ago
    thats a document-graph database though
- danpalmer 3y ago
  I think I remember the FDB team developing one that was closed source back before their acquisition. I thought the business model was going to be open core and closed, paid, layers on top. I seem to remember them benchmarking the SQL layer and it being highly performant still, despite the complexity it added. Maybe this thing still exists in close source form at Apple? It wouldn't surprise me if it does and forms the basis of a Spanner alternative, they're big enough to need it. Or maybe they canned it pre/post acquisition. Edit: ah, you've already mentioned the closed source layer that exists at Apple. There we go!
endisneigh 3y ago
I’ve been using FDB for toy projects for a while. It’s truly rock solid. In my experience it’s the best open source database I’ve used, including mariadb, Postgres and cockroach. That being said, I wish there were more layers as the functionality out of the box is very very limited. Ideally someone could implement the firestore or dynamodb api on top. https://github.com/losfair/mvsqlite https://github.com/losfair/mvsqlite Is basically distributed SQLite backed by FDB. I’ve been scared to use it since I don’t know rust and can’t attest to if mvcc had been implemented correctly. In using this I actually realized how coupled the storage engine is to the storage system and how few open source projects make the storage engine easily swap-able.
- facu17y 3y ago
  mvcc is already taken care of by fdb, no?
  - endisneigh 3y ago
    Yea, but mvsqlite implements its own to get around the limitations around transactions.
- fhrow4484 3y ago
  > That being said, I wish there were more layers as the functionality out of the box is very very limited. The record layer https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer which allows to store protobuf, and define the primary keys and index directly on those proto fields is truly amazing: https://github.com/FoundationDB/fdb-record-layer/blob/main/docs/Overview.md https://github.com/FoundationDB/fdb-record-layer/blob/main/d...
  - chrischen 3y ago
    It’s unfortunately Java only.
- tommiegannert 3y ago
  I really wanted to use FoundationDB for building a graph database, but was taken aback by the limitations in record (10+100 kB) and somewhat transaction sizes (10 MB) [1]. And the documentation [2] doesn't really give any answers than "build it yourself." mvsqlite seems to improve the transaction size [3], which is nice. Does it also improve the key/value limitations? > Transaction size cannot exceed 10,000,000 bytes of affected data. [---] Keys cannot exceed 10,000 bytes in size. Values cannot exceed 100,000 bytes in size. [1] https://apple.github.io/foundationdb/known-limitations.html https://apple.github.io/foundationdb/known-limitations.html [2] https://apple.github.io/foundationdb/largeval.html https://apple.github.io/foundationdb/largeval.html
tanepiper 3y ago
A few years ago I was working at an agency, one of their teams was building a real-time gaming system on top of FoundationDB. Apple then bought it up and shut the open source down. They had to rebuild whole layers from scratch.
- 58028641 3y ago
  As far as I can tell, FoundationDB was never open source until Apple open sourced it.
  - tanepiper 3y ago
    At least one reference on here from 2018 - https://news.ycombinator.com/item?id=16878786 https://news.ycombinator.com/item?id=16878786 And here's a news story - https://www.forbes.com/sites/benkepes/2015/03/25/a-cautionary-open-source-tale-apple-buys-and-shutters-foundationdb/ https://www.forbes.com/sites/benkepes/2015/03/25/a-cautionar...
    - endisneigh 3y ago
      It was never open source before apple. Rather the binary was freely available to be used. When apple bought them they took it away but continued to support customers with contracts. In that way it was inaccessible until it was open sourced.
      - metadat 3y ago
        Yes, I got bitten by this and will never forget- FDB abruptly shut off public access in mid-2015. Fortunately for me, it only cost half a day to migrate my system to Postgres.
    - detaro 3y ago
      ... which both state that it wasnt open-source before the apple buyout.
  - stephenr 3y ago
leetrout 3y ago
What always fascinated me is they built the simulator for the database first(ish) and relied on it as a first class citizen while building the DB: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc > We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing. https://pierrezemb.fr/posts/notes-about-foundationdb/ https://pierrezemb.fr/posts/notes-about-foundationdb/
- ajmurmann 3y ago
  Working on a distributed key/value store myself, I couldn't agree more and think what FoundationDB did for testing from the start is absolutely the way to go. Testing distributed system is very tricky and tests can be incredibly time consuming and bring everything to a halt.
- falsandtru 3y ago
  I'm loving this point. The unfortunate thing is those tests are closed source (I saw a maintainer says so probably in an issue before). It seems testable but still seems to be closed source. So we cannot fork the project even if FDB becomes totally closed source again.
  - aseipp 3y ago
    No, the simulation harness and tests are open source and you can run them. It would be impossible for anyone to contribute anyway without it, for example, Snowflake, which heavily depends on it. It's built into the server binary directly, so the same code is always used, and it's simply a different operational mode when compared to the real server. I used to have a project to do lots of simulation runs on my big 32 core server and then aggregate the logs into clickhouse for analysis. It wasn't that hard. However, they (at least at the time most of the developers were at Apple, many have now moved to Snowflake and the Apple team has grown a little I think) haven't released or integrated their nightly cluster and performance testing systems into open, nor have they integrated them with GitHub Actions or Nightly runs or anything. My understanding is that this is "just" a lot of compute cluster/platform orchestration code on top of the tests that exist in the repository. So, while Apple or Snowflake integrates changes across hundreds of concurrent fuzzing simulations on whatever platforms they have, if you write patches yourself, you're stuck with long simulation runs. Maybe that's changed; I haven't kept up since the 7.0 series. In practice if you write patches and they accept them, they will just do the testing in their runs for you, on a cluster far larger than what you could have. Failures reports will tell you how to reproduce them from the test files. As a contributor, testing the system on your own is mostly a matter of how much money or how many CPU cores you can personally stand to set on fire. Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU/compute intense, no matter how you go about it. [1] https://forums.foundationdb.org/t/how-to-use-foundationdb-unit-test/3077 https://forums.foundationdb.org/t/how-to-use-foundationdb-un... [2] https://github.com/apple/foundationdb/tree/main/tests https://github.com/apple/foundationdb/tree/main/tests
romanhn 3y ago
Back in 2014 or so, I saw the FoundationDB team demo the product at a developer conference. They had the database running across a bunch of machines, with a visual showing their health and data distribution. One team member would then be turning machines on and off (or maybe unplugging them from the network) and you could see FDB effortlessly rebalancing the data across the available nodes. It was a very striking, impressive presentation (especially as we were dealing with the challenges of distributed Cassandra at the time).
- boxcarr 3y ago
  When I saw the post about Foundation DB, I remembered the exact same demo running on a cluster of Raspberry Pi instances! Sadly, no memory of it on YouTube.
  - boxcarr 3y ago
    Wayback Machine partially to the rescue. Found the page (https://web.archive.org/web/20150325003301/https://foundationdb.com/videos/foundationdb-fault-tolerance-demo https://web.archive.org/web/20150325003301/https://foundatio...), but the Vimeo video was nuked when foundationdb.com shutdown. Here's the HN thread about that demo: https://news.ycombinator.com/item?id=5739721 https://news.ycombinator.com/item?id=5739721
    - romanhn 3y ago
      I feel like I saw something a bit more refined (I recall node statuses aggregated on one cool UI), so this may have been an earlier iteration, but the beginning of the following video has some of what we're talking about: https://youtu.be/Nrb3LN7X1Pg https://youtu.be/Nrb3LN7X1Pg
georgelyon 3y ago
FoundationDB is a truly one-of-a-kind bit of technology. Others have already linked to the testing methodology that allows them to run orders of magnitude more database hours in test than have run in production: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc A less known but also great talk is the follow which talked about what the a few of the team worked on next, effectively trying to generalize the methodology to any computer program: https://www.youtube.com/watch?v=fFSPwJFXVlw https://www.youtube.com/watch?v=fFSPwJFXVlw I liken the approach to being able to fuzz the execution space of the program, not just the inputs.
cetinsert 3y ago
https://deno.com/deploy https://deno.com/deploy is building https://deno.com/kv https://deno.com/kv on FoundationDB!
ibotty 3y ago
Every time I look at FoundationDB for replacing some Redis usage I wonder about key expiry/TTL, look for it and find nothing. Is this such a strange use case, that there is not even a blog entry about it only some forum entries?
- endisneigh 3y ago
  You would need to implement that yourself. Easily can be done by storing tuples with your expiry date. You then could watch the keys to remove expired keys automatically. FDB is very barebones by design. Alternatively (and easier): https://forums.foundationdb.org/t/designing-key-value-expiration-in-fdb/156 https://forums.foundationdb.org/t/designing-key-value-expira...
mlerner 3y ago
Really neat paper - a while ago I wrote a summary of the system: https://www.micahlerner.com/2021/06/12/foundationdb-a-distributed-unbundled-transactional-key-value-store.html https://www.micahlerner.com/2021/06/12/foundationdb-a-distri...
monstrado 3y ago
I built an online / mutable time-series database using FDB a few years back at a previous company. Not only was it rock solid, but it scaled linearly pretty effortlessly. It truly is one of novel modern pieces of technologies out there, and I wish there were more layers built on top of it.
neftaly 3y ago
I've been tooling around with "Tuple Database", which claims to be FoundationDB for the frontend (by the original dev of Notion). https://github.com/ccorcos/tuple-database/ https://github.com/ccorcos/tuple-database/ I have found it conceptually similar to Relic or Datascript, but with strong preformance guarantees - something Relic considers a potential issue. It also solves the problem of using reactive queries to trigger things like popups and fullscreen requests, which must be run in the same event loop as user input. https://github.com/wotbrew/relic https://github.com/wotbrew/relic https://github.com/tonsky/datascript https://github.com/tonsky/datascript Having a full (fast!) database as my React state manager gives me LAMP nostalgia :)
mrtracy 3y ago
FoundationDB has, in my experience, always been well regarded in DB development circles; I think their test architecture - developed to easily reproduce rare concurrency failures - is its best legacy, as mentioned in a comment above and frequently before. However, since these topics are always filled with effusive praise in the comments, let me give an example of a distributed scenario where FDB has shortcomings: OLTP SQL. First, FDB is clearly designed for “read often, update rarely” workloads, in a relative sense. It produces multiple consistent replicas which are consistently queryable at a past time stamp, without a transaction - excellent for that profile. However, its transaction consistency method is both optimistic and centralized, and can lead to difficulty writing during high contention and (brief) system-wide transaction downtime if there is a failover; while it will work, it’s not optimal for “write often, read once” workloads. Secondly, while it is an ordered key value store - facilitating building SQL on top of it - the popular thought of layering SQL on top of the distributed layer comes with many shortcomings. My key example of this is schema changes. Optimistic application, and keeping schema information entirely “above” the transaction layer, can make it extremely slow to apply changes to large tables, and possibly require taking them partially offline during the update. There are ways to manage this, but online schema changes will be a competitive advantage for other systems. Even for read-only queries, you lose opportunities to push many types of predicates down to the storage node, where they can be executed with fewer round trips. Depending on how distributed your system is, this could add up to significant additional latency. Afaik, all of the spanner-likes of the world push significant schema-specific information into their transaction layers - and utilize pessimistic locking - to facilitate these scenarios with competitive performance. For reasons like these, I think FDB will find (and has found) the most success in warehousing scenarios, where individual datum are queried often once written, and updates come in at a slower pace than the reads.
- preseinger 3y ago
  do you think the things you mention were deliberate design decisions?
  - mike_hearn 3y ago
    Yes, one of the nice things about FDB is it has extensive design docs. Optimizing for reading more often than writing is obviously a pretty normal design choice, outside of log ingestion you'll normally be reading more than writing. There are people using FDB for logs (snowflake iirc?) and it's been optimized for that sort of use case more in recent years, but it's not like it was an unreasonable choice.
jeffbee 3y ago
How hard have people pushed this thing? We get regular threads of effusive praise, but little criticism. Last time I mentioned that years ago my colleagues found half a dozen ways to lose data in FDB I got called out here and even in private emails, but it seems more valuable to know where the limits of these systems are, and not very valuable to read the positive feelings of people who used FDB in trivial and uncritical ways.
- Dave_Rosenthal 3y ago
  Yes, there are definitely a lot of big companies that have used FoundationDB very hard at huge scale for many years. That said, yeah, it feels like there are also a lot of folks on HN who just jump on the "cool, fault simulation" bandwagon and don't have a lot of personal real-world experience. What I can tell you, for sure, is that if you find an issue with something as important and fundamental as data loss the team working on FoundationDB would take it super seriously.
- eatonphil 3y ago
  Is Snowflake big enough of a deal? https://news.ycombinator.com/item?id=16880404 https://news.ycombinator.com/item?id=16880404 Also, in the post itself, authors including Apple and Snowflake devs, it mentions it's run in production by Apple and Snowflake. I haven't seen yet though what Apple uses it for.
  - tilolebo 3y ago
    It is used by CloudKit https://machinelearning.apple.com/research/foundationdb-record-layer-a-multi-tenant-structured-datastore https://machinelearning.apple.com/research/foundationdb-reco...
  - jeffbee 3y ago
    The time at which my colleagues found easy ways to lose data was well after Apple had claimed to use it in iCloud at scale. So, I don't think deployment at scale is a proof of correctness. The thing that needs doing is regularly looking in the database for things that should be there.
deleted 3y ago
[deleted]
mprime1 3y ago
Part of the FDB team (great folks) went on to create something quite incredible I have the pleasure of having early access to. If you’re into dependability check this out: https://antithesis.com/ https://antithesis.com/
- skybrian 3y ago
  Could you summarize? What kind of tool is it?
  - mrAssHat 3y ago
    That's a tool for developer productivity and customer satisfaction. Also, they are hiring.
  - mprime1 3y ago
    The intro video on the homepage does a good job at explaining it in 3 minutes. The video is not upselling. The tool can do what the video promises. Say your system is "well tested" for example with a combination of unit tests, integration tests, stress tests, failure injection tests (a la Jepsen), and more. There are probably still hundreds of nasty little bugs hidden. Not even thousands of experiment hours with a Jepsen-like approach would surface them because they are very, very unlikely. Antithesis will find these without breaking a sweat. It's designed to hunt for very unlikely (but possible) scenarios where your system misbehaves. And here's the real kicker: once a bug is found, you can observe and step-through execution of your entire _distributed_ system. Similar to attaching a debugger to a single process but for an entire system composed of many clients and servers connected by a network. It's language independent and doesn't require any modification to your system in order to use. It's pretty incredible. I would not believe this is possible if it hadn't seen it with my own eyes. Presently, I am using it for work at Synadia (makers of NATS). NATS is like lego blocks to build all kinds of distributed systems in multiple languages, so it has a very large surface area. It's well tested, stable, and deployed successfully by many large and small projects and companies. Contemporary testing approaches can hardly find bugs. Antithesis can find very insidious edge cases where things break. And we proactively investigate and fix before any user/customer can be affected by these one-in-a-million nasty bugs, which would otherwise be very hard to find and resolve.
jwr 3y ago
FoundationDB is absolutely incredible and I've been wondering why it doesn't get more popular over time. I suspect it's too complex to use directly in most applications, with people used to SQL-based solutions or simple KV stores. I always wanted my app to use a fully distributed database (for redundancy). I've been using RethinkDB in production for over 8 years now. I'm slowly rebuilding my app to use FoundationDB. What I discovered when I started using FDB surprised me a bit. To make really good use of the database you can't really use a "database layer" and fully abstract it away from your app. Your code should be fully aware of transaction boundaries, for example. To make good use of versionstamps (an incredible feature) your code needs to be somewhat aware of them. I think FDB is a great candidate for implementing a "user-friendly" database on top of it, and in fact several databases are doing exactly that (using FDB as a "lower layer"). But that abstracts away too much, at least for me. The superficial take on FDB is "waah, where are my features? it doesn't do indexing? waaah, just use Postgres!". But when you actually start mapping your app's data structures onto FDB optimally, you discover a whole new world. For example, I ended up writing my indexing code myself, in my language (Clojure). FDB gives you all the tools, and a strict serializable data model to work with — your language brings your data structures and your indexing functions. The combination is incredible. Once you define your index functions in your language, you will never want to look at SQL again. Plus, you get incredible features like versionstamps — I use them to replace RethinkDB changefeeds and implement super quick polling for recent changes. Oh, and did I mention that it is a fully distributed database that correctly implements the strict serializable consistency model? There are very few dbs that can claim that. If you understand what that means, you probably know how incredible this is and I don't have to convince you. If you think you understand, I suggest you go and explore https://jepsen.io/consistency https://jepsen.io/consistency — carefully reading and learning about the differences in various consistency models. I really worry that FoundationDB will not become popular because of its inherent complexity, while worse solutions (ahem, MongoDB) will be more fashionable. I would encourage everyone to at least take a look at FDB. It really is something quite different.
- manish_gill 3y ago
  Seems like a very similar use-case like Zookeeper - use it for distributed coordination / consistency etc and build your actual database on top of it.
hadjian 3y ago
His jokes are hilarious!
deleted 3y ago
[deleted]
brainzap 3y ago
I think FoundationDB will also have parts written in Swift, at least that is what Apple showed at WWDC.
- crabmusket 3y ago
  That was Foundation, not FoundationDB. https://developer.apple.com/documentation/foundation https://developer.apple.com/documentation/foundation
  - jen20 3y ago
    No, it was FoundationDB [1]. [1]: https://developer.apple.com/videos/play/wwdc2023/10164/?time=2300 https://developer.apple.com/videos/play/wwdc2023/10164/?time...
    - wooster 3y ago
      Interesting… https://github.com/apple/foundationdb/pull/10156 https://github.com/apple/foundationdb/pull/10156
    - crabmusket 3y ago
      Oh thanks! Not sure how I got my wires crossed there.
      - jen20 3y ago
        Easy mistake to make - they're both pretty similar in naming!
jbverschoor 3y ago
Around 2010-2013 (gaming), I found fdb, and to me it seemed like the perfect database because of their architecture. I tried it a bit, and was really happy with it. Unfortunately they were acquired by Apple, only to resurface something like 10 years later. All momentum was gone, and I’m not really aware nor interested in where they stand. I’ll stick with my rusty old Postgres for a long time before I’d try anything else out.
debussyman 3y ago
I worked next to the founders a decade ago and tried the first versions of the project (before Apple acq). Loved the concept, but it hasn't really lived up to the promise.
dangoodmanUT 3y ago
It's a shame that they have left the Go bindings back in version 6, despite 7 being out for some time now.

298 points by eatonphil 3y ago

qaq 3y ago

Suprised none used it as a foundation for a NewSQL DB, the thing is battle tested and actively developed by Apple and Snowflake.

endisneigh 3y ago
There’s https://www.tigrisdata.com/ https://www.tigrisdata.com/ It’s similar to mongo (it’s nosql)
- eatonphil 3y ago
  Not a NewSQL database though as GP mentioned. I don't think Tigris has a SQL layer.
  - endisneigh 3y ago
    Yes I know. I explicitly said it was similar to mongo. Just responding to the bit that it’s battle tested and used as a foundation (no pun intended) for another db. As far as I know it’s the only database that has a company around it that is using FDB
- qaq 3y ago
  there was poc of sqlite on top of FDB. There is also sql layer that Apple did not open source that they use at scale. Just seems a wasted opportunity.
  - endisneigh 3y ago
    It’s because you introduce a lot of latency. Cockroachdb for example (which is a great db) has a lot of latency compared to Postgres. At the time of its release it was probably hard to justify having an order of magnitude more latency than competitors (of course they were not fault tolerant, but still).
    - riku_iki 3y ago
      hypothetically, you can run cocroach with replication factor 1, and have also low latency and apples to apples comparison.
canadiantim 3y ago
I know some people have had success using FoundationDB as a KV store with SurrealDB[1] [1] https://github.com/orgs/surrealdb/discussions/25 https://github.com/orgs/surrealdb/discussions/25
- qaq 3y ago
  thats a document-graph database though
danpalmer 3y ago
I think I remember the FDB team developing one that was closed source back before their acquisition. I thought the business model was going to be open core and closed, paid, layers on top. I seem to remember them benchmarking the SQL layer and it being highly performant still, despite the complexity it added. Maybe this thing still exists in close source form at Apple? It wouldn't surprise me if it does and forms the basis of a Spanner alternative, they're big enough to need it. Or maybe they canned it pre/post acquisition. Edit: ah, you've already mentioned the closed source layer that exists at Apple. There we go!

endisneigh 3y ago

I’ve been using FDB for toy projects for a while. It’s truly rock solid. In my experience it’s the best open source database I’ve used, including mariadb, Postgres and cockroach. That being said, I wish there were more layers as the functionality out of the box is very very limited. Ideally someone could implement the firestore or dynamodb api on top. https://github.com/losfair/mvsqlite https://github.com/losfair/mvsqlite Is basically distributed SQLite backed by FDB. I’ve been scared to use it since I don’t know rust and can’t attest to if mvcc had been implemented correctly. In using this I actually realized how coupled the storage engine is to the storage system and how few open source projects make the storage engine easily swap-able.

facu17y 3y ago
mvcc is already taken care of by fdb, no?
- endisneigh 3y ago
  Yea, but mvsqlite implements its own to get around the limitations around transactions.
fhrow4484 3y ago
> That being said, I wish there were more layers as the functionality out of the box is very very limited. The record layer https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer which allows to store protobuf, and define the primary keys and index directly on those proto fields is truly amazing: https://github.com/FoundationDB/fdb-record-layer/blob/main/docs/Overview.md https://github.com/FoundationDB/fdb-record-layer/blob/main/d...
- chrischen 3y ago
  It’s unfortunately Java only.
tommiegannert 3y ago
I really wanted to use FoundationDB for building a graph database, but was taken aback by the limitations in record (10+100 kB) and somewhat transaction sizes (10 MB) [1]. And the documentation [2] doesn't really give any answers than "build it yourself." mvsqlite seems to improve the transaction size [3], which is nice. Does it also improve the key/value limitations? > Transaction size cannot exceed 10,000,000 bytes of affected data. [---] Keys cannot exceed 10,000 bytes in size. Values cannot exceed 100,000 bytes in size. [1] https://apple.github.io/foundationdb/known-limitations.html https://apple.github.io/foundationdb/known-limitations.html [2] https://apple.github.io/foundationdb/largeval.html https://apple.github.io/foundationdb/largeval.html

tanepiper 3y ago

A few years ago I was working at an agency, one of their teams was building a real-time gaming system on top of FoundationDB. Apple then bought it up and shut the open source down. They had to rebuild whole layers from scratch.

58028641 3y ago
As far as I can tell, FoundationDB was never open source until Apple open sourced it.
- tanepiper 3y ago
  At least one reference on here from 2018 - https://news.ycombinator.com/item?id=16878786 https://news.ycombinator.com/item?id=16878786 And here's a news story - https://www.forbes.com/sites/benkepes/2015/03/25/a-cautionary-open-source-tale-apple-buys-and-shutters-foundationdb/ https://www.forbes.com/sites/benkepes/2015/03/25/a-cautionar...
  - endisneigh 3y ago
    It was never open source before apple. Rather the binary was freely available to be used. When apple bought them they took it away but continued to support customers with contracts. In that way it was inaccessible until it was open sourced.
    - metadat 3y ago
      Yes, I got bitten by this and will never forget- FDB abruptly shut off public access in mid-2015. Fortunately for me, it only cost half a day to migrate my system to Postgres.
  - detaro 3y ago
    ... which both state that it wasnt open-source before the apple buyout.
- stephenr 3y ago

leetrout 3y ago

What always fascinated me is they built the simulator for the database first(ish) and relied on it as a first class citizen while building the DB: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc > We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing. https://pierrezemb.fr/posts/notes-about-foundationdb/ https://pierrezemb.fr/posts/notes-about-foundationdb/

ajmurmann 3y ago
Working on a distributed key/value store myself, I couldn't agree more and think what FoundationDB did for testing from the start is absolutely the way to go. Testing distributed system is very tricky and tests can be incredibly time consuming and bring everything to a halt.
falsandtru 3y ago
I'm loving this point. The unfortunate thing is those tests are closed source (I saw a maintainer says so probably in an issue before). It seems testable but still seems to be closed source. So we cannot fork the project even if FDB becomes totally closed source again.
- aseipp 3y ago
  No, the simulation harness and tests are open source and you can run them. It would be impossible for anyone to contribute anyway without it, for example, Snowflake, which heavily depends on it. It's built into the server binary directly, so the same code is always used, and it's simply a different operational mode when compared to the real server. I used to have a project to do lots of simulation runs on my big 32 core server and then aggregate the logs into clickhouse for analysis. It wasn't that hard. However, they (at least at the time most of the developers were at Apple, many have now moved to Snowflake and the Apple team has grown a little I think) haven't released or integrated their nightly cluster and performance testing systems into open, nor have they integrated them with GitHub Actions or Nightly runs or anything. My understanding is that this is "just" a lot of compute cluster/platform orchestration code on top of the tests that exist in the repository. So, while Apple or Snowflake integrates changes across hundreds of concurrent fuzzing simulations on whatever platforms they have, if you write patches yourself, you're stuck with long simulation runs. Maybe that's changed; I haven't kept up since the 7.0 series. In practice if you write patches and they accept them, they will just do the testing in their runs for you, on a cluster far larger than what you could have. Failures reports will tell you how to reproduce them from the test files. As a contributor, testing the system on your own is mostly a matter of how much money or how many CPU cores you can personally stand to set on fire. Someone could probably integrate this functionality into a Kubernetes operator or something so that outside engineers could run large scale simulations reliably. But it is really expensive and CPU/compute intense, no matter how you go about it. [1] https://forums.foundationdb.org/t/how-to-use-foundationdb-unit-test/3077 https://forums.foundationdb.org/t/how-to-use-foundationdb-un... [2] https://github.com/apple/foundationdb/tree/main/tests https://github.com/apple/foundationdb/tree/main/tests

romanhn 3y ago

Back in 2014 or so, I saw the FoundationDB team demo the product at a developer conference. They had the database running across a bunch of machines, with a visual showing their health and data distribution. One team member would then be turning machines on and off (or maybe unplugging them from the network) and you could see FDB effortlessly rebalancing the data across the available nodes. It was a very striking, impressive presentation (especially as we were dealing with the challenges of distributed Cassandra at the time).

boxcarr 3y ago
When I saw the post about Foundation DB, I remembered the exact same demo running on a cluster of Raspberry Pi instances! Sadly, no memory of it on YouTube.
- boxcarr 3y ago
  Wayback Machine partially to the rescue. Found the page (https://web.archive.org/web/20150325003301/https://foundationdb.com/videos/foundationdb-fault-tolerance-demo https://web.archive.org/web/20150325003301/https://foundatio...), but the Vimeo video was nuked when foundationdb.com shutdown. Here's the HN thread about that demo: https://news.ycombinator.com/item?id=5739721 https://news.ycombinator.com/item?id=5739721
  - romanhn 3y ago
    I feel like I saw something a bit more refined (I recall node statuses aggregated on one cool UI), so this may have been an earlier iteration, but the beginning of the following video has some of what we're talking about: https://youtu.be/Nrb3LN7X1Pg https://youtu.be/Nrb3LN7X1Pg

georgelyon 3y ago

FoundationDB is a truly one-of-a-kind bit of technology. Others have already linked to the testing methodology that allows them to run orders of magnitude more database hours in test than have run in production: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc A less known but also great talk is the follow which talked about what the a few of the team worked on next, effectively trying to generalize the methodology to any computer program: https://www.youtube.com/watch?v=fFSPwJFXVlw https://www.youtube.com/watch?v=fFSPwJFXVlw I liken the approach to being able to fuzz the execution space of the program, not just the inputs.

cetinsert 3y ago

https://deno.com/deploy https://deno.com/deploy is building https://deno.com/kv https://deno.com/kv on FoundationDB!

ibotty 3y ago

Every time I look at FoundationDB for replacing some Redis usage I wonder about key expiry/TTL, look for it and find nothing. Is this such a strange use case, that there is not even a blog entry about it only some forum entries?

endisneigh 3y ago
You would need to implement that yourself. Easily can be done by storing tuples with your expiry date. You then could watch the keys to remove expired keys automatically. FDB is very barebones by design. Alternatively (and easier): https://forums.foundationdb.org/t/designing-key-value-expiration-in-fdb/156 https://forums.foundationdb.org/t/designing-key-value-expira...

mlerner 3y ago

Really neat paper - a while ago I wrote a summary of the system: https://www.micahlerner.com/2021/06/12/foundationdb-a-distributed-unbundled-transactional-key-value-store.html https://www.micahlerner.com/2021/06/12/foundationdb-a-distri...

monstrado 3y ago

I built an online / mutable time-series database using FDB a few years back at a previous company. Not only was it rock solid, but it scaled linearly pretty effortlessly. It truly is one of novel modern pieces of technologies out there, and I wish there were more layers built on top of it.

neftaly 3y ago

I've been tooling around with "Tuple Database", which claims to be FoundationDB for the frontend (by the original dev of Notion). https://github.com/ccorcos/tuple-database/ https://github.com/ccorcos/tuple-database/ I have found it conceptually similar to Relic or Datascript, but with strong preformance guarantees - something Relic considers a potential issue. It also solves the problem of using reactive queries to trigger things like popups and fullscreen requests, which must be run in the same event loop as user input. https://github.com/wotbrew/relic https://github.com/wotbrew/relic https://github.com/tonsky/datascript https://github.com/tonsky/datascript Having a full (fast!) database as my React state manager gives me LAMP nostalgia :)

mrtracy 3y ago

FoundationDB has, in my experience, always been well regarded in DB development circles; I think their test architecture - developed to easily reproduce rare concurrency failures - is its best legacy, as mentioned in a comment above and frequently before. However, since these topics are always filled with effusive praise in the comments, let me give an example of a distributed scenario where FDB has shortcomings: OLTP SQL. First, FDB is clearly designed for “read often, update rarely” workloads, in a relative sense. It produces multiple consistent replicas which are consistently queryable at a past time stamp, without a transaction - excellent for that profile. However, its transaction consistency method is both optimistic and centralized, and can lead to difficulty writing during high contention and (brief) system-wide transaction downtime if there is a failover; while it will work, it’s not optimal for “write often, read once” workloads. Secondly, while it is an ordered key value store - facilitating building SQL on top of it - the popular thought of layering SQL on top of the distributed layer comes with many shortcomings. My key example of this is schema changes. Optimistic application, and keeping schema information entirely “above” the transaction layer, can make it extremely slow to apply changes to large tables, and possibly require taking them partially offline during the update. There are ways to manage this, but online schema changes will be a competitive advantage for other systems. Even for read-only queries, you lose opportunities to push many types of predicates down to the storage node, where they can be executed with fewer round trips. Depending on how distributed your system is, this could add up to significant additional latency. Afaik, all of the spanner-likes of the world push significant schema-specific information into their transaction layers - and utilize pessimistic locking - to facilitate these scenarios with competitive performance. For reasons like these, I think FDB will find (and has found) the most success in warehousing scenarios, where individual datum are queried often once written, and updates come in at a slower pace than the reads.

preseinger 3y ago
do you think the things you mention were deliberate design decisions?
- mike_hearn 3y ago
  Yes, one of the nice things about FDB is it has extensive design docs. Optimizing for reading more often than writing is obviously a pretty normal design choice, outside of log ingestion you'll normally be reading more than writing. There are people using FDB for logs (snowflake iirc?) and it's been optimized for that sort of use case more in recent years, but it's not like it was an unreasonable choice.

jeffbee 3y ago

How hard have people pushed this thing? We get regular threads of effusive praise, but little criticism. Last time I mentioned that years ago my colleagues found half a dozen ways to lose data in FDB I got called out here and even in private emails, but it seems more valuable to know where the limits of these systems are, and not very valuable to read the positive feelings of people who used FDB in trivial and uncritical ways.

Dave_Rosenthal 3y ago
Yes, there are definitely a lot of big companies that have used FoundationDB very hard at huge scale for many years. That said, yeah, it feels like there are also a lot of folks on HN who just jump on the "cool, fault simulation" bandwagon and don't have a lot of personal real-world experience. What I can tell you, for sure, is that if you find an issue with something as important and fundamental as data loss the team working on FoundationDB would take it super seriously.
eatonphil 3y ago
Is Snowflake big enough of a deal? https://news.ycombinator.com/item?id=16880404 https://news.ycombinator.com/item?id=16880404 Also, in the post itself, authors including Apple and Snowflake devs, it mentions it's run in production by Apple and Snowflake. I haven't seen yet though what Apple uses it for.
- tilolebo 3y ago
  It is used by CloudKit https://machinelearning.apple.com/research/foundationdb-record-layer-a-multi-tenant-structured-datastore https://machinelearning.apple.com/research/foundationdb-reco...
- jeffbee 3y ago
  The time at which my colleagues found easy ways to lose data was well after Apple had claimed to use it in iCloud at scale. So, I don't think deployment at scale is a proof of correctness. The thing that needs doing is regularly looking in the database for things that should be there.

deleted 3y ago

[deleted]

mprime1 3y ago

Part of the FDB team (great folks) went on to create something quite incredible I have the pleasure of having early access to. If you’re into dependability check this out: https://antithesis.com/ https://antithesis.com/

skybrian 3y ago
Could you summarize? What kind of tool is it?
- mrAssHat 3y ago
  That's a tool for developer productivity and customer satisfaction. Also, they are hiring.
- mprime1 3y ago
  The intro video on the homepage does a good job at explaining it in 3 minutes. The video is not upselling. The tool can do what the video promises. Say your system is "well tested" for example with a combination of unit tests, integration tests, stress tests, failure injection tests (a la Jepsen), and more. There are probably still hundreds of nasty little bugs hidden. Not even thousands of experiment hours with a Jepsen-like approach would surface them because they are very, very unlikely. Antithesis will find these without breaking a sweat. It's designed to hunt for very unlikely (but possible) scenarios where your system misbehaves. And here's the real kicker: once a bug is found, you can observe and step-through execution of your entire _distributed_ system. Similar to attaching a debugger to a single process but for an entire system composed of many clients and servers connected by a network. It's language independent and doesn't require any modification to your system in order to use. It's pretty incredible. I would not believe this is possible if it hadn't seen it with my own eyes. Presently, I am using it for work at Synadia (makers of NATS). NATS is like lego blocks to build all kinds of distributed systems in multiple languages, so it has a very large surface area. It's well tested, stable, and deployed successfully by many large and small projects and companies. Contemporary testing approaches can hardly find bugs. Antithesis can find very insidious edge cases where things break. And we proactively investigate and fix before any user/customer can be affected by these one-in-a-million nasty bugs, which would otherwise be very hard to find and resolve.

jwr 3y ago

FoundationDB is absolutely incredible and I've been wondering why it doesn't get more popular over time. I suspect it's too complex to use directly in most applications, with people used to SQL-based solutions or simple KV stores. I always wanted my app to use a fully distributed database (for redundancy). I've been using RethinkDB in production for over 8 years now. I'm slowly rebuilding my app to use FoundationDB. What I discovered when I started using FDB surprised me a bit. To make really good use of the database you can't really use a "database layer" and fully abstract it away from your app. Your code should be fully aware of transaction boundaries, for example. To make good use of versionstamps (an incredible feature) your code needs to be somewhat aware of them. I think FDB is a great candidate for implementing a "user-friendly" database on top of it, and in fact several databases are doing exactly that (using FDB as a "lower layer"). But that abstracts away too much, at least for me. The superficial take on FDB is "waah, where are my features? it doesn't do indexing? waaah, just use Postgres!". But when you actually start mapping your app's data structures onto FDB optimally, you discover a whole new world. For example, I ended up writing my indexing code myself, in my language (Clojure). FDB gives you all the tools, and a strict serializable data model to work with — your language brings your data structures and your indexing functions. The combination is incredible. Once you define your index functions in your language, you will never want to look at SQL again. Plus, you get incredible features like versionstamps — I use them to replace RethinkDB changefeeds and implement super quick polling for recent changes. Oh, and did I mention that it is a fully distributed database that correctly implements the strict serializable consistency model? There are very few dbs that can claim that. If you understand what that means, you probably know how incredible this is and I don't have to convince you. If you think you understand, I suggest you go and explore https://jepsen.io/consistency https://jepsen.io/consistency — carefully reading and learning about the differences in various consistency models. I really worry that FoundationDB will not become popular because of its inherent complexity, while worse solutions (ahem, MongoDB) will be more fashionable. I would encourage everyone to at least take a look at FDB. It really is something quite different.

manish_gill 3y ago
Seems like a very similar use-case like Zookeeper - use it for distributed coordination / consistency etc and build your actual database on top of it.

hadjian 3y ago

His jokes are hilarious!

deleted 3y ago

[deleted]

brainzap 3y ago

I think FoundationDB will also have parts written in Swift, at least that is what Apple showed at WWDC.

crabmusket 3y ago
That was Foundation, not FoundationDB. https://developer.apple.com/documentation/foundation https://developer.apple.com/documentation/foundation
- jen20 3y ago
  No, it was FoundationDB [1]. [1]: https://developer.apple.com/videos/play/wwdc2023/10164/?time=2300 https://developer.apple.com/videos/play/wwdc2023/10164/?time...
  - wooster 3y ago
    Interesting… https://github.com/apple/foundationdb/pull/10156 https://github.com/apple/foundationdb/pull/10156
  - crabmusket 3y ago
    Oh thanks! Not sure how I got my wires crossed there.
    - jen20 3y ago
      Easy mistake to make - they're both pretty similar in naming!

jbverschoor 3y ago

Around 2010-2013 (gaming), I found fdb, and to me it seemed like the perfect database because of their architecture. I tried it a bit, and was really happy with it. Unfortunately they were acquired by Apple, only to resurface something like 10 years later. All momentum was gone, and I’m not really aware nor interested in where they stand. I’ll stick with my rusty old Postgres for a long time before I’d try anything else out.

debussyman 3y ago

I worked next to the founders a decade ago and tried the first versions of the project (before Apple acq). Loved the concept, but it hasn't really lived up to the promise.

dangoodmanUT 3y ago

It's a shame that they have left the Go bindings back in version 6, despite 7 being out for some time now.