10 ms·

FoundationDB: A distributed unbundled transactional key value store

197 points by eatonphil 5y ago

skyde 5y ago
For Anyone with experience with FoundationDB! How does it compare to CockroachDB ?
- eatonphil 5y ago
  FoundationDB is just a key value store, so it's not comparable. There is a relational layer you can run though called Foundation Record Layer. But this doesn't support SQL AFAICT. The examples on Github just show a query builder interface in Java [0]. There does seem to be a Lucene interface but it doesn't seem to be used [1]. [0] https://github.com/FoundationDB/fdb-record-layer/blob/main/examples/src/main/java/com/apple/foundationdb/record/sample/Main.java https://github.com/FoundationDB/fdb-record-layer/blob/main/e... [1] https://forums.foundationdb.org/t/lucene-layer-on-foundationdb/1229/5 https://forums.foundationdb.org/t/lucene-layer-on-foundation...
  - StreamBright 5y ago
    There used to be some SQL layer: https://forums.foundationdb.org/t/sql-layer-in-foundationdb/94 https://forums.foundationdb.org/t/sql-layer-in-foundationdb/...
    - eatonphil 5y ago
      Neat, here [0] is a 3rd party one at the end of the thread but this isn't the one the thread is about. Doesn't seem like that one is public. [0] https://github.com/rustdesk/opentick https://github.com/rustdesk/opentick
      - aseipp 5y ago
        It was open source and if you Google a bit you can find forks with the relevant code snapshots, here's one: https://github.com/AydinSakar/sql-layer https://github.com/AydinSakar/sql-layer
  - tonyhb 5y ago
    It's comparable based off of the fundamentals and I have the same question. They both use KVs using LSMs, do range queries, support ACID transactions, handle resharding and clustering for you. They're both based off of Spanner fundamentals. They both actually have an SQL layer - foundationdb's is the record layer. Just because one has a primary SQL interface doesn't mean we can't compare. I'd really like to know a comparison of the fundamentals including hotspot management, resharding, read/write performance, resharding etc. also and have been looking for this.
    - eatonphil 5y ago
      > They both actually have an SQL layer - foundationdb's is the record layer. Can you show me their SQL layer? I was looking for it.
      - tonyhb 5y ago
        https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer is the SQL/orm like interface. This handles things in a DB like way - joins, indexes. I misspoke when I said SQL regarding this - sorry. There's a go and rust library floating around that's not as good. I've tried em :)
        eatonphil 5y ago
        The readme says > Queries - The Record Layer does not provide a query language, however it provides query APIs with the ability to scan, filter, and sort across one or more record types, and a query planner capable of automatic selection of indexes. So I'm not sure how you can call that a SQL layer when SQL is a query language and FDB Record Layer says it doesn't have a query language?
        tonyhb 5y ago
        Yeah, I did say "I misspoke when I said SQL regarding this - sorry.". I should have said "SQL-like", or at least relation-like. There were a couple SQL-ish things I found in other languages and got them mixed up. Sorry about that :)
- ledgerdev 5y ago
  I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltactivedata.com/blog/2015/04/foundationdbs-le... Also this, excellent comparison of distributed database consistency methods, mentions cockroach, but not foundation. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst... This brings up the question, what method/protocol does foundation use for distributed consistency?
  - richieartoul 5y ago
    "What method/protocol does foundation use for distributed consistency?" This is a really tough question to answer succinctly. It took me a long time to wrap my head around it, but the TLDR is: 1. It uses a quorum of Coordinator nodes to elect another node (via Paxos) whose job it is to then assigns roles to other nodes. 2. Some of those roles are replicated (Like TLogs) and some of those roles are singletons (like the thing that advances the clock and hands out versionstamps). 3. If any node in the "transaction subsystem" ever fails, the cluster goes through a recovery where it becomes unavailable for ~3s during which it reassigns the roles and keeps chugging. In terms of what it looks like for a transaction to actually commit, thats another long conversation. Its hard to draw analogies to existing systems because it really is kind of its own thing.
    - ledgerdev 5y ago
      Awesome answer, thanks! Foundation seems like magic almost, from just choosing very smart trade offs and co-ordination.
      - richieartoul 5y ago
        Yeah that’s basically it. It really is a wonderful piece of free software that was (and still is) way ahead of its time in many ways. They basically made a ton of good architectural decisions really early on and then just nailed down the details super tight over many years. Their technique of running cluster-wide recoveries to handle failures is a great example. Most distributed systems engineers find the idea of designing a database that becomes unavailable (even for a few seconds) when a single node is lost completely appalling, but it turns out this small trade off leads to massive simplifications elsewhere and the resulting system is net/net much more reliable than anything else I’ve ever seen.
- manigandham 5y ago
  FoundationDB is a key/value store, and well-known for being fast, scalable and incredibly reliable. It passed many of the distributed database tests unlike many competitors. Here's a good HN thread when Apple open-sourced it a few years ago: https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395 CockroachDB implements postgres-compatible SQL layer on top of their own key/value store called Pebble. This is similar to TiDB which implements mysql on top of their TiKV key/value store. The latter databases offer easier programming with SQL interfaces, but since you can model just about anything on top of key/value store, there are already "layers" that can implement similar abstractions on top of FDB like the Record Layer[1], Document Layer [2], and even a SQL layer [3] at one point. 1) https://www.foundationdb.org/blog/announcing-record-layer/ https://www.foundationdb.org/blog/announcing-record-layer/ 2) https://foundationdb.github.io/fdb-document-layer/ https://foundationdb.github.io/fdb-document-layer/ 3) https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer
  - EGreg 5y ago
    What is better, FoundationDB, CockroachDB, or Hypercore?
    - vvern 5y ago
      One of these is not like the others.
  - atombender 5y ago
    An important difference between CockroachDB and TiDB is that the former has an embedded key-value store; everything runs in the same process. TiDB is split into multiple parts: TiDB proper (stateless SQL/query plan layer), TiKV (the key-value store), and PD (placement driver, the main control plane and metadata layer). TiKV also isn't a pure key-value store, as it handles some particular query operations that can be "pushed down" from TiDB to reduce network round trips. FoundationDB's role similar to TiKV. It's a pure key-value store with replication built in.
  - jandrewrogers 5y ago
    There is one caveat I would add. While you can always build just about any kind of database or data model on top of a logical KV store, the scalability and performance is almost always going to be qualitatively worse for queries than using a storage engine purpose-built for the intended type of database or data model. It is simpler to implement, which is why people do it, but you give up a lot of selectivity and locality on the query side. The target design case for KV stores is when all your queries are essentially trivial. Layering SQL on top of it implies the expected use case is complex queries of the type where a KV store won't perform well. Good for a compatibility interface, but not performant.
- richieartoul 5y ago
  I've never used CockroachDB, but FoundationDB has been an absolute dream to work with. There are a few downsides to using it: - There is (currently) no hosted solution you can use, so you have to run it yourself. - There is no great "layer" that provides a nice document/SQL layer so you kind of have to work with the raw sorted KV abstraction. - You have to structure your workload to avoid a high rate of conflicts on individual keys. - It won't scale to extremely high write volume workloads like Cassandra/Scylla will. That said, in exchange you get: - Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. Jepsen refuses to test FDB because FDB's simulation testing is much more rigorous than anything Jepsen does, but AFAIK the most recent releases of Cockroach/Mongo/Cassandra/Fauna/etc have never fully passed a Jepsen test. - Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput. - A strongly consistent store you can actually use in anger and that you don't have to baby. We've never managed to actually break a FoundationDB cluster, the worst case scenario we've ever seen is that we slam it with too many writes/reads and it starts to apply backpressure by increasing latency. - Write/Read throughput scales ~linearly as nodes are added. - Built in backup/restore from blob storage. - Strongly consistent disaster recovery. This one is amazing and its hard to go back to other databases that don't support it, but basically you can run 2 FDB clusters in DR mode where one is the leader (accepting writes + reads) and the other is passively replicating from it. You can then fail over from the primary FDB to the secondary *consistently* in such a way that the two clusters coordinate to lock the primary and then unlock the secondary so that your application is guaranteed to not see any inconsistency or data loss with no participation from the application required (except teaching it how to talk to 2 different FDB clusters at once). Basically its lacking some niceties for building traditional applications on top of it, but it is an amazing primitive for building rock solid distributed systems and will almost never let you down once you've developed some operational experience with it. Its the closest thing to "magic" that I've ever worked with in the distributed storage space.
  - sigstoat 5y ago
    to the folks asking why you'd use FDB, i'd highlight the main points as: > Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. the transaction guarantees on FDB are just the best there is in any product you can get for free. not to fault the rest, FDB just went past what ~98% of folks will ever need. but if you're in that last 2%, god speed trying to get anything else to offer you the same semantics. > A strongly consistent store you can actually use in anger and that you don't have to baby. i've seen some folks trying to run a cluster without proper monitoring, who were nearly driving the poor thing into the ground, with hosts dying, and having so much pressure put on them they failed out, but it never lost anything. i don't think it ever even became unavailable. > Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput eh, in defense of everything else, FDB is a key value store, and the keys are byte strings, so the situation is a bit easier for them.
chrisweekly 5y ago
FTA: > FoundationDB, a transactional key-value store that supports multi-key strictly serializable transactions across its entire key-space. FoundationDB (FDB, for short) is opensource. The paper says that: "FDB is the underpinning of cloud infrastructure at Apple, Snowflake and other companies, due to its consistency, robustness and availability for storing user data, system metadata and configuration, and other critical information." > The main idea in FDB is to decouple transaction processing from logging and storage. Such an unbundled architecture enables the separation and horizontal scaling of both read and write handling.
- deleted 5y ago
  [deleted]
  - deleted 5y ago
    [deleted]
rapsey 5y ago
Fdb's main selling point was reliability. But with the proliferation of good raft/paxos implementations that I think is not that much of an advantage anymore.
- manigandham 5y ago
  There's a lot more to reliability than just the consensus algo and implementation.
  - rapsey 5y ago
    The storage engine can be taken off the shelf. Lmdb, rocksdb etc. These are the key parts.
- EGreg 5y ago
  Is there a proven BFT version of that? RAFT aint it
  - rapsey 5y ago
    Bft?
- frakkingcylons 5y ago
  You say that like it’s trivial to build a reliable OLTP key value store once you have a correct raft implementation. Never mind real world performance or reliability.
  - rapsey 5y ago
    The hairiest bits like consensus and storage engines can however be taken off the shelf.
- deleted 5y ago
  [deleted]
- ovaistariq 5y ago
  There is more to a database than a raft implementation. A raft implementation would provide data replication between members and leader election. That is just one piece in a distributed database architecture. You then need a storage engine to persist the data, and a query engine to query the data. To be able to scale, the database would most likely be composed of several raft clusters which brings couple of other challenges: - You need a sharding layer so that the queries can be routed to the right cluster - You need an orchestrator to be able to manage these clusters It gets trickier as you try to run transactions across multiple raft clusters. You end up needing to implement 2PC. FoundationDB solves all of the above except for the query engine part (given it is a KV).
dang 5y ago
Kind of interesting to list the past threads. Were there other major ones? I omitted a few trivial ones. FoundationDB: A distributed unbundled transactional key value store - https://news.ycombinator.com/item?id=28740497 https://news.ycombinator.com/item?id=28740497 - Oct 2021 (44 comments) FoundationDB: A distributed, unbundled, transactional key value store [pdf] - https://news.ycombinator.com/item?id=27424605 https://news.ycombinator.com/item?id=27424605 - June 2021 (99 comments) Swizzle Clogging - https://news.ycombinator.com/item?id=25176931 https://news.ycombinator.com/item?id=25176931 - Nov 2020 (7 comments) Testing Distributed Systems with Deterministic Simulation (2014) [video] - https://news.ycombinator.com/item?id=23289593 https://news.ycombinator.com/item?id=23289593 - May 2020 (8 comments) Redwood: The Future of the FoundationDB Storage Engine - https://news.ycombinator.com/item?id=20498836 https://news.ycombinator.com/item?id=20498836 - July 2019 (5 comments) Apple Open-Sources FoundationDB Record Layer - https://news.ycombinator.com/item?id=18910748 https://news.ycombinator.com/item?id=18910748 - Jan 2019 (32 comments) FoundationDB Record Layer - https://news.ycombinator.com/item?id=18906341 https://news.ycombinator.com/item?id=18906341 - Jan 2019 (78 comments) FoundationDB Summit: video roundup - https://news.ycombinator.com/item?id=18826096 https://news.ycombinator.com/item?id=18826096 - Jan 2019 (3 comments) FoundationDB Document Layer - https://news.ycombinator.com/item?id=18563179 https://news.ycombinator.com/item?id=18563179 - Nov 2018 (35 comments) FoundationDB 6.0 released, featuring multi-region support - https://news.ycombinator.com/item?id=18488879 https://news.ycombinator.com/item?id=18488879 - Nov 2018 (70 comments) FoundationDB Summit Program Announced - https://news.ycombinator.com/item?id=18304906 https://news.ycombinator.com/item?id=18304906 - Oct 2018 (23 comments) FoundationDB high contention allocator - https://news.ycombinator.com/item?id=17817946 https://news.ycombinator.com/item?id=17817946 - Aug 2018 (9 comments) FoundationDB community highlights, two weeks in - https://news.ycombinator.com/item?id=16994579 https://news.ycombinator.com/item?id=16994579 - May 2018 (49 comments) Snowflake Metadata powered by FoundationDB - https://news.ycombinator.com/item?id=16880379 https://news.ycombinator.com/item?id=16880379 - April 2018 (8 comments) Apple open-sources FoundationDB - https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395 - April 2018 (441 comments) FoundationDB's Lesson: A Fast Key-Value Store Is Not Enough - https://news.ycombinator.com/item?id=9306297 https://news.ycombinator.com/item?id=9306297 - April 2015 (32 comments) Apple Acquires FoundationDB - https://news.ycombinator.com/item?id=9259986 https://news.ycombinator.com/item?id=9259986 - March 2015 (376 comments) FoundationDB and the New NoSQL - https://news.ycombinator.com/item?id=8745333 https://news.ycombinator.com/item?id=8745333 - Dec 2014 (19 comments) Databases at 14.4Mhz - https://news.ycombinator.com/item?id=8729420 https://news.ycombinator.com/item?id=8729420 - Dec 2014 (81 comments) The Future of NoSQL - https://news.ycombinator.com/item?id=8599198 https://news.ycombinator.com/item?id=8599198 - Nov 2014 (24 comments) Testing Distributed Systems with Deterministic Simulation [video] - https://news.ycombinator.com/item?id=8345030 https://news.ycombinator.com/item?id=8345030 - Sept 2014 (6 comments) On Lowered Expectations: Transactions, Scaling, and Honesty - https://news.ycombinator.com/item?id=7869623 https://news.ycombinator.com/item?id=7869623 - June 2014 (5 comments) FoundationDB vs. Jepsen - https://news.ycombinator.com/item?id=7467037 https://news.ycombinator.com/item?id=7467037 - March 2014 (23 comments) Google F1 vs. FoundationDB's SQL Layer - https://news.ycombinator.com/item?id=6718282 https://news.ycombinator.com/item?id=6718282 - Nov 2013 (12 comments) Which modern databases support ACID transactions? - https://news.ycombinator.com/item?id=6636541 https://news.ycombinator.com/item?id=6636541 - Oct 2013 (54 comments) FoundationDB 1.0 - https://news.ycombinator.com/item?id=6244631 https://news.ycombinator.com/item?id=6244631 - Aug 2013 (6 comments) FoundationDB marries NoSQL & SQL with Akiban acquisition - https://news.ycombinator.com/item?id=6057440 https://news.ycombinator.com/item?id=6057440 - July 2013 (3 comments) FoundationDB Fault Tolerance Demo Video - https://news.ycombinator.com/item?id=5739721 https://news.ycombinator.com/item?id=5739721 - May 2013 (15 comments) A NoSQL Database with ACID Transactions - https://news.ycombinator.com/item?id=5318307 https://news.ycombinator.com/item?id=5318307 - March 2013 (55 comments) FoundationDB — Not Your Standard NoSQL Database - https://news.ycombinator.com/item?id=4503400 https://news.ycombinator.com/item?id=4503400 - Sept 2012 (17 comments) FoundationDB: A new generation of NoSQL database - https://news.ycombinator.com/item?id=4294719 https://news.ycombinator.com/item?id=4294719 - July 2012 (73 comments)
tiffanyh 5y ago
I feel dumb asking, what’s the common use case for FDB. And how does it compare to Redis? I’ve read lots about how fantastic FDB (and unique) but I’m struggling to understand the use case / problem to use it for.
- bbulkow 5y ago
  Redis loses data, it has to be a cache. And it range scans only on individual machines. Eg, it doesn't scale, you have to scale it. Foundation is transactional correct on a cluster, and supports range scans.
  - mkranjec 5y ago
    Care to comment a bit further about losing data? There are two different persistence methods offered by Redis at the moment. https://redis.io/topics/persistence https://redis.io/topics/persistence
- snissn 5y ago
  from website "Multi-model data store. FoundationDB is multi-model, meaning you can store many types data in a single database. All data is safely stored, distributed, and replicated in the Key-Value Store component. Easily scalable and fault tolerant. FoundationDB is easy to install, grow, and manage. It has a distributed architecture that gracefully scales out, and handles faults while acting like a single ACID database." I've never used it personally but the claims are ACID transactions, persistence, distributed, and reliable.. So basically if you want something like Redis, but also want transactions, scale up across multiple machines and higher reliability.
- deleted 5y ago
  [deleted]
- manigandham 5y ago
  Most apps can use a single-instance relational database or document store. Fast enough with plenty of features. When requirements increase, there are many distributed relational and document stores that provide more scalability and performance. Key/value stores offer a low-level API to handle extreme requirements when nothing else works. An example is adtech platforms where user profiles and contextual data need to be accessed in a fraction of a second to respond to an ad request, often hitting queries-per-second in the 6/7 figure range. Another example is CDNs that need to lookup a specific piece of data across a large dataset very quickly. As of 2022, there are plenty of key/value stores like Redis, HBase/Bigtable, Aerospike, ScyllaDB that have various trade-offs in speed, scale and flexibility. What sets FDB apart is strong consistency (supporting transactions across multiple keys) and extreme reliability which makes it a good choice as a core metadata and coordination service that other complex apps and distributed systems can rely on. For example, Apple uses it as a backend for CloudKit, and Snowflake uses it to store the metadata and operations of its cloud data warehouse. Here's a video playlist from several other companies talking about it at FoundationDB Summit: https://www.youtube.com/playlist?list=PLbzoR-pLrL6oWYrC950yAhbLk8FRRB_Bt https://www.youtube.com/playlist?list=PLbzoR-pLrL6oWYrC950yA...
gigatexal 5y ago
FoundationDB also powers CloudKit
- dominotw 5y ago
  and snowflake
didip 5y ago
Is there a FoundationDB Helm chart?
- transactional 5y ago
  I understand nothing of helm, but https://github.com/FoundationDB/fdb-kubernetes-operator/pull/59 https://github.com/FoundationDB/fdb-kubernetes-operator/pull... seems to suggest yes?
justanotheratom 5y ago
Is there a managed version of Foundation DB?
- sortalongo 5y ago
  Not as far as I know. The primary users & contributors are apple and snowflake. It was only made OSS a few years ago, and there were already a number of comparable alternatives (scalable, consistent OLTP like Cockroach, Yugabyte, Fauna) with companies behind them.
kitd 5y ago
TIL it's SQLite under the covers! (for now at least)
- danpalmer 5y ago
  I suspect the page talking about that might be out of date as in 2018 at a conference about FDB the team were talking about the phasing out of SQLite in favour of their more optimised Redwood storage backend. I would assume that has been completed by now.
  - native_samples 5y ago
    Not necessarily. One of the unfortunate problems with FDB is the glacial release process. Check their blog and announcement forums and you'll see the last release they felt worthy of announcing was in 2019. GitHub releases shows an enigmatically named 7.0 release but given it doesn't have any release notes and the website still claims 6.3 is the latest version, it may still not be ready to go yet.
    - jen20 5y ago
      > One of the unfortunate problems with FDB is the glacial release process. One person’s “glacial release process” is another’s “stable production-grade database”. Not everything has to rev like an npm library…
      - native_samples 5y ago
        There's a balance, though. Three years without an announced release would be a long time even in the enterprise world. It's not like FDB is perfect. The way it handles cross-region replication in particular is not ideal.
    - transactional 5y ago
      (Disclaimer: affiliated) Redwood isn't "completed", but it's not unavailable. Evaluating it via `fdbcli> configure ssd-redwood-1-experimental ` is encouraged, but it hasn't seen sufficiently deep evaluation and verification that it's been set as the default ssd storage engine yet. Not all releases got blog posts. Which ones did honestly had far more to do with the people involved in the releases at the time than any technical merits. This is the first feedback that I've seen where the blog posts were used as a signal of worthiness or stability, so we'll see if we can try to be a bit more responsible about making release posts. Releases that are posted to the downloads page are all equally considered "ready to go". They appear a bit slower than what gets tagged on github, as the production environments in the core set of companies supporting FDB development are used as the last stage of QA before an official release. (Though this distinction might be less clear as the downloads page was recently redirected to the github release artifacts page.) You can find the 7.0 release notes at https://github.com/apple/foundationdb/blob/main/documentation/sphinx/source/release-notes/release-notes-700.rst https://github.com/apple/foundationdb/blob/main/documentatio..., which will appear in the documentation once the public site is updated to 7.0 (which happens once the first official, public release is blessed). All that said, the regular release cadence so far has been about every 6 months, which I think still does qualify as "glacial".
shrumm 5y ago
I was wondering who else uses FDB beyond Apple and Snowflake. Found that CouchDB is rewriting their storage layer to use FoundationDB for their v4 release. That said I couldn’t find much info on progress or release dates. https://speakerdeck.com/wohali/couchdb-4-dot-0-1-plus-2-equals-4 https://speakerdeck.com/wohali/couchdb-4-dot-0-1-plus-2-equa...
- yakak 5y ago
  IBM was funding that work and just recently announced that they no longer want to go ahead with the couchdb-fdb implementation and will refocus their support toward V3 related tasks. I'm not sure of their reasons but Apple recently backed away from things like providing windows fdb binaries and had some problems with their foundation site during the jlog incident that may have been interpreted as a lack of corporate commitment toward costs like hosting for the FOSS community, so IBM may have decided there was too much risk in having this dependency.
  - shrumm 5y ago
    That's disappointing. CouchDB itself has also dropped off the radar these days which may have been a consideration too in terms of funding ROI.
    - yakak 5y ago
      Yeah, though it's hard to say which parts matter to IBM's bottom line since in terms of cloud DB sales they must be relying more on the trapped audience "not getting fired for choosing IBM" than on upsell to developers who try couchdb. Really around 2.0 things like pouchDB were drowned out by a commercially funded choices and things more directly sponsored to work with each of the frameworks.
- tilolebo 5y ago
  Wavefront is also using hit at scale. From the list of talks at the 2019 FDB Summit, it also seems to be used at Segment, eBay and Uber
philovivero 5y ago
I was watching the old 2014 video: https://youtu.be/4fFDFbi3toc?t=1081 https://youtu.be/4fFDFbi3toc?t=1081 I've included the timestamp of the relevant bit. He is talking about swizzling, which is like clogging++. He says they take a subset of the network connections, and have them timeout, but then bring them all back in reverse order, and that this somehow exposes far more bugs than other methods. This seems like a very interesting "well, that's funny!" moment that might lead to understanding something foundational about how distributed systems fail. Does anyone have more insight on this?
- hinkley 5y ago
  That sounds reasonable to me. In a cascading failure where network congestion ramps up (eg, old school NFS), the machines with the worst network bandwidth or latency would time out first, and be the last to recover. In a heterogeneous cluster, I might expect the fastest machines to die last, and either a human operator concentrating on getting the most capacity back online first, or just automation getting the fastest one online first, since a faster machine can go from hard down to green health check sooner than the slow machine. Not precisely reverse chronological order, but certainly with large statistical clusters. And on top of that, exponential back-off would cause the same phenomenon. But if your question is "why more bugs that way?" your guess is as good as mine. Mine is 'consensus is hard.'.

197 points by eatonphil 5y ago

skyde 5y ago

For Anyone with experience with FoundationDB! How does it compare to CockroachDB ?

eatonphil 5y ago
FoundationDB is just a key value store, so it's not comparable. There is a relational layer you can run though called Foundation Record Layer. But this doesn't support SQL AFAICT. The examples on Github just show a query builder interface in Java [0]. There does seem to be a Lucene interface but it doesn't seem to be used [1]. [0] https://github.com/FoundationDB/fdb-record-layer/blob/main/examples/src/main/java/com/apple/foundationdb/record/sample/Main.java https://github.com/FoundationDB/fdb-record-layer/blob/main/e... [1] https://forums.foundationdb.org/t/lucene-layer-on-foundationdb/1229/5 https://forums.foundationdb.org/t/lucene-layer-on-foundation...
- StreamBright 5y ago
  There used to be some SQL layer: https://forums.foundationdb.org/t/sql-layer-in-foundationdb/94 https://forums.foundationdb.org/t/sql-layer-in-foundationdb/...
  - eatonphil 5y ago
    Neat, here [0] is a 3rd party one at the end of the thread but this isn't the one the thread is about. Doesn't seem like that one is public. [0] https://github.com/rustdesk/opentick https://github.com/rustdesk/opentick
    - aseipp 5y ago
      It was open source and if you Google a bit you can find forks with the relevant code snapshots, here's one: https://github.com/AydinSakar/sql-layer https://github.com/AydinSakar/sql-layer
- tonyhb 5y ago
  It's comparable based off of the fundamentals and I have the same question. They both use KVs using LSMs, do range queries, support ACID transactions, handle resharding and clustering for you. They're both based off of Spanner fundamentals. They both actually have an SQL layer - foundationdb's is the record layer. Just because one has a primary SQL interface doesn't mean we can't compare. I'd really like to know a comparison of the fundamentals including hotspot management, resharding, read/write performance, resharding etc. also and have been looking for this.
  - eatonphil 5y ago
    > They both actually have an SQL layer - foundationdb's is the record layer. Can you show me their SQL layer? I was looking for it.
    - tonyhb 5y ago
      https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer is the SQL/orm like interface. This handles things in a DB like way - joins, indexes. I misspoke when I said SQL regarding this - sorry. There's a go and rust library floating around that's not as good. I've tried em :)
      - eatonphil 5y ago
        The readme says > Queries - The Record Layer does not provide a query language, however it provides query APIs with the ability to scan, filter, and sort across one or more record types, and a query planner capable of automatic selection of indexes. So I'm not sure how you can call that a SQL layer when SQL is a query language and FDB Record Layer says it doesn't have a query language?
        tonyhb 5y ago
        Yeah, I did say "I misspoke when I said SQL regarding this - sorry.". I should have said "SQL-like", or at least relation-like. There were a couple SQL-ish things I found in other languages and got them mixed up. Sorry about that :)
ledgerdev 5y ago
I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltactivedata.com/blog/2015/04/foundationdbs-le... Also this, excellent comparison of distributed database consistency methods, mentions cockroach, but not foundation. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst... This brings up the question, what method/protocol does foundation use for distributed consistency?
- richieartoul 5y ago
  "What method/protocol does foundation use for distributed consistency?" This is a really tough question to answer succinctly. It took me a long time to wrap my head around it, but the TLDR is: 1. It uses a quorum of Coordinator nodes to elect another node (via Paxos) whose job it is to then assigns roles to other nodes. 2. Some of those roles are replicated (Like TLogs) and some of those roles are singletons (like the thing that advances the clock and hands out versionstamps). 3. If any node in the "transaction subsystem" ever fails, the cluster goes through a recovery where it becomes unavailable for ~3s during which it reassigns the roles and keeps chugging. In terms of what it looks like for a transaction to actually commit, thats another long conversation. Its hard to draw analogies to existing systems because it really is kind of its own thing.
  - ledgerdev 5y ago
    Awesome answer, thanks! Foundation seems like magic almost, from just choosing very smart trade offs and co-ordination.
    - richieartoul 5y ago
      Yeah that’s basically it. It really is a wonderful piece of free software that was (and still is) way ahead of its time in many ways. They basically made a ton of good architectural decisions really early on and then just nailed down the details super tight over many years. Their technique of running cluster-wide recoveries to handle failures is a great example. Most distributed systems engineers find the idea of designing a database that becomes unavailable (even for a few seconds) when a single node is lost completely appalling, but it turns out this small trade off leads to massive simplifications elsewhere and the resulting system is net/net much more reliable than anything else I’ve ever seen.
manigandham 5y ago
FoundationDB is a key/value store, and well-known for being fast, scalable and incredibly reliable. It passed many of the distributed database tests unlike many competitors. Here's a good HN thread when Apple open-sourced it a few years ago: https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395 CockroachDB implements postgres-compatible SQL layer on top of their own key/value store called Pebble. This is similar to TiDB which implements mysql on top of their TiKV key/value store. The latter databases offer easier programming with SQL interfaces, but since you can model just about anything on top of key/value store, there are already "layers" that can implement similar abstractions on top of FDB like the Record Layer[1], Document Layer [2], and even a SQL layer [3] at one point. 1) https://www.foundationdb.org/blog/announcing-record-layer/ https://www.foundationdb.org/blog/announcing-record-layer/ 2) https://foundationdb.github.io/fdb-document-layer/ https://foundationdb.github.io/fdb-document-layer/ 3) https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer
- EGreg 5y ago
  What is better, FoundationDB, CockroachDB, or Hypercore?
  - vvern 5y ago
    One of these is not like the others.
- atombender 5y ago
  An important difference between CockroachDB and TiDB is that the former has an embedded key-value store; everything runs in the same process. TiDB is split into multiple parts: TiDB proper (stateless SQL/query plan layer), TiKV (the key-value store), and PD (placement driver, the main control plane and metadata layer). TiKV also isn't a pure key-value store, as it handles some particular query operations that can be "pushed down" from TiDB to reduce network round trips. FoundationDB's role similar to TiKV. It's a pure key-value store with replication built in.
- jandrewrogers 5y ago
  There is one caveat I would add. While you can always build just about any kind of database or data model on top of a logical KV store, the scalability and performance is almost always going to be qualitatively worse for queries than using a storage engine purpose-built for the intended type of database or data model. It is simpler to implement, which is why people do it, but you give up a lot of selectivity and locality on the query side. The target design case for KV stores is when all your queries are essentially trivial. Layering SQL on top of it implies the expected use case is complex queries of the type where a KV store won't perform well. Good for a compatibility interface, but not performant.
richieartoul 5y ago
I've never used CockroachDB, but FoundationDB has been an absolute dream to work with. There are a few downsides to using it: - There is (currently) no hosted solution you can use, so you have to run it yourself. - There is no great "layer" that provides a nice document/SQL layer so you kind of have to work with the raw sorted KV abstraction. - You have to structure your workload to avoid a high rate of conflicts on individual keys. - It won't scale to extremely high write volume workloads like Cassandra/Scylla will. That said, in exchange you get: - Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. Jepsen refuses to test FDB because FDB's simulation testing is much more rigorous than anything Jepsen does, but AFAIK the most recent releases of Cockroach/Mongo/Cassandra/Fauna/etc have never fully passed a Jepsen test. - Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput. - A strongly consistent store you can actually use in anger and that you don't have to baby. We've never managed to actually break a FoundationDB cluster, the worst case scenario we've ever seen is that we slam it with too many writes/reads and it starts to apply backpressure by increasing latency. - Write/Read throughput scales ~linearly as nodes are added. - Built in backup/restore from blob storage. - Strongly consistent disaster recovery. This one is amazing and its hard to go back to other databases that don't support it, but basically you can run 2 FDB clusters in DR mode where one is the leader (accepting writes + reads) and the other is passively replicating from it. You can then fail over from the primary FDB to the secondary *consistently* in such a way that the two clusters coordinate to lock the primary and then unlock the secondary so that your application is guaranteed to not see any inconsistency or data loss with no participation from the application required (except teaching it how to talk to 2 different FDB clusters at once). Basically its lacking some niceties for building traditional applications on top of it, but it is an amazing primitive for building rock solid distributed systems and will almost never let you down once you've developed some operational experience with it. Its the closest thing to "magic" that I've ever worked with in the distributed storage space.
- sigstoat 5y ago
  to the folks asking why you'd use FDB, i'd highlight the main points as: > Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. the transaction guarantees on FDB are just the best there is in any product you can get for free. not to fault the rest, FDB just went past what ~98% of folks will ever need. but if you're in that last 2%, god speed trying to get anything else to offer you the same semantics. > A strongly consistent store you can actually use in anger and that you don't have to baby. i've seen some folks trying to run a cluster without proper monitoring, who were nearly driving the poor thing into the ground, with hosts dying, and having so much pressure put on them they failed out, but it never lost anything. i don't think it ever even became unavailable. > Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput eh, in defense of everything else, FDB is a key value store, and the keys are byte strings, so the situation is a bit easier for them.

chrisweekly 5y ago

FTA: > FoundationDB, a transactional key-value store that supports multi-key strictly serializable transactions across its entire key-space. FoundationDB (FDB, for short) is opensource. The paper says that: "FDB is the underpinning of cloud infrastructure at Apple, Snowflake and other companies, due to its consistency, robustness and availability for storing user data, system metadata and configuration, and other critical information." > The main idea in FDB is to decouple transaction processing from logging and storage. Such an unbundled architecture enables the separation and horizontal scaling of both read and write handling.

deleted 5y ago
[deleted]
- deleted 5y ago
  [deleted]

rapsey 5y ago

Fdb's main selling point was reliability. But with the proliferation of good raft/paxos implementations that I think is not that much of an advantage anymore.

manigandham 5y ago
There's a lot more to reliability than just the consensus algo and implementation.
- rapsey 5y ago
  The storage engine can be taken off the shelf. Lmdb, rocksdb etc. These are the key parts.
EGreg 5y ago
Is there a proven BFT version of that? RAFT aint it
- rapsey 5y ago
  Bft?
frakkingcylons 5y ago
You say that like it’s trivial to build a reliable OLTP key value store once you have a correct raft implementation. Never mind real world performance or reliability.
- rapsey 5y ago
  The hairiest bits like consensus and storage engines can however be taken off the shelf.
deleted 5y ago
[deleted]
ovaistariq 5y ago
There is more to a database than a raft implementation. A raft implementation would provide data replication between members and leader election. That is just one piece in a distributed database architecture. You then need a storage engine to persist the data, and a query engine to query the data. To be able to scale, the database would most likely be composed of several raft clusters which brings couple of other challenges: - You need a sharding layer so that the queries can be routed to the right cluster - You need an orchestrator to be able to manage these clusters It gets trickier as you try to run transactions across multiple raft clusters. You end up needing to implement 2PC. FoundationDB solves all of the above except for the query engine part (given it is a KV).

dang 5y ago

Kind of interesting to list the past threads. Were there other major ones? I omitted a few trivial ones. FoundationDB: A distributed unbundled transactional key value store - https://news.ycombinator.com/item?id=28740497 https://news.ycombinator.com/item?id=28740497 - Oct 2021 (44 comments) FoundationDB: A distributed, unbundled, transactional key value store [pdf] - https://news.ycombinator.com/item?id=27424605 https://news.ycombinator.com/item?id=27424605 - June 2021 (99 comments) Swizzle Clogging - https://news.ycombinator.com/item?id=25176931 https://news.ycombinator.com/item?id=25176931 - Nov 2020 (7 comments) Testing Distributed Systems with Deterministic Simulation (2014) [video] - https://news.ycombinator.com/item?id=23289593 https://news.ycombinator.com/item?id=23289593 - May 2020 (8 comments) Redwood: The Future of the FoundationDB Storage Engine - https://news.ycombinator.com/item?id=20498836 https://news.ycombinator.com/item?id=20498836 - July 2019 (5 comments) Apple Open-Sources FoundationDB Record Layer - https://news.ycombinator.com/item?id=18910748 https://news.ycombinator.com/item?id=18910748 - Jan 2019 (32 comments) FoundationDB Record Layer - https://news.ycombinator.com/item?id=18906341 https://news.ycombinator.com/item?id=18906341 - Jan 2019 (78 comments) FoundationDB Summit: video roundup - https://news.ycombinator.com/item?id=18826096 https://news.ycombinator.com/item?id=18826096 - Jan 2019 (3 comments) FoundationDB Document Layer - https://news.ycombinator.com/item?id=18563179 https://news.ycombinator.com/item?id=18563179 - Nov 2018 (35 comments) FoundationDB 6.0 released, featuring multi-region support - https://news.ycombinator.com/item?id=18488879 https://news.ycombinator.com/item?id=18488879 - Nov 2018 (70 comments) FoundationDB Summit Program Announced - https://news.ycombinator.com/item?id=18304906 https://news.ycombinator.com/item?id=18304906 - Oct 2018 (23 comments) FoundationDB high contention allocator - https://news.ycombinator.com/item?id=17817946 https://news.ycombinator.com/item?id=17817946 - Aug 2018 (9 comments) FoundationDB community highlights, two weeks in - https://news.ycombinator.com/item?id=16994579 https://news.ycombinator.com/item?id=16994579 - May 2018 (49 comments) Snowflake Metadata powered by FoundationDB - https://news.ycombinator.com/item?id=16880379 https://news.ycombinator.com/item?id=16880379 - April 2018 (8 comments) Apple open-sources FoundationDB - https://news.ycombinator.com/item?id=16877395 https://news.ycombinator.com/item?id=16877395 - April 2018 (441 comments) FoundationDB's Lesson: A Fast Key-Value Store Is Not Enough - https://news.ycombinator.com/item?id=9306297 https://news.ycombinator.com/item?id=9306297 - April 2015 (32 comments) Apple Acquires FoundationDB - https://news.ycombinator.com/item?id=9259986 https://news.ycombinator.com/item?id=9259986 - March 2015 (376 comments) FoundationDB and the New NoSQL - https://news.ycombinator.com/item?id=8745333 https://news.ycombinator.com/item?id=8745333 - Dec 2014 (19 comments) Databases at 14.4Mhz - https://news.ycombinator.com/item?id=8729420 https://news.ycombinator.com/item?id=8729420 - Dec 2014 (81 comments) The Future of NoSQL - https://news.ycombinator.com/item?id=8599198 https://news.ycombinator.com/item?id=8599198 - Nov 2014 (24 comments) Testing Distributed Systems with Deterministic Simulation [video] - https://news.ycombinator.com/item?id=8345030 https://news.ycombinator.com/item?id=8345030 - Sept 2014 (6 comments) On Lowered Expectations: Transactions, Scaling, and Honesty - https://news.ycombinator.com/item?id=7869623 https://news.ycombinator.com/item?id=7869623 - June 2014 (5 comments) FoundationDB vs. Jepsen - https://news.ycombinator.com/item?id=7467037 https://news.ycombinator.com/item?id=7467037 - March 2014 (23 comments) Google F1 vs. FoundationDB's SQL Layer - https://news.ycombinator.com/item?id=6718282 https://news.ycombinator.com/item?id=6718282 - Nov 2013 (12 comments) Which modern databases support ACID transactions? - https://news.ycombinator.com/item?id=6636541 https://news.ycombinator.com/item?id=6636541 - Oct 2013 (54 comments) FoundationDB 1.0 - https://news.ycombinator.com/item?id=6244631 https://news.ycombinator.com/item?id=6244631 - Aug 2013 (6 comments) FoundationDB marries NoSQL & SQL with Akiban acquisition - https://news.ycombinator.com/item?id=6057440 https://news.ycombinator.com/item?id=6057440 - July 2013 (3 comments) FoundationDB Fault Tolerance Demo Video - https://news.ycombinator.com/item?id=5739721 https://news.ycombinator.com/item?id=5739721 - May 2013 (15 comments) A NoSQL Database with ACID Transactions - https://news.ycombinator.com/item?id=5318307 https://news.ycombinator.com/item?id=5318307 - March 2013 (55 comments) FoundationDB — Not Your Standard NoSQL Database - https://news.ycombinator.com/item?id=4503400 https://news.ycombinator.com/item?id=4503400 - Sept 2012 (17 comments) FoundationDB: A new generation of NoSQL database - https://news.ycombinator.com/item?id=4294719 https://news.ycombinator.com/item?id=4294719 - July 2012 (73 comments)

tiffanyh 5y ago

I feel dumb asking, what’s the common use case for FDB. And how does it compare to Redis? I’ve read lots about how fantastic FDB (and unique) but I’m struggling to understand the use case / problem to use it for.

bbulkow 5y ago
Redis loses data, it has to be a cache. And it range scans only on individual machines. Eg, it doesn't scale, you have to scale it. Foundation is transactional correct on a cluster, and supports range scans.
- mkranjec 5y ago
  Care to comment a bit further about losing data? There are two different persistence methods offered by Redis at the moment. https://redis.io/topics/persistence https://redis.io/topics/persistence
snissn 5y ago
from website "Multi-model data store. FoundationDB is multi-model, meaning you can store many types data in a single database. All data is safely stored, distributed, and replicated in the Key-Value Store component. Easily scalable and fault tolerant. FoundationDB is easy to install, grow, and manage. It has a distributed architecture that gracefully scales out, and handles faults while acting like a single ACID database." I've never used it personally but the claims are ACID transactions, persistence, distributed, and reliable.. So basically if you want something like Redis, but also want transactions, scale up across multiple machines and higher reliability.
deleted 5y ago
[deleted]
manigandham 5y ago
Most apps can use a single-instance relational database or document store. Fast enough with plenty of features. When requirements increase, there are many distributed relational and document stores that provide more scalability and performance. Key/value stores offer a low-level API to handle extreme requirements when nothing else works. An example is adtech platforms where user profiles and contextual data need to be accessed in a fraction of a second to respond to an ad request, often hitting queries-per-second in the 6/7 figure range. Another example is CDNs that need to lookup a specific piece of data across a large dataset very quickly. As of 2022, there are plenty of key/value stores like Redis, HBase/Bigtable, Aerospike, ScyllaDB that have various trade-offs in speed, scale and flexibility. What sets FDB apart is strong consistency (supporting transactions across multiple keys) and extreme reliability which makes it a good choice as a core metadata and coordination service that other complex apps and distributed systems can rely on. For example, Apple uses it as a backend for CloudKit, and Snowflake uses it to store the metadata and operations of its cloud data warehouse. Here's a video playlist from several other companies talking about it at FoundationDB Summit: https://www.youtube.com/playlist?list=PLbzoR-pLrL6oWYrC950yAhbLk8FRRB_Bt https://www.youtube.com/playlist?list=PLbzoR-pLrL6oWYrC950yA...

gigatexal 5y ago

FoundationDB also powers CloudKit

dominotw 5y ago
and snowflake

didip 5y ago

Is there a FoundationDB Helm chart?

transactional 5y ago
I understand nothing of helm, but https://github.com/FoundationDB/fdb-kubernetes-operator/pull/59 https://github.com/FoundationDB/fdb-kubernetes-operator/pull... seems to suggest yes?

justanotheratom 5y ago

Is there a managed version of Foundation DB?

sortalongo 5y ago
Not as far as I know. The primary users & contributors are apple and snowflake. It was only made OSS a few years ago, and there were already a number of comparable alternatives (scalable, consistent OLTP like Cockroach, Yugabyte, Fauna) with companies behind them.

kitd 5y ago

TIL it's SQLite under the covers! (for now at least)

danpalmer 5y ago
I suspect the page talking about that might be out of date as in 2018 at a conference about FDB the team were talking about the phasing out of SQLite in favour of their more optimised Redwood storage backend. I would assume that has been completed by now.
- native_samples 5y ago
  Not necessarily. One of the unfortunate problems with FDB is the glacial release process. Check their blog and announcement forums and you'll see the last release they felt worthy of announcing was in 2019. GitHub releases shows an enigmatically named 7.0 release but given it doesn't have any release notes and the website still claims 6.3 is the latest version, it may still not be ready to go yet.
  - jen20 5y ago
    > One of the unfortunate problems with FDB is the glacial release process. One person’s “glacial release process” is another’s “stable production-grade database”. Not everything has to rev like an npm library…
    - native_samples 5y ago
      There's a balance, though. Three years without an announced release would be a long time even in the enterprise world. It's not like FDB is perfect. The way it handles cross-region replication in particular is not ideal.
  - transactional 5y ago
    (Disclaimer: affiliated) Redwood isn't "completed", but it's not unavailable. Evaluating it via `fdbcli> configure ssd-redwood-1-experimental ` is encouraged, but it hasn't seen sufficiently deep evaluation and verification that it's been set as the default ssd storage engine yet. Not all releases got blog posts. Which ones did honestly had far more to do with the people involved in the releases at the time than any technical merits. This is the first feedback that I've seen where the blog posts were used as a signal of worthiness or stability, so we'll see if we can try to be a bit more responsible about making release posts. Releases that are posted to the downloads page are all equally considered "ready to go". They appear a bit slower than what gets tagged on github, as the production environments in the core set of companies supporting FDB development are used as the last stage of QA before an official release. (Though this distinction might be less clear as the downloads page was recently redirected to the github release artifacts page.) You can find the 7.0 release notes at https://github.com/apple/foundationdb/blob/main/documentation/sphinx/source/release-notes/release-notes-700.rst https://github.com/apple/foundationdb/blob/main/documentatio..., which will appear in the documentation once the public site is updated to 7.0 (which happens once the first official, public release is blessed). All that said, the regular release cadence so far has been about every 6 months, which I think still does qualify as "glacial".

shrumm 5y ago

I was wondering who else uses FDB beyond Apple and Snowflake. Found that CouchDB is rewriting their storage layer to use FoundationDB for their v4 release. That said I couldn’t find much info on progress or release dates. https://speakerdeck.com/wohali/couchdb-4-dot-0-1-plus-2-equals-4 https://speakerdeck.com/wohali/couchdb-4-dot-0-1-plus-2-equa...

yakak 5y ago
IBM was funding that work and just recently announced that they no longer want to go ahead with the couchdb-fdb implementation and will refocus their support toward V3 related tasks. I'm not sure of their reasons but Apple recently backed away from things like providing windows fdb binaries and had some problems with their foundation site during the jlog incident that may have been interpreted as a lack of corporate commitment toward costs like hosting for the FOSS community, so IBM may have decided there was too much risk in having this dependency.
- shrumm 5y ago
  That's disappointing. CouchDB itself has also dropped off the radar these days which may have been a consideration too in terms of funding ROI.
  - yakak 5y ago
    Yeah, though it's hard to say which parts matter to IBM's bottom line since in terms of cloud DB sales they must be relying more on the trapped audience "not getting fired for choosing IBM" than on upsell to developers who try couchdb. Really around 2.0 things like pouchDB were drowned out by a commercially funded choices and things more directly sponsored to work with each of the frameworks.
tilolebo 5y ago
Wavefront is also using hit at scale. From the list of talks at the 2019 FDB Summit, it also seems to be used at Segment, eBay and Uber

philovivero 5y ago

I was watching the old 2014 video: https://youtu.be/4fFDFbi3toc?t=1081 https://youtu.be/4fFDFbi3toc?t=1081 I've included the timestamp of the relevant bit. He is talking about swizzling, which is like clogging++. He says they take a subset of the network connections, and have them timeout, but then bring them all back in reverse order, and that this somehow exposes far more bugs than other methods. This seems like a very interesting "well, that's funny!" moment that might lead to understanding something foundational about how distributed systems fail. Does anyone have more insight on this?

hinkley 5y ago
That sounds reasonable to me. In a cascading failure where network congestion ramps up (eg, old school NFS), the machines with the worst network bandwidth or latency would time out first, and be the last to recover. In a heterogeneous cluster, I might expect the fastest machines to die last, and either a human operator concentrating on getting the most capacity back online first, or just automation getting the fastest one online first, since a faster machine can go from hard down to green health check sooner than the slow machine. Not precisely reverse chronological order, but certainly with large statistical clusters. And on top of that, exponential back-off would cause the same phenomenon. But if your question is "why more bugs that way?" your guess is as good as mine. Mine is 'consensus is hard.'.