13 ms·

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distrib

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the really interesting part is that it provides an extremely efficient and low-level interface for any other system that needs to scalably store consistent state. At FoundationDB (the company) our initial push was to use this to write multiple different database frontends with different data models and query languages (a SQL database, a document database, etc.) which all stored their data in the same underlying system. A customer could then pick whichever one they wanted, or even pick a bunch of them and only have to worry about operating one distributed stateful thing. But if anything, that’s too modest a vision! It’s trivial to implement the Zookeeper API on top of FoundationDB, so there’s another thing you don’t have to run. How about metadata storage for a distributed filesystem? Perfect use case. How about distributed task queues? Bring it on. How about replacing your Lucene/ElasticSearch index with something that actually scales and works? Great idea! And this is why this move is actually genius for Apple too. There are a hundred such layers that could be written, SHOULD be written. But Apple is a focused company, and there’s no reason they should write them all themselves. Each one that the community produces, however, will help Apple to further leverage their investment in FoundationDB. It’s really smart. I could talk about this system for ages, and am happy to answer questions in this thread. But for now, HUGE congratulations to the FDB team at Apple and HUGE thanks to the executives and other stakeholders who made this happen. Now I’m going to go think about what layers I want to build… [1] Yes, yes, we ran Jepsen on it ourselves and found no problems. In fact, our everyday testing was way more brutal than Jepsen, I gave a talk about it here: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc

eklavya 8y ago
Sorry if I missed it but can you or someone else please link where the client wire protocol is documented?
- voidmain 8y ago
  The client is complex and needs very sophisticated testing, so there is only one implementation. All the language bindings use the C library.
  - eklavya 8y ago
    Ok, thanks :)
  - ddorian43 8y ago
    Curious why the client is complicated compared to other dbs in same space ?
    - voidmain 8y ago
      In some distributed databases the client just connects to some machine in the cluster and tells it what it wants to do. You pay the extra latency as it redirects these requests where they should go. In FDB's envisioned architecture, the "client" is usually a (stateless, higher layer) database node itself! So the client encompasses the first layer of distributed database technology, connects directly to services throughout the cluster, does reads directly (1xRTT happy path) from storage replicas, etc. It simulates read-your-writes ordering within a transaction, using a pretty complex data structure. It shares a lot of code with the rest of the database. If you wanted, you could write a "FDB API service" over the client and connect to it with a thin client, reproducing the more conventional design (but you had better have a good async RPC system!)
      - ddorian43 8y ago
        Wouldn't layers be hard to be built on the server (since you have to also change the client) and slow to be built as a layer (since it will be another separate service) ?
        qaq 8y ago
        I am guessign you'd pretty much embed the client into your higher layer
foobaw 8y ago
It is a genius move from Apple. I just wish they'd apply this logic to a lot of their other stuff.
- woolvalley 8y ago
  They are, slowly. Swift is open source, clang is open source. They are moving parts of the xcode IDE into open source, like with sourcekitd and now recently clangd. I don't think they will ever move 'secret sauce' into open source, but infrastructural things like DBs and dev tooling seems to be going in that direction.
  - mercutio2 8y ago
    Well, clangd is a Google project, which Apple has decided to start contributing to, so probably doesn’t belong on your list. Apple, like everyone else, wants to commoditize their complements.
    - deleted 8y ago
      [deleted]
    - mattkrea 8y ago
      Clang was an Apple project from the start.. I'm not sure what is telling you it is a Google project
      - gnat 8y ago
        In case a citation is needed: https://en.wikipedia.org/wiki/Clang https://en.wikipedia.org/wiki/Clang > Apple chose to develop a new compiler front end from scratch, supporting C, Objective-C and C++. This "clang" project was open-sourced in July 2007.
      - kalleboo 8y ago
        I believe the "d" in the GP comment was not a typo https://news.ycombinator.com/item?id=16874734 https://news.ycombinator.com/item?id=16874734
itp 8y ago
I have nothing specific to add to this either other than I also spent years working there, and I'm unbelievably happy about this.
voidmain 8y ago
Will said what I wanted to say, but: me too. I'm super happy about this and grateful to the team that made it happen! (I was one of the co-founders of FoundationDB-the-company and was the architect of the product for a long time. Now that it's open source, I can rejoin the community!)
- nlavezzo 8y ago
  Another (non-technical) founder here - and I echo everything voidmain just said. We built a product that is unmatched in so many important ways, and it's fantastic that it's available to the world again. Will be exciting to watch a community grow around it - this is a product that can benefit hugely from OS contributions as layers that sit on top of the core KV store.
  - lmorris84 8y ago
    I wrote a java port of the python counter many moons ago [1]. Will have to resurrect it! [1] https://github.com/leemorrisdev/foundationcounter https://github.com/leemorrisdev/foundationcounter
    - voidmain 8y ago
      It should probably be pointed out that atomic increment is in most situations a more efficient solution for high contention counters in modern FDB.
      - lmorris84 8y ago
        Ah I don’t believe that was available last time I used it - I’ll check it out thanks!
- rcgenova 8y ago
  Ditto. So glad to see FoundationDB available again! Both the tech and the company have been missed. (Former FoundationDB Solutions Engineer :) )
- 8y ago
bsaul 8y ago
Your talk was one of the best talk i've seen , and i keep mentionning it to people whenever they ask me about distributed systems, database and testing. i'm incredibly impatient to have a look at what the community is going to build on top of that very promising technology.
- srigi 8y ago
  Can you share that talk here too?
  - nunb 8y ago
    link to the [talk] from the parent comment [by wwilson] [talk] https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc [by wwilson] https://news.ycombinator.com/item?id=16877401 https://news.ycombinator.com/item?id=16877401
thefounder 8y ago
I'm not familiar with FDB but what you say sounds almost too good to be true. Can I use it to implement the Google Datastore api? I'm trying for years to find a suitable backend so that I can leave the Google land. Everything I tried either required a schema or lacked transactions or key namespaces.
- wwilson 8y ago
  As an existence proof: before the acquisition we built an ANSI SQL database and a wire-compatible clone of the MongoDB API. I see no reason you wouldn't be able to implement Datastore. In fact here's a public source claiming that Firestore (which I believe is its successor) is implemented on top of Spanner: https://www.theregister.co.uk/2017/10/04/google_backs_up_firebase_with_a_second_realtime_data_store/ https://www.theregister.co.uk/2017/10/04/google_backs_up_fir...
  - X-Istence 8y ago
    Is the wire-compatible clone of the MongoDB API available?
    - wwilson 8y ago
      It doesn't look like Apple open-sourced the Document Layer, which is a slight bummer. But I echo what Dave said below: what we got is incredible, let's not get greedy! Also TBH now that I don't have commercial reasons to push interop, if I write another document database on top of FDB, I doubt I'd make it Mongo compatible. That API is gnarly.
      - chx 8y ago
        > That API is gnarly. Out of the many MongoDB criticisms, this one is valid. This https://www.linkedin.com/pulse/mongodb-frankenstein-monster-nosql-databases-john-de-goes/ https://www.linkedin.com/pulse/mongodb-frankenstein-monster-... article is quite right about it (note: endorsing the article does not mean I endorse its author by far). Other than that, they totally did a fake it until you make it with MongoDB 3.4 passing Jepsen a year ago and MongoDB BI 2.0 containing their own SQL engine instead of wrapping PostgreSQL.
jtchang 8y ago
I just watched the demo of 5 machines and 2 getting unplugged. The remaining 3 can form a quorum. What happens if it was 3 and 3? Do they both form quorums?
- wwilson 8y ago
  A subset of the processes in a FoundationDB cluster have the job of maintaining coordination state (via disk Paxos). In any partition situation, if one of the partitions contains a majority of the coordinators then it will stay live, while minority partitions become unavailable.
  - jtchang 8y ago
    What if the number of boxes are even? Or do you get around this by not deploying an uneven number of boxes?
    - gregmac 8y ago
      This makes me wonder also what happens with multiple partitions, for example: 5 into 2, 2, and 1 7 into 3, 2, and 2
      - traviscj 8y ago
        No majorities, I guess?
    - voidmain 8y ago
      The number of coordinators is separate from the number of boxes. You don't have to have a coordinator on every box. I think you can set the number of coordinators to be even, but you never should - the fault tolerance will be strictly better if you decrease it by one.
    - maccam94 8y ago
      The majority is always N/2 + 1, where N is the number of members. A 6 member is less fault-tolerant than a 5 member cluster (quorum is 4 nodes instead of 3, and it still only allows for 2 nodes to fail).
jasode 8y ago
>It’s in widespread production use at Apple Anybody know if Apple migrated any projects from Cassandra to FoundationDB? Or was every project using FoundationDB a greenfield project?
- shawn-butler 8y ago
  There were rumors iMessage moved there but ¯\_(ツ)_/¯ Would be great to see from Apple an engineering blog "strengths and challenges at scale" post now that it has been opened again.
StreamBright 8y ago
Fantastic talk! Very engaging and insightful.
jamesblonde 8y ago
I am one of the designers of probably the best known metadata storage engine for a distributed filesystem, hopsfs - www.hops.io. When I looked at FoundationDB before Apple bought you, you supported transactions - great. But we need much more to scale. Can you tell me which of the following you have: row-level locks partition-pruned index scans non-serialized cross-partition transactions (that is, a transaction coordinator per DB node) distribution-aware transactions (hints on which TC to start a transaction on) The relative performance contribution of most of those features (cross-partition transactions not evaluated -it's a must) can be seen in our paper at Usenix Fast: https://www.usenix.org/system/files/conference/fast17/fast17-niazi.pdf https://www.usenix.org/system/files/conference/fast17/fast17...
- spullara 8y ago
  You don't need to worry about shards / partitions with FDB. Their transactions can involve any number of rows across any number of shards. It is by far the best foundation for your bespoke database.
- voidmain 8y ago
  It's somewhat hard to answer your questions because the architecture (and hence, terminology) of FoundationDB is a little different than I think you are used to. But I will give it a shot. FoundationDB uses optimistic concurrency, so "conflict ranges" rather than "locks". Each range is a (lexicographic) interval of one or more keys read or written by a transaction. The minimum granularity is a single key. FoundationDB doesn't have a feature for indexing per se at all. Instead indexes are represented directly in the key/value store and kept consistent with the data through transactions. The scalability of this approach is great, because index queries never have to be broadcast to all nodes, they just go to where the relevant part of the index is stored. FoundationDB delivers serializable isolation and external consistency for all transactions. There's nothing particularly special about transactions that are "cross-partition"; because of our approach to indexing and data model design generally we expect the vast majority of transactions to be in that category. So rather than make single-partition transactions fast and everything else very slow, we focused on making the general case as performant as possible. Transaction coordination is pretty different in FoundationDB than in 2PC-based systems. The job of determining which conflict ranges intersect is done by a set of internal microservices called "resolvers", which partition up the keyspace totally independently of the way it is partitioned for data storage. Please tell me if that leaves questions unresolved for you!
tjb1982 8y ago
That is so exciting! Can't wait to poke around. I wonder if there's anything in there I might recognize
ccifuentes 8y ago
Do we really need another rookie database system? To be honest, Postgres and MongoDB is all you need to achieve any project. I respect hobby projects though, and if that’s the case then great!
- foobarbazetc 8y ago
  Friends don’t let friends use MongoDB.
- qaq 8y ago
  https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc I am big fan of PostgreSQL but it would prob be a good idea to look into a thing you are commenting about.
- ianamartin 8y ago
  Postgres operates great as a document store, btw. You don’t really need mongo at all. And if you need to distribute because you’ve outgrown what you can do one a single postgres node, you don’t want to use mongo anyway. If you’ve read any of the comments or have been following the project, it should be pretty obvious that this is far from rookie. This is a game changer, not a hobby project. This is the first distributed data store that offers enough safety and a good enough understand of CAP theorem trade offs that it can be safely used as a primary data store.
- 15155 8y ago
  > I respect hobby projects though https://techcrunch.com/2015/03/24/apple-acquires-durable-database-company-foundationdb/ https://techcrunch.com/2015/03/24/apple-acquires-durable-dat... FDB raised >$22m and was acquired by Apple. If you consider that a "hobby project," please, teach me how to hobby.
avip 8y ago
"we also simulate dumb sysadmins". That was a really inspiring talk, thanks!
segmondy 8y ago
Or perhaps it's not so incredible? Maybe it wasn't such a huge hit for Apple and didn't leave up to expectation so they figure they can give it away and earn some community goodwill.
- banana-guy 8y ago
  there were rumors that fdb had never been adopted at Apple because of some limitations and they still ran an in-house version of cassandra
bryanrasmussen 8y ago
How would you replace a Lucene/Elasticsearch index with foundationDb?
- voidmain 8y ago
  It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.
  - fizx 8y ago
    It'd take a low number of hours to wire up FoundationDB as a Lucene filesystem (Directory) implementation. Shared filesystem with a local RAM cache has been practical for a while in Lucene, and was briefly supported then deprecated in Elasticsearch. I've used Lucene on top of HDFS and S3 quite nicely. If you have a reason to use FoundationDB over HDFS, NFS, S3, etc, then this will work well. Doing a Lucene+DB implementation where each entry posting lists are stored natively in the key-value system was explored for Lucene+Cassandra as (https://github.com/tjake/Solandra https://github.com/tjake/Solandra). It was horrifically slow, not because Cassandra was slow, but because posting lists are optimized and putting them in a generalized b-tree or LSM-tree variant will remove some locality and many of the possible optimizations. I'm still holding out some hope for a hybrid implementation where posting list ranges are stored in a kv store.
    - ddorian43 8y ago
      See elassandra for a better solandra, keeping lucene indexes and sstables separate. It should be better than keeping posting list in kv-store
    - voidmain 8y ago
      I think you are on the right track. Storing every individual (term, document, ...) in the key value store will not be efficient, but you should be able to take Lucene's nice fast immutable data structure and stuff blocks of it (at the term level or below) into FDB values very efficiently. And of course you can do caching (and represent invalidation data structures in FDB), and... FDB leaves room for a lot of creativity in optimizing higher layers. Transactions mean that you can use data structures with global invariants.
tinco 8y ago
Would it be possible to build a tree DB on top of it like MonetDB/Xquery? I always wondered why XML databases never took off, I've never seen anything else quit as powerful. Document databases if du jour seem comparatively lame.
- spullara 8y ago
  They had a document layer that wasn't released that was MongoDB compatible — so you can do this.
  - tinco 8y ago
    MongoDB is just a database with less features than a SQL database. An XML/XQuery database is fundamentally different, so I figured if FoundationDB layers are really so powerful, they might be able to model a tree DB as well.
    - spullara 8y ago
      Ah misread. Maybe? Storing hierarchical data would be pretty natural.
- CMCDragonkai 8y ago
  Yes you can. You need a tree index basically. Any kv store can serve as the backing data structure. I've been writing one for config file bidirectional transformation.
ece 8y ago
Apple also uses HBase for Siri I believe, what are some of the cluster sizes that FoundationDB scales to? Could it be used to replace HBase or Hadoop? I was in attendance at your talk, and thought it was one of the best at the conference. Apple I think broke some hearts completely going closed-source for a while, but glad to see them open sourcing a promising technology.
- mzeier 8y ago
  If scale is a function of read/writes, very large. In fact with relatively minimal (virtual) hardware it's not insane to see a cluster doing around 1M writes/second.
  - ece 8y ago
    I was talking more about large file storage like HDFS, and the MapReduce model of bringing computation to data. HBase does the latter, and it's strongly consistent like FoundationDB, though FoundationDB provides better guarantees. As a K/V I understand what you and OP say.
- killertypo 8y ago
  https://news.ycombinator.com/item?id=16879392 https://news.ycombinator.com/item?id=16879392 massive
ashishnm 8y ago
I echo what wwilson has said. I work at Snowflake Computing, we're a SQL analytics database in the cloud, and we have been using FoundationDB as our metadata store for over 4 years. It is a truly awesome product and has proven to be rock-solid over this time. It is a core piece in our architecture, and is heavily used by all our services. Some of the layers that wwilson is talking about, we've built them. metadata storage , object-mapping layer , lock manager , notifications system . In conjunction with these layers, FoundationDB has allowed us to build features that are unique to Snowflake. Check out our blog titled, "How FoundationDB powers Snowflake Metadata forward" [1] Kudos to the FoundationDB team and Apple for open sourcing this wonderful product. We're cheering you all along! And we look forward to contributing to the open source and community. [1] https://www.snowflake.net/how-foundationdb-powers-snowflake-metadata-forward/ https://www.snowflake.net/how-foundationdb-powers-snowflake-...
- ddorian43 8y ago
  Where is the object-mapping-layer ? Inside the db in c++ or a service (different process etc) outside of it.
  - ashishnm 8y ago
    Latter. It's a Java library used by our services.
- polskibus 8y ago
  That's impressive! Are you going to contribute your changes to foundationdb back?
KyleBrandt 8y ago
Know of any ideas around using this for long term Time Series data? Wonder if maybe something like OpenTSDB but with this as backend instead of hbase (which can be a sort of operational hell)
- mzeier 8y ago
  Yes :) it's the core of how Wavefront stores telemetry.
  - infinite8s 8y ago
    https://www.wavefront.com/wavefront-foundationdb-open-source-project/ https://www.wavefront.com/wavefront-foundationdb-open-source...
- wenc 8y ago
  Not quite the same but TimescaleDB is Postgres-based and is showing a lot of promise for time-series data.
sydcanem 8y ago
How does this compare with CockroachDB? I'm planning to use CockroachDB for a project but would love to get an idea if I can get better results with FoundationDB.
- capkutay 8y ago
  well FoundationDB for one doesn't resort to clickbaity pseudo-benchmark marketing tactics https://www.cockroachlabs.com/blog/performance-part-two/ https://www.cockroachlabs.com/blog/performance-part-two/
  - zzzcpan 8y ago
    They might be targeting the wrong market, hence the desperate marketing. For people who use MySQL/PostgreSQL a compatible, slower, but distributed database probably just doesn't solve any problem. Those people need a managed solution, not a distributed one.
lars_francke 8y ago
> How about replacing your Lucene/ElasticSearch index with something that actually scales and works? Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale. It's definitely interesting but I'm cautious. The track record for FoundationDB and Apple has not been great here. IIRC they acquired the company and took the software offline leaving people in the rain? Could this be like it happened with Cassandra at Facebook where they dropped the code and then more or less stopped contributing? Also I haven't seen many contributions from Apple to open-source projects like Hadoop etc. in the past few years. Looking for "@apple.com" email addresses in the mailing lists doesn't yield a lot of results. I understand that this is a different team and that people might use different E-Mail addresses and so on. In general I'm also always cautious (but open-minded) when there's lots of enthusiasm and there seems to be no downside. I'm sure FoundationDB has its dirty little secrets and it would be great to know what those are.
- jacobparker 8y ago
  > Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale. https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-... The statement was blunt but the reputation is not exactly unearned. These problems become worse proportional to scale.
deleted 8y ago
[deleted]
catwell 8y ago
> In fact, our everyday testing was way more brutal than Jepsen, I gave a talk about it here: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Unrelated to the original topic, but I had never come across that talk and it is great. I use the same basic approach to testing distributed systems (simulating all non-deterministic I/O operations) and that talk is a very good introduction to the principle.
- wenc 8y ago
  In reinforcement learning, the same approach (simulation) to addressing a data deficit is used, although the goal there is learning and not testing. When certain events happen rarely, you have fake them deterministically so that you have reproducible coverage.
jahhein 8y ago
That presentation was really good! Well explained on the simulations. If one wanted to get into this exciting event and create something with FoundationDB but no database experience (I do know many programming languages) where would I start? If anyone could point me in the direction, I'd greatly appreciate it.
edwinyzh 8y ago
How scalable and feasible to implement a SQL Layer on top of SQLite's Virtual Table Mechanism (https://www.sqlite.org/vtab.html https://www.sqlite.org/vtab.html) which redirects the read/write of the record data from/to foundationdb?
- voidmain 8y ago
  Long before we acquired Akiban, I prototyped a sql layer using (now defunct) sqlite4, which used a K/V store abstraction as its storage layer. I would guess that a virtual table implementation would be similar: easy to get working, and it would work, but the performance is never going to be amazing. To get great performance for SQL on FoundationDB you really want an asynchronous execution engine that can take full advantage of the ability to hide latency by submitting multiple queries to the K/V store in parallel. For example if you are doing a nested loop join of tables A and B you will be reading rows of B randomly based on the foreign keys in A, but you want to be requesting hundreds or thousands of them simultaneously, not one by one. Even our SQL Layer (derived from Akiban) only got this mostly right - its execution engine was not originally designed to be asynchronous and we modified it to do pipelining which can get a good amount of parallelism but still leaves something on the table especially in small fast queries.
  - edwinyzh 8y ago
    @voidmain, Thank you, it's very insightful and clear! I mean, I can see the disadvantage if such SQL layer is implemented directly through SQLite's virtual tables.

eklavya 8y ago

Sorry if I missed it but can you or someone else please link where the client wire protocol is documented?

voidmain 8y ago
The client is complex and needs very sophisticated testing, so there is only one implementation. All the language bindings use the C library.
- eklavya 8y ago
  Ok, thanks :)
- ddorian43 8y ago
  Curious why the client is complicated compared to other dbs in same space ?
  - voidmain 8y ago
    In some distributed databases the client just connects to some machine in the cluster and tells it what it wants to do. You pay the extra latency as it redirects these requests where they should go. In FDB's envisioned architecture, the "client" is usually a (stateless, higher layer) database node itself! So the client encompasses the first layer of distributed database technology, connects directly to services throughout the cluster, does reads directly (1xRTT happy path) from storage replicas, etc. It simulates read-your-writes ordering within a transaction, using a pretty complex data structure. It shares a lot of code with the rest of the database. If you wanted, you could write a "FDB API service" over the client and connect to it with a thin client, reproducing the more conventional design (but you had better have a good async RPC system!)
    - ddorian43 8y ago
      Wouldn't layers be hard to be built on the server (since you have to also change the client) and slow to be built as a layer (since it will be another separate service) ?
      - qaq 8y ago
        I am guessign you'd pretty much embed the client into your higher layer

foobaw 8y ago

It is a genius move from Apple. I just wish they'd apply this logic to a lot of their other stuff.

woolvalley 8y ago
They are, slowly. Swift is open source, clang is open source. They are moving parts of the xcode IDE into open source, like with sourcekitd and now recently clangd. I don't think they will ever move 'secret sauce' into open source, but infrastructural things like DBs and dev tooling seems to be going in that direction.
- mercutio2 8y ago
  Well, clangd is a Google project, which Apple has decided to start contributing to, so probably doesn’t belong on your list. Apple, like everyone else, wants to commoditize their complements.
  - deleted 8y ago
    [deleted]
  - mattkrea 8y ago
    Clang was an Apple project from the start.. I'm not sure what is telling you it is a Google project
    - gnat 8y ago
      In case a citation is needed: https://en.wikipedia.org/wiki/Clang https://en.wikipedia.org/wiki/Clang > Apple chose to develop a new compiler front end from scratch, supporting C, Objective-C and C++. This "clang" project was open-sourced in July 2007.
    - kalleboo 8y ago
      I believe the "d" in the GP comment was not a typo https://news.ycombinator.com/item?id=16874734 https://news.ycombinator.com/item?id=16874734

itp 8y ago

I have nothing specific to add to this either other than I also spent years working there, and I'm unbelievably happy about this.

voidmain 8y ago

Will said what I wanted to say, but: me too. I'm super happy about this and grateful to the team that made it happen! (I was one of the co-founders of FoundationDB-the-company and was the architect of the product for a long time. Now that it's open source, I can rejoin the community!)

nlavezzo 8y ago
Another (non-technical) founder here - and I echo everything voidmain just said. We built a product that is unmatched in so many important ways, and it's fantastic that it's available to the world again. Will be exciting to watch a community grow around it - this is a product that can benefit hugely from OS contributions as layers that sit on top of the core KV store.
- lmorris84 8y ago
  I wrote a java port of the python counter many moons ago [1]. Will have to resurrect it! [1] https://github.com/leemorrisdev/foundationcounter https://github.com/leemorrisdev/foundationcounter
  - voidmain 8y ago
    It should probably be pointed out that atomic increment is in most situations a more efficient solution for high contention counters in modern FDB.
    - lmorris84 8y ago
      Ah I don’t believe that was available last time I used it - I’ll check it out thanks!
rcgenova 8y ago
Ditto. So glad to see FoundationDB available again! Both the tech and the company have been missed. (Former FoundationDB Solutions Engineer :) )
8y ago

bsaul 8y ago

Your talk was one of the best talk i've seen , and i keep mentionning it to people whenever they ask me about distributed systems, database and testing. i'm incredibly impatient to have a look at what the community is going to build on top of that very promising technology.

srigi 8y ago
Can you share that talk here too?
- nunb 8y ago
  link to the [talk] from the parent comment [by wwilson] [talk] https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc [by wwilson] https://news.ycombinator.com/item?id=16877401 https://news.ycombinator.com/item?id=16877401

thefounder 8y ago

I'm not familiar with FDB but what you say sounds almost too good to be true. Can I use it to implement the Google Datastore api? I'm trying for years to find a suitable backend so that I can leave the Google land. Everything I tried either required a schema or lacked transactions or key namespaces.

wwilson 8y ago
As an existence proof: before the acquisition we built an ANSI SQL database and a wire-compatible clone of the MongoDB API. I see no reason you wouldn't be able to implement Datastore. In fact here's a public source claiming that Firestore (which I believe is its successor) is implemented on top of Spanner: https://www.theregister.co.uk/2017/10/04/google_backs_up_firebase_with_a_second_realtime_data_store/ https://www.theregister.co.uk/2017/10/04/google_backs_up_fir...
- X-Istence 8y ago
  Is the wire-compatible clone of the MongoDB API available?
  - wwilson 8y ago
    It doesn't look like Apple open-sourced the Document Layer, which is a slight bummer. But I echo what Dave said below: what we got is incredible, let's not get greedy! Also TBH now that I don't have commercial reasons to push interop, if I write another document database on top of FDB, I doubt I'd make it Mongo compatible. That API is gnarly.
    - chx 8y ago
      > That API is gnarly. Out of the many MongoDB criticisms, this one is valid. This https://www.linkedin.com/pulse/mongodb-frankenstein-monster-nosql-databases-john-de-goes/ https://www.linkedin.com/pulse/mongodb-frankenstein-monster-... article is quite right about it (note: endorsing the article does not mean I endorse its author by far). Other than that, they totally did a fake it until you make it with MongoDB 3.4 passing Jepsen a year ago and MongoDB BI 2.0 containing their own SQL engine instead of wrapping PostgreSQL.

jtchang 8y ago

I just watched the demo of 5 machines and 2 getting unplugged. The remaining 3 can form a quorum. What happens if it was 3 and 3? Do they both form quorums?

wwilson 8y ago
A subset of the processes in a FoundationDB cluster have the job of maintaining coordination state (via disk Paxos). In any partition situation, if one of the partitions contains a majority of the coordinators then it will stay live, while minority partitions become unavailable.
- jtchang 8y ago
  What if the number of boxes are even? Or do you get around this by not deploying an uneven number of boxes?
  - gregmac 8y ago
    This makes me wonder also what happens with multiple partitions, for example: 5 into 2, 2, and 1 7 into 3, 2, and 2
    - traviscj 8y ago
      No majorities, I guess?
  - voidmain 8y ago
    The number of coordinators is separate from the number of boxes. You don't have to have a coordinator on every box. I think you can set the number of coordinators to be even, but you never should - the fault tolerance will be strictly better if you decrease it by one.
  - maccam94 8y ago
    The majority is always N/2 + 1, where N is the number of members. A 6 member is less fault-tolerant than a 5 member cluster (quorum is 4 nodes instead of 3, and it still only allows for 2 nodes to fail).

jasode 8y ago

>It’s in widespread production use at Apple Anybody know if Apple migrated any projects from Cassandra to FoundationDB? Or was every project using FoundationDB a greenfield project?

shawn-butler 8y ago
There were rumors iMessage moved there but ¯\_(ツ)_/¯ Would be great to see from Apple an engineering blog "strengths and challenges at scale" post now that it has been opened again.

StreamBright 8y ago

Fantastic talk! Very engaging and insightful.

jamesblonde 8y ago

I am one of the designers of probably the best known metadata storage engine for a distributed filesystem, hopsfs - www.hops.io. When I looked at FoundationDB before Apple bought you, you supported transactions - great. But we need much more to scale. Can you tell me which of the following you have: row-level locks partition-pruned index scans non-serialized cross-partition transactions (that is, a transaction coordinator per DB node) distribution-aware transactions (hints on which TC to start a transaction on) The relative performance contribution of most of those features (cross-partition transactions not evaluated -it's a must) can be seen in our paper at Usenix Fast: https://www.usenix.org/system/files/conference/fast17/fast17-niazi.pdf https://www.usenix.org/system/files/conference/fast17/fast17...

spullara 8y ago
You don't need to worry about shards / partitions with FDB. Their transactions can involve any number of rows across any number of shards. It is by far the best foundation for your bespoke database.
voidmain 8y ago
It's somewhat hard to answer your questions because the architecture (and hence, terminology) of FoundationDB is a little different than I think you are used to. But I will give it a shot. FoundationDB uses optimistic concurrency, so "conflict ranges" rather than "locks". Each range is a (lexicographic) interval of one or more keys read or written by a transaction. The minimum granularity is a single key. FoundationDB doesn't have a feature for indexing per se at all. Instead indexes are represented directly in the key/value store and kept consistent with the data through transactions. The scalability of this approach is great, because index queries never have to be broadcast to all nodes, they just go to where the relevant part of the index is stored. FoundationDB delivers serializable isolation and external consistency for all transactions. There's nothing particularly special about transactions that are "cross-partition"; because of our approach to indexing and data model design generally we expect the vast majority of transactions to be in that category. So rather than make single-partition transactions fast and everything else very slow, we focused on making the general case as performant as possible. Transaction coordination is pretty different in FoundationDB than in 2PC-based systems. The job of determining which conflict ranges intersect is done by a set of internal microservices called "resolvers", which partition up the keyspace totally independently of the way it is partitioned for data storage. Please tell me if that leaves questions unresolved for you!

tjb1982 8y ago

That is so exciting! Can't wait to poke around. I wonder if there's anything in there I might recognize

ccifuentes 8y ago

Do we really need another rookie database system? To be honest, Postgres and MongoDB is all you need to achieve any project. I respect hobby projects though, and if that’s the case then great!

foobarbazetc 8y ago
Friends don’t let friends use MongoDB.
qaq 8y ago
https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc I am big fan of PostgreSQL but it would prob be a good idea to look into a thing you are commenting about.
ianamartin 8y ago
Postgres operates great as a document store, btw. You don’t really need mongo at all. And if you need to distribute because you’ve outgrown what you can do one a single postgres node, you don’t want to use mongo anyway. If you’ve read any of the comments or have been following the project, it should be pretty obvious that this is far from rookie. This is a game changer, not a hobby project. This is the first distributed data store that offers enough safety and a good enough understand of CAP theorem trade offs that it can be safely used as a primary data store.
15155 8y ago
> I respect hobby projects though https://techcrunch.com/2015/03/24/apple-acquires-durable-database-company-foundationdb/ https://techcrunch.com/2015/03/24/apple-acquires-durable-dat... FDB raised >$22m and was acquired by Apple. If you consider that a "hobby project," please, teach me how to hobby.

avip 8y ago

"we also simulate dumb sysadmins". That was a really inspiring talk, thanks!

segmondy 8y ago

Or perhaps it's not so incredible? Maybe it wasn't such a huge hit for Apple and didn't leave up to expectation so they figure they can give it away and earn some community goodwill.

banana-guy 8y ago
there were rumors that fdb had never been adopted at Apple because of some limitations and they still ran an in-house version of cassandra

bryanrasmussen 8y ago

How would you replace a Lucene/Elasticsearch index with foundationDb?

voidmain 8y ago
It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.
- fizx 8y ago
  It'd take a low number of hours to wire up FoundationDB as a Lucene filesystem (Directory) implementation. Shared filesystem with a local RAM cache has been practical for a while in Lucene, and was briefly supported then deprecated in Elasticsearch. I've used Lucene on top of HDFS and S3 quite nicely. If you have a reason to use FoundationDB over HDFS, NFS, S3, etc, then this will work well. Doing a Lucene+DB implementation where each entry posting lists are stored natively in the key-value system was explored for Lucene+Cassandra as (https://github.com/tjake/Solandra https://github.com/tjake/Solandra). It was horrifically slow, not because Cassandra was slow, but because posting lists are optimized and putting them in a generalized b-tree or LSM-tree variant will remove some locality and many of the possible optimizations. I'm still holding out some hope for a hybrid implementation where posting list ranges are stored in a kv store.
  - ddorian43 8y ago
    See elassandra for a better solandra, keeping lucene indexes and sstables separate. It should be better than keeping posting list in kv-store
  - voidmain 8y ago
    I think you are on the right track. Storing every individual (term, document, ...) in the key value store will not be efficient, but you should be able to take Lucene's nice fast immutable data structure and stuff blocks of it (at the term level or below) into FDB values very efficiently. And of course you can do caching (and represent invalidation data structures in FDB), and... FDB leaves room for a lot of creativity in optimizing higher layers. Transactions mean that you can use data structures with global invariants.

tinco 8y ago

Would it be possible to build a tree DB on top of it like MonetDB/Xquery? I always wondered why XML databases never took off, I've never seen anything else quit as powerful. Document databases if du jour seem comparatively lame.

spullara 8y ago
They had a document layer that wasn't released that was MongoDB compatible — so you can do this.
- tinco 8y ago
  MongoDB is just a database with less features than a SQL database. An XML/XQuery database is fundamentally different, so I figured if FoundationDB layers are really so powerful, they might be able to model a tree DB as well.
  - spullara 8y ago
    Ah misread. Maybe? Storing hierarchical data would be pretty natural.
CMCDragonkai 8y ago
Yes you can. You need a tree index basically. Any kv store can serve as the backing data structure. I've been writing one for config file bidirectional transformation.

ece 8y ago

Apple also uses HBase for Siri I believe, what are some of the cluster sizes that FoundationDB scales to? Could it be used to replace HBase or Hadoop? I was in attendance at your talk, and thought it was one of the best at the conference. Apple I think broke some hearts completely going closed-source for a while, but glad to see them open sourcing a promising technology.

mzeier 8y ago
If scale is a function of read/writes, very large. In fact with relatively minimal (virtual) hardware it's not insane to see a cluster doing around 1M writes/second.
- ece 8y ago
  I was talking more about large file storage like HDFS, and the MapReduce model of bringing computation to data. HBase does the latter, and it's strongly consistent like FoundationDB, though FoundationDB provides better guarantees. As a K/V I understand what you and OP say.
killertypo 8y ago
https://news.ycombinator.com/item?id=16879392 https://news.ycombinator.com/item?id=16879392 massive

ashishnm 8y ago

I echo what wwilson has said. I work at Snowflake Computing, we're a SQL analytics database in the cloud, and we have been using FoundationDB as our metadata store for over 4 years. It is a truly awesome product and has proven to be rock-solid over this time. It is a core piece in our architecture, and is heavily used by all our services. Some of the layers that wwilson is talking about, we've built them. metadata storage , object-mapping layer , lock manager , notifications system . In conjunction with these layers, FoundationDB has allowed us to build features that are unique to Snowflake. Check out our blog titled, "How FoundationDB powers Snowflake Metadata forward" [1] Kudos to the FoundationDB team and Apple for open sourcing this wonderful product. We're cheering you all along! And we look forward to contributing to the open source and community. [1] https://www.snowflake.net/how-foundationdb-powers-snowflake-metadata-forward/ https://www.snowflake.net/how-foundationdb-powers-snowflake-...

ddorian43 8y ago
Where is the object-mapping-layer ? Inside the db in c++ or a service (different process etc) outside of it.
- ashishnm 8y ago
  Latter. It's a Java library used by our services.
polskibus 8y ago
That's impressive! Are you going to contribute your changes to foundationdb back?

KyleBrandt 8y ago

Know of any ideas around using this for long term Time Series data? Wonder if maybe something like OpenTSDB but with this as backend instead of hbase (which can be a sort of operational hell)

mzeier 8y ago
Yes :) it's the core of how Wavefront stores telemetry.
- infinite8s 8y ago
  https://www.wavefront.com/wavefront-foundationdb-open-source-project/ https://www.wavefront.com/wavefront-foundationdb-open-source...
wenc 8y ago
Not quite the same but TimescaleDB is Postgres-based and is showing a lot of promise for time-series data.

sydcanem 8y ago

How does this compare with CockroachDB? I'm planning to use CockroachDB for a project but would love to get an idea if I can get better results with FoundationDB.

capkutay 8y ago
well FoundationDB for one doesn't resort to clickbaity pseudo-benchmark marketing tactics https://www.cockroachlabs.com/blog/performance-part-two/ https://www.cockroachlabs.com/blog/performance-part-two/
- zzzcpan 8y ago
  They might be targeting the wrong market, hence the desperate marketing. For people who use MySQL/PostgreSQL a compatible, slower, but distributed database probably just doesn't solve any problem. Those people need a managed solution, not a distributed one.

lars_francke 8y ago

> How about replacing your Lucene/ElasticSearch index with something that actually scales and works? Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale. It's definitely interesting but I'm cautious. The track record for FoundationDB and Apple has not been great here. IIRC they acquired the company and took the software offline leaving people in the rain? Could this be like it happened with Cassandra at Facebook where they dropped the code and then more or less stopped contributing? Also I haven't seen many contributions from Apple to open-source projects like Hadoop etc. in the past few years. Looking for "@apple.com" email addresses in the mailing lists doesn't yield a lot of results. I understand that this is a different team and that people might use different E-Mail addresses and so on. In general I'm also always cautious (but open-minded) when there's lots of enthusiasm and there seems to be no downside. I'm sure FoundationDB has its dirty little secrets and it would be great to know what those are.

jacobparker 8y ago
> Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale. https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html https://www.elastic.co/guide/en/elasticsearch/resiliency/cur... https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0 https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-... The statement was blunt but the reputation is not exactly unearned. These problems become worse proportional to scale.

deleted 8y ago

[deleted]

catwell 8y ago

> In fact, our everyday testing was way more brutal than Jepsen, I gave a talk about it here: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Unrelated to the original topic, but I had never come across that talk and it is great. I use the same basic approach to testing distributed systems (simulating all non-deterministic I/O operations) and that talk is a very good introduction to the principle.

wenc 8y ago
In reinforcement learning, the same approach (simulation) to addressing a data deficit is used, although the goal there is learning and not testing. When certain events happen rarely, you have fake them deterministically so that you have reproducible coverage.

jahhein 8y ago

That presentation was really good! Well explained on the simulations. If one wanted to get into this exciting event and create something with FoundationDB but no database experience (I do know many programming languages) where would I start? If anyone could point me in the direction, I'd greatly appreciate it.

edwinyzh 8y ago

How scalable and feasible to implement a SQL Layer on top of SQLite's Virtual Table Mechanism (https://www.sqlite.org/vtab.html https://www.sqlite.org/vtab.html) which redirects the read/write of the record data from/to foundationdb?

voidmain 8y ago
Long before we acquired Akiban, I prototyped a sql layer using (now defunct) sqlite4, which used a K/V store abstraction as its storage layer. I would guess that a virtual table implementation would be similar: easy to get working, and it would work, but the performance is never going to be amazing. To get great performance for SQL on FoundationDB you really want an asynchronous execution engine that can take full advantage of the ability to hide latency by submitting multiple queries to the K/V store in parallel. For example if you are doing a nested loop join of tables A and B you will be reading rows of B randomly based on the foreign keys in A, but you want to be requesting hundreds or thousands of them simultaneously, not one by one. Even our SQL Layer (derived from Akiban) only got this mostly right - its execution engine was not originally designed to be asynchronous and we modified it to do pipelining which can get a good amount of parallelism but still leaves something on the table especially in small fast queries.
- edwinyzh 8y ago
  @voidmain, Thank you, it's very insightful and clear! I mean, I can see the disadvantage if such SQL layer is implemented directly through SQLite's virtual tables.