33 ms·

Apple open-sources FoundationDB

• killertypo 9y ago
  this is amazing and this is huge for us - FoundationDB is one of the fastest most scalable distributed KV stores.
• adolph 9y ago
  "because it is an ordered key-value store, FoundationDB can use range reads to efficiently scan large swaths of data" https://apple.github.io/foundationdb/features.html https://apple.github.io/foundationdb/features.html I wonder how it compares to MUMPS databases like Intersystems Cache and FIS GtM?
  • emersonrsantos 9y ago
    Caché is more oriented towards vertical scaling, big monolithic servers, where FoundationDB is more towards horizontal scaling. Also, the obvious difference is that Caché is closed source and prohibitively expensive. FiS is only FOSS on selected platforms (Linux x86, OpenVMS Alpha), and proprietary on all other platforms.
  • chdsbd 9y ago
    It seems that MUMPS doesn't have serializable isolation.
    • adolph 9y ago
      Yeah, when I read about that I thought it sounded neat—make sure the index updates with the data. On the other hand, I think of CAP theorem as an iron triangle. If you are gaining consistency, what’s the trade off?
      • jakevn 9y ago
        This _is_ the usual trade off, but what makes FoundationDB so crazy is that it's a CP system that has a performance profile that AP systems would have a hard time matching.
  • TylerE 9y ago
    Oh, Cache... Worked a job once where that was the underlying data store. I was only allowed to touch the SQL interface to it, which was....weird. The SQL dialect was ancient, felt like something from about 1990 (and this was in.... 2012 or so, so not THAT long ago). Query performance seemed invariant. A simple select * from foo where id=X and a monster programatically generated join across 15 tables would both take about 1.5 seconds to return results.
• wll 9y ago
  What a fantastic news!
• arthursilva 9y ago
  This is great news! Unfortunately it looks like they striped out some important things, notably the storage engine (there's now a sqlite fallback). Edit: Apparently it was always sqlite as per replies bellow.
  • spullara 9y ago
    The storage engine has always been SQLite for the SSD engine.
  • fooster 9y ago
    I don't think that is correct. My understanding from using fdb for years in production was that sqlite was the storage engine.
  • voidmain 9y ago
    The storage engine is and always was a fairly heavily modified asynchronous version of sqlite's btree. It's been extremely reliable, which was always our top priority, and the performance isn't bad. But honestly when there was a problem with it our development velocity improving it wasn't great. It's super easily pluggable[1], so now that it is open source people can experiment with other engines. I think there is a lot of room for improvement. Also architecturally it's designed in anticipation of being able to run different storage engines for different key ranges and for different replicas. For example, you might keep one replica in a btree on SSD (for random reads) and two on spinning disks in a log structured engine. [1] https://github.com/apple/foundationdb/blob/master/fdbserver/IKeyValueStore.h https://github.com/apple/foundationdb/blob/master/fdbserver/... It looks to me like Apple has made a pretty complete release of the key/value store. What's missing is (1) Layers! Everything from relational databases to full text search engines to message queues (2) Monitoring stuff. Unsurprisingly it doesn't look like we have the tools for monitoring log files, etc. Wavefront (also a major user!) is a great commercial solution, but there should be something OSS
    • mzeier 9y ago
      Truthfully at Wavefront we've taken the json status directly into telegraf. Plus a bunch of python tooling to massage additional telemetry on a clusters health (coordinator reachability for example). Plus even more tooling (mostly Ansible) for managing large fleets.
• _xzxj 9y ago
  Google please take a few notes here: 1. It’s in its own repo 2. The build instructions are concise and clear. Dependencies are listed. You have to follow a total of 0 links. 3. They use a common build system and not an in-house thing.
  • itp 9y ago
    Speaking as the original author of this monstrosity of a build system, please be careful before offering praise here. To be clear, there is a top-level, non-recursive Makefile that uses the second expansion feature of GNU make, translating Visual Studio project files into generated Makefile inputs that are transformed into targets to power the build. Although it starts by running `make`, it's about as in-house as a thing can be.
    • jgh 9y ago
      Fair enough, I stand corrected. Superficially it seems like a more pleasant experience than dealing with gn/ninja as an end user.
      • itp 9y ago
        As an end user it's absolutely better. I'm just torn between pride and embarrassment thinking about how it was implemented.
        • wwilson 9y ago
          I think my favorite part of the build process was when we frobnicated libstdc++: https://github.com/apple/foundationdb/blob/master/Makefile#L159 https://github.com/apple/foundationdb/blob/master/Makefile#L... And then there were the hijinks we went through to build a cross-compiler with modern gcc and ancient libc (plus the steps to make sure no dependency on later glibc symbols snuck in): https://github.com/apple/foundationdb/blob/master/build/link-validate.sh https://github.com/apple/foundationdb/blob/master/build/link... Ahh... now that was a build system.
• protomyth 9y ago
  Apple builds their OS with a lot of software from FreeBSD, but when they opensource FoundationDB they don't provide a distribution that will work on it. I know the license says Apple doesn't have to do anything, but it just seems wrong that they didn't provide a download.
  • cnlwsu 9y ago
    https://www.foundationdb.org/download/ https://www.foundationdb.org/download/
    • jaboutboul 9y ago
      And there we go...
      • protomyth 9y ago
        I don’t see FreeBSD listed.
        • rurounijones 9y ago
          Based on your earlier comment I guess you think that Apple owes the FreeBSD community a version of FoundationDB that will work with it because apple uses FreeBSD technology. I make no comment on the validity of the stance but I think you are probably in the minority. However since it is now open source at least it is possible for someone to do the work to get it on FreeBSD at least.
        • wolfspider 9y ago
          It's probably similar to GRPC just changing the compiler over to Clang in the Makefile like so: https://github.com/wolfspider/grpc-freebsd-examples/blob/master/routeguide/Makefile https://github.com/wolfspider/grpc-freebsd-examples/blob/mas... Maybe these two would go good together? ;)
  • jaboutboul 9y ago
    I’m sure one will be shortly forthcoming. They did a good job up actually putting up the code quickly following (in parallel) with the announcement.
• Osmium 9y ago
  Original thread for when Apple acquired FoundationDB for those curious: https://news.ycombinator.com/item?id=9259986 https://news.ycombinator.com/item?id=9259986
  • edwinyzh 9y ago
    Ahah! That thread's so sad vs. this thread's so exciting :)
• wwilson 9y ago
  This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the really interesting part is that it provides an extremely efficient and low-level interface for any other system that needs to scalably store consistent state. At FoundationDB (the company) our initial push was to use this to write multiple different database frontends with different data models and query languages (a SQL database, a document database, etc.) which all stored their data in the same underlying system. A customer could then pick whichever one they wanted, or even pick a bunch of them and only have to worry about operating one distributed stateful thing. But if anything, that’s too modest a vision! It’s trivial to implement the Zookeeper API on top of FoundationDB, so there’s another thing you don’t have to run. How about metadata storage for a distributed filesystem? Perfect use case. How about distributed task queues? Bring it on. How about replacing your Lucene/ElasticSearch index with something that actually scales and works? Great idea! And this is why this move is actually genius for Apple too. There are a hundred such layers that could be written, SHOULD be written. But Apple is a focused company, and there’s no reason they should write them all themselves. Each one that the community produces, however, will help Apple to further leverage their investment in FoundationDB. It’s really smart. I could talk about this system for ages, and am happy to answer questions in this thread. But for now, HUGE congratulations to the FDB team at Apple and HUGE thanks to the executives and other stakeholders who made this happen. Now I’m going to go think about what layers I want to build… [1] Yes, yes, we ran Jepsen on it ourselves and found no problems. In fact, our everyday testing was way more brutal than Jepsen, I gave a talk about it here: https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc
  • eklavya 9y ago
    Sorry if I missed it but can you or someone else please link where the client wire protocol is documented?
    • voidmain 9y ago
• alexnewman 9y ago
  If this was done years ago it'd be a big deal. Now I worry the competition will make this a non event.
  • eropple 9y ago
    To the best of my knowledge (n.b.: I am not an expert, though I follow this field) there still aren't any direct competitors to the breadth of what FoundationDB can do and do well.
    • killertypo 9y ago
      There is not The speed and durability has gone unmatched. We can write millions of transactions per second w/ millions of reads w/o issue. We have never lost data or found data to be inconsistent.
    • nlavezzo 9y ago
      Founder here. To the best of my knowledge, you are correct. ACID is getting thrown around a lot these days, and not being used in its strictest sense.
    • throwaway84742 9y ago
      Google Spanner? Not OSS, but globally consistent, scalable, and extensively battle tested.
      • eropple 9y ago
        Spanner can do global consistency and (some?) transactions but I'm unaware of it being able to do the sort of layering Foundation can internally to expose largely different database forms on top of it. I have never used Spanner, though, so I'm open to being corrected.
        • throwaway84742 9y ago
          What do you mean by “forms”? Spanner is also layered. The bottom layer is basically a key-value store. On top of that there’s a full blown SQL layer, which, BTW can work with hierarchical records as well as flat tables. Both support transactions and guarantee global consistency.
• deleted 9y ago
  [deleted]
• angrygoat 9y ago
  Awesome - the code is up on github: https://github.com/apple/foundationdb/ https://github.com/apple/foundationdb/
• samspenc 9y ago
  How does this stack up against HBase and Cassandra, which seem to have gotten traction already in the same areas that FoundationDB seems best suited for?
  • Bahamut 9y ago
    An interesting fact is that Apple is probably one of the biggest users, if not the biggest user, of Cassandra out there. Can't speak to HBase, but one thing Cassandra doesn't guarantee is ACID - I've seen some data consistency issues that has arisen from Cassandra in our usage, although it hasn't been a huge problem for us. That difference alone probably brings a lot of value to FoundationDB.
    • Joeri 9y ago
      My understanding of hbase (which could be wrong) is that writes of a particular key always go to the region master first, so if you read from the master you always get the latest value of a key. The tricky part is that when the region master goes offline another one needs to take its place, and you can get inconsistency or unavailability depending on how it is set up. I’d like to see a deep dive of how foundationdb handles this. It has to trade-off consistency for availability at some point and it would be nice to know exactly where.
      • poooogles 9y ago
        From what I understand that shouldn't happen as writes are written to a WAL on HDFS (been a while since I've used it). From my memory writes within a row are atomic. It seems to pass Jepsen as well [1]. 1. https://www.google.co.uk/amp/s/yokota.blog/2015/09/30/call-me-maybe-hbase/amp/ https://www.google.co.uk/amp/s/yokota.blog/2015/09/30/call-m...
        • deleted 9y ago
          [deleted]
      • necubi 9y ago
• openasocket 9y ago
  I hadn't heard of FoundationDB before, so I did some digging into the features: https://apple.github.io/foundationdb/features.html https://apple.github.io/foundationdb/features.html . It seems to claim ACID transactions with serializable isolation, but also says later on that it uses MVCC, slower clients won't slow down operations, and that it allows true interactive queries. I didn't think an MVCC implementation could provide that level of isolation, and I'm not even sure how you provide that level of isolation and those other guarantees with any implementation, am I misunderstanding something?
  • saurik 9y ago
    I mean, PostgreSQL is MVCC and it clearly supports serializable as an isolation level... have you tried using Google? https://wiki.postgresql.org/wiki/Serializable https://wiki.postgresql.org/wiki/Serializable
    • saurik 9y ago
      A lot of people apparently hated that I attacked this person, and downvoted me; but what the person who posted this comment was doing is intentionally throwing shade at and casting doubt on a project by saying "I think that the claims on this webpage don't make any sense as this should be impossible". It comes off as "oh come on, this doesn't even sound plausible; I call bullshit". This then causes people who aren't versed with the product or the technology to decrease their perception of the product, and puts the team behind it in a position of having to not just come to its defense but to do so quickly due to the perception concerns. Meanwhile, if they just do a basic search for "MVCC serializable" they would find that they were wrong; which means that it took more time to leave this insulting comment than it would have taken them to learn how this can work. As a community, we really really really really need to beat down on casual cynics like this, who like to lazily "call bullshit" or play the "citation needed" card as a way to undermine the credibilty of other peoples' products. We live in a future where the answers to these kinds of doubts are a moment away: this particular form of debate tactic needs to die.
      • michaelcampbell 9y ago
• saagarjha 9y ago
  It's under Apache 2.0 for those curious: https://github.com/apple/foundationdb/blob/master/LICENSE https://github.com/apple/foundationdb/blob/master/LICENSE. Also, side note: it looks like this was a private GitHub repository for at least a couple months, since they have pull requests going back for at least that long. I find this interesting, since Apple normally "cleans up" history before open sourcing.
  • Longhanks 9y ago
    Swift, too, was published on GitHub including its whole version control history, dating back to 2010 :)
• ah- 9y ago
  This is amazing! Is there a roadmap of what is being worked on or is this mainly a dump of their existing codebase? From looking at the commit history it seems like this is pretty actively developed.
• kawera 9y ago
  Do we know where/how Apple uses FoundationDB in production ?
  • endlessvoid94 9y ago
    My first thought was...iCloud?
    • ifcaselet 9y ago
      iCloud is an umbrella term for several different systems.
      • endlessvoid94 9y ago
        Right. All I'm saying is that FoundationDB is underneath it somewhere.
  • optimuspaul 9y ago
    I had always assumed they acquired the team more than the ip. I'm not sure this confirms that or not. I'd be curious what the answer to this question is as well, but wouldn't be surprised if the answer was "they don't"
  • rudedogg 9y ago
    Maybe CloudKit? I was looking at the documentation for it recently and it's a key-value store.
    • dunham 9y ago
      CloudKit is using Cassandra/Solr: - http://www.vldb.org/pvldb/vol11/p540-shraer.pdf http://www.vldb.org/pvldb/vol11/p540-shraer.pdf
      • yoda_sl 9y ago
        From what I have heard, CloudKit is an API abstraction either of Cassandra/Solr and more recently Foundation DB. So new CloudKit codebase nowadays is being run on top of Foundation DB
• misterbowfinger 9y ago
  I hate to be that person, but when I hear "ACID transactions in a distributed database", I hear Citus/Spanner/CockroachDB. I'm positive that Citus & Spanner are quite different from FoundationDB, but I have no idea how. Googling didn't help much. Can someone provide an overview of the differences?
  • nlavezzo 9y ago
    Well for one, Spanner is not available except as a service.
  • voidmain 9y ago
    I think Citus is not really ACID. Spanner (and to an extent its less mature OSS descendants Cockroach and TiKV) has more comparable goals, but is fairly different architecturally. For example, FoundationDB only requires N+1 replicas instead of 2N+1 to achieve N failure tolerance (even lots of databases with much weaker guarantees are in the latter category!), doesn't trust clocks at all, doesn't lose performance when transactions cross replica sets, and uses optimistic instead of pessimistic concurrency. Also FoundationDB (and TiKV) make a distributed, transactional key/value store available as an API, while Spanner and Cockroach expose only a relational database layer. FoundationDB is designed philosophically with the idea that you want to have a single storage layer to manage operationally but should be able to mix and match data models and query engines above that layer. On the other hand, FoundationDB doesn't currently have any full fledged high level database layer available. Someone will probably dig up our SQL layer (which was AGPL, I think) but I wouldn't really recommend using it in production because there is no active development team. Someone will probably try porting the SQL layers from TiDB and Cockroach. Maybe Apple will open source more stuff in the future, but let's not get too greedy!
    • infogulch 9y ago
      Wait how can it be ACID if it can tolerate N failures from N+1 nodes? Doesn't that kill consistency almost by definition? What level of isolation does it support? Snapshot? Serializable?
• throwaway6497 9y ago
  Broken documentation: Architecture image is missing. https://apple.github.io/foundationdb/architecture.html https://apple.github.io/foundationdb/architecture.html Hope someone from the team reads this.
  • zzzcpan 9y ago
    It's a pdf: https://apple.github.io/foundationdb/_images/Architecture.pdf https://apple.github.io/foundationdb/_images/Architecture.pd...
    • jayrhynas 9y ago
      Looks like Safari will happily load a pdf in an img tag, but Chrome won't
      • lut4rp 9y ago
        I wonder if that's a potential security issue waiting to happen.
• pron 9y ago
  Any documents on the core algorithms?
• foota 9y ago
  This seems like it would be really great for large game servers, I'm thinking things like eve.
• djtriptych 9y ago
  I went to the same high school as the founders[1]. They were about the 2 best software engineers in a school with a LOT of very smart software engineers. Another pair founded Yext, which went public last year. I still consider that school the group with the highest concentration of raw brain power I've ever been a part of. I'm probably a 1% engineer, been hired by M$, FB, and Google. These guys were light years ahead of me. I'm not sure I'm as good now as they were at like 17 years old. In fact I'm probably only a decent engineer from having observed the stuff they were doing back then and finding inspiration. 1: https://en.wikipedia.org/wiki/Thomas_Jefferson_High_School_for_Science_and_Technology https://en.wikipedia.org/wiki/Thomas_Jefferson_High_School_f...
  • dhimes 9y ago
    That's a great school. I hired some students there when I ran a tech camp for middle-schoolers and they were...beyond.
  • mixmastamyk 9y ago
    I did chuckle at the part above about the "best software engineers in high school," but can't argue with the results.
    • qaq 9y ago
      That's a very special high school
      • Glide 9y ago
        It definitely was when I went there.
  • verulito 9y ago
    Just checked the rankings. My first high school made #172 on the list ("International School of the Americas"). My second high school is unranked ("Texas Academy of Math and Science"). I don't think it qualifies as a high school. Seems we haven't done a great job of identifying the accomplishments of our alumni, based on the Wikipedia page. No doubt it would rank near the top though. My year alone Caltech accepted about 30 of us, more than any other high school in the country. Makes me wonder what my peers have been up to. Anyway, I'd agree that these tech high schools have some amazingly smart people attending them. 2: https://en.wikipedia.org/wiki/Texas_Academy_of_Mathematics_and_Science#Notable_alumni https://en.wikipedia.org/wiki/Texas_Academy_of_Mathematics_a...
• m0meni 9y ago
  How's this compare to https://github.com/pingcap/tikv https://github.com/pingcap/tikv? It's a relatively new distributed KV store written in Rust that also is transactional and backs the new TiDB database.
  • warmwaffles 9y ago
    Well on first guess, I would assume that FoundationDB is more mature than tikv, although I have never used either but tikv looks cool
    • scv119 9y ago
      Agreed, technology-wise FDB and TIDB look almost the same.
  • muizelaar 9y ago
    Here's a brief comparison by a tikv developer: https://github.com/pingcap/tikv/issues/2986#issuecomment-383355986 https://github.com/pingcap/tikv/issues/2986#issuecomment-383...
• isoos 9y ago
  How does it compare to CockroachDB or TiDB?
  • evanweaver 9y ago
    It's closest to TiDB's key-value layer; a building block for more complex systems. More traditional, monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility for the benefits in performance and operations that come from very tight coupling. In my understanding, FoundationDB's transaction management is closest to FaunaDB's; read/write sets are linearized in memory in preprocessing nodes and distributed asynchronously to the replicas rather than locked on the replica leaders like Spanner or CockroachDB. This is why FoundationDB doesn't support long-lived transactions. It's interesting that the FoundationDB team chose to unwind their service architecture (there used to be separate transaction manager and replica processes), I assume in the interests of ease of operations. It is not clear to me how leader election and failover works for the transaction management role. Maybe somebody from the team can clarify.
    • spullara 9y ago
      I don't recall them ever having separate processes by default. You can set that up though by limiting roles.
      • evanweaver 9y ago
        Maybe it was the same binary. Pretty sure you had to operationally manage the roles separately and now they are integrated. On the other hand, I don't really know.
        • voidmain 9y ago
          No, it's always been possible to have as little as one fdbserver process and have a complete key/value store. Internally it is "microservices" though - it will start a "proxy", a "resolver", a "log", a "storage", etc within that one process.
    • 9y ago
• borplk 9y ago
  Anyone wants to give a summary of what primitives/structures/algorithms FoundationDB uses?
• fooster 9y ago
  Does anyone know if the SQL layer was open sourced, or whether there are any plans to open source it?
• ChuckMcM 9y ago
  Wow this is super exciting. I had a half dozen things that I though FDB would be good for and then poof! it got sucked up into the Apple spaceship. Now to have it emerge unscathed is pretty awesome. Another step closer to a library appliance.
• groguelon 9y ago
  Originally, FDB was a DB supporting 3 models: - KV - Document - Graph It seems the announcement concerns only the KV one. Someone has information for the 2 other ones? Thank you.
  • nlavezzo 9y ago
    The core product is a distributed, highly fault tolerant ordered Key Value store with true serializable ACID transactions. All of the layers (including document, graph) sit on top of that and inherit its ACID properties, scalability, fault tolerance, etc. It doesn't appear to me that they released any of the top level layers, but those are MUCH simpler to build, and that's where the OS community can step in.
    • sunw 9y ago
      How is FoundationDB's graph performance/feature set compared to those of other graph and multi-model databases?
      • nlavezzo 9y ago
        FoundationDB at its core is not a graph database. You could build a graph database on top of it, using FoundationDB as a very strong and feature rich storage engine, however you'd like. It would be much simpler to do than building a new (especially a distributed) graph database from scratch.
        • pluma 9y ago
          So FDB's role is more comparable to that of, say, RocksDB or LevelDB than an application-level database like Postgres? I didn't really pay much attention to Foundation before Apple bought them and am unsure how it fits in the wider database ecosystem.
          • voidmain 9y ago
            You can think of FDB as a distributed storage engine. It has the same low level data model as the engines you mention, but has distributed transactions, fault tolerance, automatic data partitioning, operational tooling, etc built in. So if you build an "application level database" or library on top of it, it is automatically a distributed database.
• dbcurtis 9y ago
  Question for the FoundationDB gurus that are hanging out on this thread: How well does it deal with spotty connectivity? I'm asking because I work on mobile robots, and WiFi and/or LTE connections are always coming and going in unpredictable ways as the vehicle moves about it's environment. Reconnecting every few minutes is normal.
  • djtriptych 9y ago
    You should have seen their trade show demos, where they'd demonstrate fault tolerance by literally powering down random computers and unplugging network cables, while visualizing how the system managed fault tolerance. ah found it: https://www.youtube.com/watch?v=oLGYMdo2q2g https://www.youtube.com/watch?v=oLGYMdo2q2g
  • voidmain 9y ago
    The fault tolerance is pretty much flawless. You won't be able to get the database "stuck" or see anomalies. But performance is going to suck if you run server nodes over unreliable connections. I have trouble seeing a FoundationDB cluster running on mobile robots as more than a trade show gimmick. Albeit an awesome gimmick. So in summary you should totally do that.
    • dbcurtis 9y ago
      I can see that. There are various vectors to performance. I hear you saying transactions per second would be unimpressive. A couple of less time sensitive applications are: 1. distrubuting information the entire fleet should eventually know, 2. event log aggregation with fine-grained time alignment among nodes. Both are probably silly problems to solve with a database, killing houseflys with sledghammers and all that, but it never hurts to explore creative tool misuse :)
      • alexis_read 9y ago
        I think Scuttlebutt might be more suitable for you? Extract the text from a conversation on each robot. The offline story is easy here. The other option would be some sort or CRDT-based system - Antidote perhaps?
• uasnew 9y ago
  How does this compare to redis?
• DimitarIbra9987 9y ago
  I found out foundationDB vienna VA office is closed. Did apple move all people to Cupertino?
• jaytaylor 9y ago
  Firstly: Wow! this is amazing news!!! I'm also kind of confused.. is the single repo complete? What about the SQL Layer [0]? Where is all this stuff in the new GH repo? Or is only the KV part be open-source? Looking forward to some CockroachDB vs. FDB benchmark showdowns :) [0] https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer
  • cooervo 9y ago
    also super interested in CockroachDB, but I just can't find enough war stories, or stories of people using it in production...
• dman 9y ago
  Noticed the Visual Studio project files. Was Windows a primary development platform for foundationdb?
  • panghy 9y ago
    Nope, the actor compiler is down with mono and your need windows to compile the windows client. When we were independently working on it, we just ditched the windows code completely (we are a mac/linux shop)
    • polskibus 9y ago
      Does it mean that client is unusable on Windows?
  • voidmain 9y ago
    Before FDB was acquired by Apple, a lot of the engineers used Visual Studio on Windows. It's a good IDE for C++. You certainly don't need it to develop though! Windows was never the most important deployment platform for the product. I'm not sure how performant the server is. But it should work, and I would think the client is fine.
• xvilka 9y ago
  These days most people know Hadoop for a distributed storage. In my opinion, though CEPH [1] has the bigger potential. [1] https://ceph.com/ https://ceph.com/
  • manigandham 9y ago
    There are several distributed storage systems like Ceph and they all have problems. Ceph is not good because it's an object storage system trying to provide block storage and a filesystem on top, which will never work well.
    • noahdesu 9y ago
      Not that this really has anything to do with FoundationDB, but why do you say that object storage is a poor substrate for file and block abstractions? There are many high-performance block and file systems built on object storage.
      • manigandham 9y ago
        Block level is the lowest form of addressing bytes on devices. Filesystems are an abstraction on top of block devices. Object stores are an abstraction on filesystems. Emulating a low-level layer on a higher-level abstraction (which itself is using this hierarchy) will never match the speed, scale, or reliability of doing it correctly.
        • noahdesu 9y ago
          As a general rule of thumb, I would definitely agree that increasing the number of abstraction layers should be done with consideration to performance concerns. Interestingly, in the latest version of Ceph the abstractions are a bit different than you listed. Ceph is now using an object store built directly on top of raw devices. It's the file system, block, and object abstractions that exist on top of that.
        • spullara 9y ago
          I mostly agree, but if you want a distributed, reliable, scalable block store it is pretty easy to build one on FDB. Here is a super simple one that acts as an NBClient server that you can mount from Linux: https://github.com/spullara/nbd https://github.com/spullara/nbd Remember to format it with something like XFS rather than Ext to avoid writing superblocks all over the place.
• DimitarIbra9987 9y ago
  is Transaction authority in-memory or on disk? this architecture seems kinda clunky. Wondering what's the performance and is this a good fit for quick k-v store use cases?
  • YongMan 9y ago
    I also wonder the performance when concurrency writes to a few keys.
• tomc1985 9y ago
  I remember reading about Apple purchasing FoundationDB and everyone being disappointed and hoping they'd open source it. And now they have!
• Rafuino 9y ago
  Found this section on performance very interesting. https://apple.github.io/foundationdb/performance.html#throughput-per-core https://apple.github.io/foundationdb/performance.html#throug... I wonder what the SSD engine performance would look with NVMe standard NAND or an Optane SSD instead of SATA. Any FoundationDB guys/gals on this thread able to comment? Another Q: what's more commonly used in current FoundationDB deployments: memory engine or storage engine?
  • panghy 9y ago
    We run both, memory engine hasn't seen too many updates in the last 3 years though. For SSDs, we use i3 as well as EBS-backed volumes.
• SubuSS 9y ago
  This is great news, when I was with dynamo, FoundationDB was the other green shore for me :). They did so many things so well. A tiny bit of caution for folks trying to run systems like this though: It is frigging hard at any reasonable scale. The whole thing might be documented / OSS and what not, but very soon you are going to run into deep enough problems that's going to require very core knowledge to debug, energy to deep dive. Both of which you probably don't want to invest your time into. Do evaluate the cloud offerings / supported offerings before spinning these up. Else ensure you have hired experts who can keep this going. They are great as a learning tool, pretty hard as an enterprise solution. I have seen the same issue a ton of times with a bunch of software (redis/kafka/cassandra/mongo...) by now. IMO In the stateful world, operating/running the damn thing is 85% of the work, 15% being active dev. (Stateless world is a little better, but still painful).
  • shawn-butler 9y ago
    This sounds like a business opportunity to me rather than a cautionary note. I remember all the people who bashed Apple when they acquired FoundationDB. I hope they are appropriately ashamed now.
    • 15155 9y ago
      > I remember all the people who bashed Apple when they acquired FoundationDB. I'm not ashamed about deriding apple for Apple taking a really, really great product and hiding it from the world for years to come. This is definitely some atonement, but does not totally absolve Apple from the many times they've taken tech private.
  • rdsubhas 9y ago
    > IMO In the stateful world, operating/running the damn thing is 85% of the work, 15% being active dev. (Stateless world is a little better, but still painful). The number of newbie engineers who see docker/kubernetes, and think "let me docker run or helm install" a stateful service in a couple of minutes - is mind boggling. I'll remember this quote when trying to talk sense to them.
• ram_rar 9y ago
  This is such a great news. I had seen, FoundationDB guys present in new england database summit couple of years ago. Their demo reminded me of sun microsystems famous demo of screwing their hard disk.
• daveheq 9y ago
  So what does Facebook get out of it?
• dboreham 9y ago
  Is it known whether Apple runs this code in production?
• helper 9y ago
  I'm very interested in hearing more about what running FoundationDB in production is like. I believe that FoundationDB stores rows in lexicographical order by key. Other databases like Cassandra strongly push you toward not storing data this way as it can easily lead to hotspots in the cluster. How do you deploy a FoundationDB cluster without leading to hotspots, or perhaps what operational actions are available to rebalance data?
  • cakoose 9y ago
    Does Cassandra hash the primary key to get a more even distribution? If you have a sorted data store, you can get the same distribution by keying off a hash of the "real" primary key, right?
    • helper 9y ago
      Cassandra allows you to configure a "partitioner" that determines which nodes a primary key belongs to. There is a ByteOrderPartitioner that stores partitions in order lexicographically by primary key. There is also a Murmor3 hash based partitioner (which is the recommended default). Cassandra allows you to store multiple records in sorted order within a partition. The normal recommended way to get data locality is to store records that are frequently accessed together in the same partition.
• stijnsanders 9y ago
  The Windows installer has Lorem Ipsum instead of EULA text... https://twitter.com/stijnsanders/status/987042633691394050 https://twitter.com/stijnsanders/status/987042633691394050
  • max23_ 9y ago
    Also, Windows Defender SmartScreen complains about unknown publisher when trying to execute the installer.
• foobarbazetc 9y ago
  Is there a canonical php client for this?
• sheeshkebab 9y ago
  It’s neat although there is no sql front end. Bloomberg’s comdb2 was open sourced recently https://github.com/bloomberg/comdb2 https://github.com/bloomberg/comdb2 - it seems similar, but would be interesting to see comparison.
  • ddorian43 9y ago
    comdb2 has no sharding ?
  • pdeva1 9y ago
    comdb2 is not a big data db per say. it compares more to mysql than foundationdb https://blog.dripstat.com/first-look-at-bloombergs-amazing-comdb2/ https://blog.dripstat.com/first-look-at-bloombergs-amazing-c...
    • sheeshkebab 9y ago
      Great write up!
• monkeydust 9y ago
  Can someone explain to business / product guy why this is so exciting? (honestly keen to understand)
  • voidmain 9y ago
    Well, it's infrastucture. Moreover, it's infrastructure for infrastructure! So if that sounds super boring you don't have to be excited :-) But I would tell the story something like this: state storage is the root of (almost) all operational evil. It's very easy to make a system reliable if it's totally stateless. Even most bugs can be lived with if the worst you have to do is restart a service and carry on! But to do anything interesting you have to store state somewhere, and you have to modify that state concurrently without screwing it up. And the many challenges of operating stateful systems are greatly multiplied if you have a lot of different ones. For example, if you have a datacenter outage and some but not all of your stateful systems deal with it correctly, probably your application as a whole is still down. So as one more stateful system, does FoundationDB just make that worse? Well, FoundationDB is designed specifically to be a foundation for many very different stateful systems - not just different kinds of databases but things like search engines or message queues that you normally don't think of in the same category. So that almost any system can map to it efficiently, it has a lowest common denominator data model (key/value) and the highest possible guarantees in terms of concurrency control. And you can run diverse systems supporting an application on the same FoundationDB cluster, or on different clusters with the same exact operational requirements. Some few users of FoundationDB have been able to get the benefits of this vision, consolidating lots of different stuff into a single, operationally desirable system. But for more people to be able to, not just does the key/value store have to be available to them, but also lots of stuff has to be built on top of it. By releasing FoundationDB under a very liberal open source license, Apple has hopefully made that possible. In the long run, hopefully it will make all server-side computing more reliable. Also, it's a really good key/value store, if you happen to need one of those!
• nlavezzo 9y ago
  For those asking about use in production, Wavefront just posted this: "Wavefront by VMware’s Ongoing Commitment to the FoundationDB Open Source Project" https://www.wavefront.com/wavefront-foundationdb-open-source-project/ https://www.wavefront.com/wavefront-foundationdb-open-source...
• panghy 9y ago
  We (Wavefront) has been operating petabyte scale clusters for the last 5 years with FoundationDB (we got the source code via escrow) and we are super excited to be involved in the opensourcing of FDB. We have operated over 50 clusters on all kinds of aws instances and I can talk about all the amazing things we have done with it. https://www.wavefront.com/wavefront-foundationdb-open-source-project/ https://www.wavefront.com/wavefront-foundationdb-open-source...
  • panghy 9y ago
    We basically replaced mySQL, Zookeeper and HBase with a single KV store that supports transactions, watches, and scales. It's not a trivial point that you can just develop code against a single API (finally Java 8 CompletableFutures) and not have to set up a ton of dependencies when you are building on top of FDB. We are (obviously) experts at monitoring FoundationDB with Wavefront and we hope to release the metric harvesting libraries and template dashboards that we use to do so. Almost 5 years in and we have not lost any data (but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS =p).
    • qaq 9y ago
      "but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS " <=== This I wish more people realized that is a day to day reality if you are in AWS at scale.
      • killertypo 9y ago
        And FoundationDB has held our data durable through all of this.
        • qaq 9y ago
          Sounds like however bolts on PG compatible SQL layer on top will have a killer product on their hands :)
• mavilein 9y ago
  Wow this is some really exciting news! I think it would be amazing to create a GraphQL API for FoundationDB. Therefore i have created a feature request for this in the Prisma repo. For those who don't know Prisma is a GraphQL database mapper for various databases. https://github.com/graphcool/prisma/issues/2240 https://github.com/graphcool/prisma/issues/2240
• deleted 9y ago
  [deleted]
• sleepyams 9y ago
  Does anyone know if the python API for fdb works with asyncio?
  • sleepyams 9y ago
    Answered my own question (seems like the answer is yes): https://github.com/apple/foundationdb/blob/master/bindings/python/fdb/impl.py#L1584 https://github.com/apple/foundationdb/blob/master/bindings/p...
• bogomipz 9y ago
  Does anyone know what services at Apple are built on top of FoundationDB?
• deleted 9y ago
  [deleted]
• Rygu 9y ago
  Found the NodeJS bindings in the 5.0 branch: https://github.com/apple/foundationdb/blob/release-5.0/bindings/nodejs/package.json.in https://github.com/apple/foundationdb/blob/release-5.0/bindi... Seems dated as it requires NodeJS either 0.8 or 0.10.
  • faizshah 9y ago
    More info on this in this issue: https://github.com/apple/foundationdb/issues/129 https://github.com/apple/foundationdb/issues/129
• NelsonMinar 9y ago
  Why did it take three years? Apple killed FoundationDB as a product in March 2015.
• osrec 9y ago
  I can see everyone's extremely happy about this, which is great. As someone who's never used it, I'd like to know more about FoundationDB and how it compares to other offerings such as MySQL or Postgres, and which use cases is it most suited to. I would especially love to hear the thoughts of those with direct experience of using Foundation DB. Thanks!
  • degenerate 9y ago
    Seconding this. Instead of hearing from bloggers or contributors, what is using this DB like in the trenches? What simple use case would fit this DB perfectly? Is there a lot of setup on second/third deploy? Easy to maintain, or requires a lot of tweaking/tuning? How is memory usage with 10M records? 100M?
  • rando444 9y ago
    Personally I'd be more interested in hearing how this compares to other distributed noSQL implementations like Cassandra.
    • roboyoshi 9y ago
      It probably is better, because Apple switched from Cassandra to FoundationDB, according to all the rumors http://www.businessinsider.com/why-apple-bought-foundationdb-2015-3?IR=T http://www.businessinsider.com/why-apple-bought-foundationdb... . But as we know Apple they probably won't tell us.
    • cooervo 9y ago
      or CockroachDB
• rammy1234 9y ago
  May you do good and not evil. May you find forgiveness for yourself and forgive others. May you share freely, never taking more than you give. from their source code blessing. notes.
  • puddums 9y ago
    That's from sqlite. (Awesome tech, awesome license). Related snippet from the "Distinctive Features Of SQLite" page[1] from the sqlite project: The source code files for other SQL database engines typically begin with a comment describing your legal rights to view and copy that file. The SQLite source code contains no license since it is not governed by copyright. Instead of a license, the SQLite source code offers a blessing: May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. [1] https://sqlite.org/different.html https://sqlite.org/different.html
• zackmorris 9y ago
  Looks promising, but does anyone know if FoundationDB has external events or triggers, similar to Firebase or RethinkDB? I can't seem to find much on it. If not, then a lot of potential is being left on the table, because usage would require wrapping FoundationDB in a proxy or middleware of some kind to synthesize events, which can be extremely difficult to get right (due to race conditions, atomicity issues, etc). Without events, apps can find themselves polling or rolling their own pub/sub metaphor over and over again. If anyone with sway is reading this, events are very high on the priority list for me thanx!
  • spullara 9y ago
    It has the ability to watch keys so building a notification system on top of it is pretty easy. Really only limited by your imagination.
    • voidmain 9y ago
      In addition to single-key asynchronous watches, there are also versionstamped ops (for maintaining your own, sophisticated log in a layer) and configurable key range transaction logging (but see the caveats in my other post on the topic). I'm not sure it has every feature it will ever need in this area, but it's a pretty good starting point for building "reactive" stuff.
      • elcritch 9y ago
        Your response is both very exciting and slightly intimidating! Would love to see a “NewSQL” and/or event store built on top of FDB tech. Would the key ranges and versioned ops be capable of providing/emulating a performant “atomic broadcast” similar to that in Kafka?
      • polskibus 9y ago
        Does the trigger execute in scope of triggering transaction with the same isolation level?
        • 9y ago
• rurounijones 9y ago
  Seems like a pretty comprehensive release in terms of OS and language support. One oddity I see is that the ruby gem is not available on rubygems.org and therefore cannot be easily installed and maintained using the ruby package manager which is a bit of a pain.
• josephg 9y ago
  This is wonderful news. I built a proof-of-concept realtime collaborative editor on top of foundationdb a few years ago, and was very disappointed when I couldn't use it in production. I'm really excited to use this in some projects I'm working on. Quick question: I know there's a watch API, but is there any way to subscribe to a change feed from foundationdb? I'd like to consume the FDB event log to do external indexing & map-reduce work.
  • voidmain 9y ago
    Yes. I don't know how well documented it is, but there is an API (well, system keyspace) that can configure the database to log transactions for a selected key range (up to and including the whole database) into another selected key range. It is used by backup and asynchronous replication tools. The format of the configuration keys and transaction logs should be considered less stable than the core key/value API, which basically never breaks backward compatibility, so by using it you are shouldering a maintenance burden to keep up with e.g. changes in the log format and new types of mutations when new versions of FoundationDB come along. So it shouldn't be used too casually. But it's there. Alternatively, your application or layer can use the "versionstamp" atomic operations to write its own ordered log of what it is doing, or other indexing tricks. Depending on your data model this might be able to be much more efficient. For example, for external indexing you probably don't need to preserve a history of prior values but only be able to identify all the values that have changed. This can be done with a very simple and compact index that doesn't need to duplicate all the data to be indexed.
    • josephg 9y ago
      > Alternatively, your application or layer can use the "versionstamp" atomic operations to write its own ordered log of what it is doing, or other indexing tricks. I'm not sure I understand. Are you suggesting having a second key space at `ops/{VERSIONSTAMP}` or something where values contain enough information about the operation to be able to process changes in an indexer? The indexer could then clean up after itself, deleting the operations once they had been ingested? ... Effectively using a portion of the keyspace as a queue?
• brisance 9y ago
  The macOS compilation instructions are interesting; it needs Boost, Mono and a JDK.
• thx4thefish 9y ago
  This seems nice. Besides a bunch of fanboy comments coming from the creators and devs, why is this exciting to the rest of us where things like the capability to join tables in an rdbms is trivial.
  • thx4thefish 9y ago
    Thanks for the downvote. My question still remains.
    • deepanchor 9y ago
      Didn't downvote you, but I believe the excitement is over FoundationDB's ability to perform ACID compliant distributed transactions without sacrificing performance -which to my knowledge no current RDBMS or even NoSql can do.
      • pacala 9y ago
        > no current RDBMS can do https://cloud.google.com/spanner/ https://cloud.google.com/spanner/ ?
        • deepanchor 9y ago
          Meant to say "no current open-source RDBMS"
      • geeostation 9y ago
        Cosmos DB on Azure has been doing it for a while now
  • rapsey 9y ago
    FDB is lower layer. It makes it possible to build a distributed rdbms on top of it. There are is some competition in this area, nothing remotely as mature as FDB.
• nickpsecurity 9y ago
  For anyone interested in learning more, you can still read their blog articles via Wayback here: https://web.archive.org/web/20150304035646/http://blog.foundationdb.com https://web.archive.org/web/20150304035646/http://blog.found...
• bobjordan 9y ago
  Nice documentation around the python API. At this point, docs state it supports 2.7-3.4. What scale of workload is required to catch it up to 3.6 compatability? https://apple.github.io/foundationdb/api-python.html https://apple.github.io/foundationdb/api-python.html
• jinqueeny 9y ago
  Congratulations! It’s very exciting to see FoudationDB back to the open source community!
• manigandham 9y ago
  I wish Apple just became a customer instead of buying and burying this for 3 years. Glad to see it come back but community momentum is hard to rebuild.
• AlphaOne1 9y ago
  I am not a database expert by any means but have been curious about distributed data systems and had not heard of FoundationDB till now and was very excited to read about it. On reading through the documentation, I encountered a section on "Known Limitations"[1] which stated that keys could not be larger than 10kb and values cannot be larger than 100kb. This seems to be a major limitation. Am I missing something or is this strictly for storing text? [1] https://apple.github.io/foundationdb/known-limitations.html#large-keys-and-values https://apple.github.io/foundationdb/known-limitations.html#...
  • voidmain 9y ago
    Because the data model is ordered, large blobs can and normally should be mapped to a bunch of adjacent keys and read with a range read, not a single huge value. That also allows you to read or write just part of one efficiently.
• rakibtg 9y ago
  So you need a mac book pro to use foundation db? :D
• edpichler 9y ago
  Wow, a lot of people excited here, with several use cases of success. I am impressed. I didn't know that FoundationDB was so good till now.
• shahbaz16 9y ago
  honest questions - why are there so many databases?
• polskibus 9y ago
  Is foundationdb capable of providing a linearizable data store? As I remember from Martin Kleppman's books Serializable Snapshot Isolation is not linearizable because the snapshot does not include writes more recent than itself.
  • voidmain 9y ago
    Yes. Linearizable means both serializable and externally consistent (or is sometimes used as just a synonym for the latter), and FDB has these properties with respect to transactions.
    • polskibus 9y ago
      That would mean that SSI is not used to provide Serializable isolation. level. If so, what is used instead? 2 phase locking? I thought it's not very scalable ?
      • voidmain 9y ago
        I explain the basics of our concurrency control here: https://news.ycombinator.com/item?id=16877950 https://news.ycombinator.com/item?id=16877950 I guess textbook SSI is willing to "reorder" conflicting transactions if the result is still serializable, which could violate external consistency if you don't have any other bounds on the order. In the language of SSI, fdb simply aborts the later of any pair of read/write transactions with an rw-conflict, in accordance with a fixed ordering which is externally consistent. I guess it could also be that your book uses an idiosyncratic definition of linearizable, like trying to apply it to individual operations within transactions, which might rule out any optimistic concurrency method. It might just be better to delete this word from your vocabulary in the database field because there is no wide agreement on what it means. The first two hits on Google for me are Wikipedia and Peter Bailis, and they give clearly conflicting definitions, though I think fdb satisfies both!
        • polskibus 9y ago
          Thanks, I'd love to have a bit more of your attention, foundationdb seems very interesting but I need to know a bit more :) Let me expand the definition in Kleppmann's book then.I think it is important because it creates a difference between SSI and typical Serializable level based on 2PL. The below is paraphrasing the definitions on p. 324-329. The book references http://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf http://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf. (I must admit, I read the book, not the paper). Basic idea - make a system appear as if there were only one copy of the data and ALL operations on it are atomic. In this model, there may be replicas, but we don't care about them. As soon as a client completes a write to the db, all clients reading the db must be able to see the value just written. In SSI this is not true, because you may the snapshot may not include writes more recent than the snapshot -> reads from the snapshot are not lineraizable. Linearizable CAS register is equivalent to consensus, and can provide total order. It is therefore what most developers would love to have (if cost was not an issue :) )
• gigatexal 9y ago
  this might not be the right place to ask, but if running foundationdb in a container, how does one connect to it via python?
• kindkid 9y ago
  Very cool. I noticed that all the write benchmarks in https://apple.github.io/foundationdb/benchmarking.html https://apple.github.io/foundationdb/benchmarking.html are for random writes. Is write throughput affected by highly-sequential writes (e.g. - time series) vs random writes? How do you avoid hot-spotting on recent ranges? How efficient are range deletes? On https://apple.github.io/foundationdb/performance.html https://apple.github.io/foundationdb/performance.html I read "The memory engine is optimized for datasets that entirely fit in memory, with secondary storage used for durable writes but not reads." I'd like some clarification: (1) Which memory does "entirely fit in memory" refer to? A single machine? Or SingleNodeMemory * Nodes / ReplicationFactor? (2) If only recently-written data is likely to be queried, and all recently-written data fits entirely in memory, is that sufficient? If so, would an unexpected query of old data cause a huge impact on write throughput? (3) What is the structure/format of the data stored on disk? How is it updated? I'm wondering how well this could be used for time series data. I saw mention here that wavefront uses FoundationDB for this, but would like more details if any are available.
  • voidmain 9y ago
    Sequential writes should be a little faster than random at the individual storage node level, but if your entire write workload is a single ordered log scalability will suffer. It might be theoretically possible for fdb to scale in this situation by creating shards on the fly during transaction processing, but no one has seriously tried to make that work. You can mitigate by designing your key structure/data ordering to not have that property. The memory engine requires your data to fit in memory (total across all your nodes, after replication). It writes interleaved snapshots and updates to disk, and reads the whole dataset back into memory when restarted. You can do great modeling of time series data in FDB, though it will take some care and thought. You should ask these questions on the forum. This article is falling off HN, I am going to lose track of it, and it doesn't look like the Apple team is answering questions here.
• wejick 9y ago
  Never heard about fdb but the concept is very intriguing. We know many nosql db on top kv store rocksdb or level db (like dgraph before moving to new storage engine). That it still needs more to write until it can be called distributed and scalable. By using foundation db we can skip many parts and focus on other part like query language and API. That's why I like the layer concept, unfortunately the is very little documentation about it. Found some layer written in python in the repo, but I don't understand where is the position of layer in the general architecture. I thought it would like plugin but I think that's not the case.
  • voidmain 9y ago
    A "layer" uses the fdb client much as it might use rocksdb. The layer can be a library embedded in your application, or a network service, it's up to you.
• ile 9y ago
  It is strange, why ArangoDB isn't mentioned often on HN, or in this thread. It is multi-model (KV, document, graph) database (with transactions) and I have been happily using it for a while now. Haven't scaled it yet to any large installations, so I can speak about how well it does that.
• hintzemichael89 9y ago
  Very cool
• dtheodor 9y ago
  Since becoming open source and the system's architecture is in the open, I wonder how this criticism of FoundationDB from a VoltDB architect https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-fast-key-value-store-not-enough/ https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-... fairs against the knowledge that has now become available. (To summarize, the author argues that building an SQL layer on top of an ordered key-value store is suboptimal)
• truth_seeker 9y ago
  I think Apache Ignite distributed KV store is much better choice as it has 2PC, indexes, distributed computing idioms and can be embedded as library in JVM app. Plus it supports SQL engine, thanks to H2 SQL parser engine. You can also create graph layer over it using gremlin in a day or two.
• EGreg 9y ago
  How does FoundationDB compare against CockroachDB?