6 ms·

FoundationDB community highlights, two weeks in

137 points by wwilson 8y ago

coldcode 8y ago
Hoping someone makes a Swift interface, given this is coming from Apple, one would hope there would be one.
- superlopuh 8y ago
  It's a race between a Swift native client and Swift-Python interop :)
Mister_Snuggles 8y ago
It's even been ported and packaged for FreeBSD: https://www.freshports.org/databases/foundationdb/ https://www.freshports.org/databases/foundationdb/
- aseipp 8y ago
  I've also created a package, as well as module support, for FoundationDB on NixOS: https://hydra.nixos.org/job/nixpkgs/trunk/foundationdb51.x86_64-linux Hopefully it will be in a good place by the time NixOS 18.09 is out.
carlhjerpe 8y ago
I'd like to see the ANSI SQL layer open-sourced, and some benchmarks.
- ansible 8y ago
  If you need SQL now, you might consider CockroachDB: https://www.cockroachlabs.com/ https://www.cockroachlabs.com/ It is on-the-wire compatible with PostgreSQL clients.
  - zdw 8y ago
    Except where it isn't: https://github.com/cockroachdb/cockroachdb-python/pull/14#issuecomment-304429967 https://github.com/cockroachdb/cockroachdb-python/pull/14#is...
    - ansible 8y ago
      I apologize if I was being unclear. It is wire-compatible with PostgreSQL clients, meaning that if you have a PostgreSQL client library for your favorite programming language, you can use that to connect to CockroachDB. The database features and SQL dialect itself are not 100% compatible with PostgreSQL, and likely never will be. Also, even if all the SQL you do want to use is supported, it is likely a sub-optimal idea to just port some SQL over to the CockroachDB platform without additional investigation. Some issues, like interleaving values from different tables, should be examined to ensure good performance on CockroachDB. And even setting aside issues like that, the access patterns you'd use for a multi-region distributed database are going to be a bit different than a high-availability PostgreSQL cluster. Data locality and all that. The wide availability of client drivers just means you can get started quickly on popular platforms.
- zzzcpan 8y ago
  What would be the point of benchmarks? Benchmarks say pretty much nothing about distributed databases. But you do want to know how such database behaves in general and during failures with some load and some data, say 100M records and 3 nodes. What kind of latency it has when nodes are added, replaced, moved etc. Including 90th percentile latency. How long does it take to move 1 TB worth of records, what if this process was interrupted, can it resume later or not. How does it deal with HDDs and bad sectors, especially wrt to performance, since disks like to retry bad sectors for a few seconds, completely blocking all I/O, requiring special handling. Or is it only suitable for SSDs.
- jfindley 8y ago
  Last time I looked, the SQL layer was really slow. I know they invested a ton of effort to improving it, but I don't think they ever really cracked the problem and I think you're likely better off with CockroachDB, as mentioned in another reply.
  - fizx 8y ago
    I've built a toy SQL database before that actually did a good job with query planning and used existing highly optimized storage engines (lucene or rocksdb). It was slow AF, because the actual query execution has a ton of method dispatches, etc to e.g. check index conditions or assemble rows and it never quite gets fully JIT'd by the JVM. Every reasonable SQL engine i'm aware of compiles queries to lower-level code. A cursory inspection of their code indicates they don't do this, so its not surprising its slow.
  - carlhjerpe 8y ago
    https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer Here's the old layer :)
ihnorton 8y ago
I’m curious how they plan to do continuous integration. The Apple hardware-only clause of the macOS SDK license agreement really crimps the options for CI, especially for open source projects with limited budget. Travis is a godsend, but there are startup latency and capacity issues there even in the paid tier. Apple has made major contributions to open source, as evidenced by FoundationDB here, LLVM, WebKit, and other projects. But they no longer make servers, and the fallback — Mac mini — is on life support, which puts projects and service providers who need to treat macOS as a first-class target in a real bind. Even something limited like the IE Edge and Visual Studio images Microsoft provides would be a huge boost.
- fnord123 8y ago
  >I’m curious how they plan to do continuous integration. The Apple hardware-only clause of the macOS SDK license agreement really crimps the options for CI, especially for open source projects with limited budget. FoundationDB is a distributed key value store. These almost always run on servers. macOS is not a server OS. Is there any reason to build this on macOS, over, say, FreeBSD.
  - jen20 8y ago
    Many developers want to install it for local development - presumably this is a particular popular sentiment _at Apple_. There is a FreeBSD port already, however [1]. [1]: https://www.freshports.org/databases/foundationdb/ https://www.freshports.org/databases/foundationdb/
    - Mister_Snuggles 8y ago
      Sadly, as I discovered today, the FreeBSD port is incomplete. First up, it doesn't create the fdb.cluster file. I reverse engineered that logic out of the Mac OS packaging script, which wasn't a big deal. Once I created the file I was able to start it up and connect to it with fdbcli. Second, the Python bindings do not work on FreeBSD. Trying to set the API version (the first step in calling the API) gives an error specifically saying they're not supported on FreeBSD. Still though, for something that was released two weeks ago to be already ported to FreeBSD is pretty impressive!
dman 8y ago
I am very curious on what Apple uses as the server OS in its datacenters. Do they use OSX or do they use Linux / FreeBSD?
- FBISurveillance 8y ago
  A company of Apple size likely uses multiple OS', I wouldn't be surprised if they even run Windows servers somewhere. Judging from their "DevOps" position requirements, their stack is similar to many others out there Chef, Ansible, Docker, Kubernetes, Java, Python (with accent on Python so maybe Salt Stack), Ruby, Go, Git, C++. Given their extensive Hadoop infrastructure, likely CentOS or Debian flavor. Some job positions list "RHEL or derivative a plus".
  - e12e 8y ago
    > Ansible,... Python (with accent on Python so maybe Salt Stack) I'm not sure it makes a lot of sense to use ansible and salt stack (both automation tolls written in python). They strike me more as alternatives than being complimentary.
    - krylon 8y ago
      Apple is a big company. Different teams/divisions within the company might plausibly use both.
  - solarkraft 8y ago
    > I wouldn't be surprised if they even run Windows servers somewhere. I would be a little, because they seem to proud to do it. At least their public facing infrastructure seems very well selected and uniform.
fnord123 8y ago
To add to the list in TFA, Rust bindings are here: https://github.com/bluejekyll/foundationdb-rs https://github.com/bluejekyll/foundationdb-rs
- bluejekyll 8y ago
  You beat me to it! I think we're getting close to a 0.2 release. We have enough working for the class scheduling tutorial, so seems like a good next point release soon.
marknadal 8y ago
Could somebody clarify the license? I was excited this was Open Source, but then some people say you are only allowed to run it on MacOS? How can that be? Also, how does FoundationDB compare (in general, and in performance) to: - CockroachDB (competitor) https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ - Redis (competitor) https://redis.io/topics/benchmarks https://redis.io/topics/benchmarks - GUN (ours) https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... - Fauna (competitor) https://fauna.com/enterprise#Scalability https://fauna.com/enterprise#Scalability - VoltDB (competitor) https://www.voltdb.com/blog/2017/10/02/comparing-fast-data-performance-a-comparison-of-voltdb-and-cassandra-benchmarks/ https://www.voltdb.com/blog/2017/10/02/comparing-fast-data-p... - RocksDB (competitor) https://github.com/facebook/rocksdb/wiki/performance-benchmarks https://github.com/facebook/rocksdb/wiki/performance-benchma... Is the the big boom in FoundationDB just that it is "now Open Source", or that Apple vetted it (why didn't they keep it closed source?), or is there something distinctly unique about FoundationDB that it surpasses a bunch of other tried and true systems?
- owaislone 8y ago
  I think one major distinction is that Foundation DB can be used to build most of the databases you mentioned above because of its architecture.
  - marknadal 8y ago
    I'm pretty sure that is true the other way around though: Back in 2011 I built a couple DBs with Redis, and I've definitely seen people swapping other databases (MySQL, MongoDB, etc.) storage engine for RocksDB and others. I also know the storage engine we wrote (Radix Storage Engine) could be used to build LevelDB and Cassandra on top. Even Cockroach makes for a pretty solid storage engine. Could you give an example of why you couldn't use VoltDB to get FoundationDB like properties?
    - owaislone 8y ago
      It is possible to write code to add a compatibility layer on top of any DB to make it work like any other DB. Foundation DB's architecture specifically accounts for this. It was designed from the ground up to allow building different interfaces/layers. We can write compilers for any language to compile it to Python bytecode. That does not make python VM and .NET CLR the same thing. CLR clearly has a design goal of allowing multiple languages to run.
ex3ndr 8y ago
Probably a silly question, but does someone running it in production already? I mean not Apple, but someone from community.
- manigandham 8y ago
  As the article states, Snowflake Data [1] and Wavefront [2] have been using it. SkuVault [3] is another company. https://www.snowflake.net/how-foundationdb-powers-snowflake-metadata-forward/ https://www.snowflake.net/how-foundationdb-powers-snowflake-... https://www.wavefront.com/wavefront-foundationdb-open-source-project/ https://www.wavefront.com/wavefront-foundationdb-open-source... https://abdullin.com/sku-vault/foundationdb-layers/ https://abdullin.com/sku-vault/foundationdb-layers/ There are probably several other companies from the original releases that haven't posted anything publicly, but it'll be awhile before we get full deployments from the latest open-source versions.
praveenhm 8y ago
Is there any apache Spark foundationDB integration, like drivers?
- gregwebs 8y ago
  You might be interested in TiDB's TiSpark which is all built on top of TiKV (equivalent to FoundationDB). https://github.com/pingcap/tispark https://github.com/pingcap/tispark
whateveracct 8y ago
Another highlight: FoundationDB is in nixpkgs unstable!

137 points by wwilson 8y ago

coldcode 8y ago

Hoping someone makes a Swift interface, given this is coming from Apple, one would hope there would be one.

superlopuh 8y ago
It's a race between a Swift native client and Swift-Python interop :)

Mister_Snuggles 8y ago

It's even been ported and packaged for FreeBSD: https://www.freshports.org/databases/foundationdb/ https://www.freshports.org/databases/foundationdb/

aseipp 8y ago
I've also created a package, as well as module support, for FoundationDB on NixOS: https://hydra.nixos.org/job/nixpkgs/trunk/foundationdb51.x86_64-linux Hopefully it will be in a good place by the time NixOS 18.09 is out.

carlhjerpe 8y ago

I'd like to see the ANSI SQL layer open-sourced, and some benchmarks.

ansible 8y ago
If you need SQL now, you might consider CockroachDB: https://www.cockroachlabs.com/ https://www.cockroachlabs.com/ It is on-the-wire compatible with PostgreSQL clients.
- zdw 8y ago
  Except where it isn't: https://github.com/cockroachdb/cockroachdb-python/pull/14#issuecomment-304429967 https://github.com/cockroachdb/cockroachdb-python/pull/14#is...
  - ansible 8y ago
    I apologize if I was being unclear. It is wire-compatible with PostgreSQL clients, meaning that if you have a PostgreSQL client library for your favorite programming language, you can use that to connect to CockroachDB. The database features and SQL dialect itself are not 100% compatible with PostgreSQL, and likely never will be. Also, even if all the SQL you do want to use is supported, it is likely a sub-optimal idea to just port some SQL over to the CockroachDB platform without additional investigation. Some issues, like interleaving values from different tables, should be examined to ensure good performance on CockroachDB. And even setting aside issues like that, the access patterns you'd use for a multi-region distributed database are going to be a bit different than a high-availability PostgreSQL cluster. Data locality and all that. The wide availability of client drivers just means you can get started quickly on popular platforms.
zzzcpan 8y ago
What would be the point of benchmarks? Benchmarks say pretty much nothing about distributed databases. But you do want to know how such database behaves in general and during failures with some load and some data, say 100M records and 3 nodes. What kind of latency it has when nodes are added, replaced, moved etc. Including 90th percentile latency. How long does it take to move 1 TB worth of records, what if this process was interrupted, can it resume later or not. How does it deal with HDDs and bad sectors, especially wrt to performance, since disks like to retry bad sectors for a few seconds, completely blocking all I/O, requiring special handling. Or is it only suitable for SSDs.
jfindley 8y ago
Last time I looked, the SQL layer was really slow. I know they invested a ton of effort to improving it, but I don't think they ever really cracked the problem and I think you're likely better off with CockroachDB, as mentioned in another reply.
- fizx 8y ago
  I've built a toy SQL database before that actually did a good job with query planning and used existing highly optimized storage engines (lucene or rocksdb). It was slow AF, because the actual query execution has a ton of method dispatches, etc to e.g. check index conditions or assemble rows and it never quite gets fully JIT'd by the JVM. Every reasonable SQL engine i'm aware of compiles queries to lower-level code. A cursory inspection of their code indicates they don't do this, so its not surprising its slow.
- carlhjerpe 8y ago
  https://github.com/jaytaylor/sql-layer https://github.com/jaytaylor/sql-layer Here's the old layer :)

ihnorton 8y ago

I’m curious how they plan to do continuous integration. The Apple hardware-only clause of the macOS SDK license agreement really crimps the options for CI, especially for open source projects with limited budget. Travis is a godsend, but there are startup latency and capacity issues there even in the paid tier. Apple has made major contributions to open source, as evidenced by FoundationDB here, LLVM, WebKit, and other projects. But they no longer make servers, and the fallback — Mac mini — is on life support, which puts projects and service providers who need to treat macOS as a first-class target in a real bind. Even something limited like the IE Edge and Visual Studio images Microsoft provides would be a huge boost.

fnord123 8y ago
>I’m curious how they plan to do continuous integration. The Apple hardware-only clause of the macOS SDK license agreement really crimps the options for CI, especially for open source projects with limited budget. FoundationDB is a distributed key value store. These almost always run on servers. macOS is not a server OS. Is there any reason to build this on macOS, over, say, FreeBSD.
- jen20 8y ago
  Many developers want to install it for local development - presumably this is a particular popular sentiment _at Apple_. There is a FreeBSD port already, however [1]. [1]: https://www.freshports.org/databases/foundationdb/ https://www.freshports.org/databases/foundationdb/
  - Mister_Snuggles 8y ago
    Sadly, as I discovered today, the FreeBSD port is incomplete. First up, it doesn't create the fdb.cluster file. I reverse engineered that logic out of the Mac OS packaging script, which wasn't a big deal. Once I created the file I was able to start it up and connect to it with fdbcli. Second, the Python bindings do not work on FreeBSD. Trying to set the API version (the first step in calling the API) gives an error specifically saying they're not supported on FreeBSD. Still though, for something that was released two weeks ago to be already ported to FreeBSD is pretty impressive!

dman 8y ago

I am very curious on what Apple uses as the server OS in its datacenters. Do they use OSX or do they use Linux / FreeBSD?

FBISurveillance 8y ago
A company of Apple size likely uses multiple OS', I wouldn't be surprised if they even run Windows servers somewhere. Judging from their "DevOps" position requirements, their stack is similar to many others out there Chef, Ansible, Docker, Kubernetes, Java, Python (with accent on Python so maybe Salt Stack), Ruby, Go, Git, C++. Given their extensive Hadoop infrastructure, likely CentOS or Debian flavor. Some job positions list "RHEL or derivative a plus".
- e12e 8y ago
  > Ansible,... Python (with accent on Python so maybe Salt Stack) I'm not sure it makes a lot of sense to use ansible and salt stack (both automation tolls written in python). They strike me more as alternatives than being complimentary.
  - krylon 8y ago
    Apple is a big company. Different teams/divisions within the company might plausibly use both.
- solarkraft 8y ago
  > I wouldn't be surprised if they even run Windows servers somewhere. I would be a little, because they seem to proud to do it. At least their public facing infrastructure seems very well selected and uniform.

fnord123 8y ago

To add to the list in TFA, Rust bindings are here: https://github.com/bluejekyll/foundationdb-rs https://github.com/bluejekyll/foundationdb-rs

bluejekyll 8y ago
You beat me to it! I think we're getting close to a 0.2 release. We have enough working for the class scheduling tutorial, so seems like a good next point release soon.

marknadal 8y ago

Could somebody clarify the license? I was excited this was Open Source, but then some people say you are only allowed to run it on MacOS? How can that be? Also, how does FoundationDB compare (in general, and in performance) to: - CockroachDB (competitor) https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/ - Redis (competitor) https://redis.io/topics/benchmarks https://redis.io/topics/benchmarks - GUN (ours) https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-Atom-CPU https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... - Fauna (competitor) https://fauna.com/enterprise#Scalability https://fauna.com/enterprise#Scalability - VoltDB (competitor) https://www.voltdb.com/blog/2017/10/02/comparing-fast-data-performance-a-comparison-of-voltdb-and-cassandra-benchmarks/ https://www.voltdb.com/blog/2017/10/02/comparing-fast-data-p... - RocksDB (competitor) https://github.com/facebook/rocksdb/wiki/performance-benchmarks https://github.com/facebook/rocksdb/wiki/performance-benchma... Is the the big boom in FoundationDB just that it is "now Open Source", or that Apple vetted it (why didn't they keep it closed source?), or is there something distinctly unique about FoundationDB that it surpasses a bunch of other tried and true systems?

owaislone 8y ago
I think one major distinction is that Foundation DB can be used to build most of the databases you mentioned above because of its architecture.
- marknadal 8y ago
  I'm pretty sure that is true the other way around though: Back in 2011 I built a couple DBs with Redis, and I've definitely seen people swapping other databases (MySQL, MongoDB, etc.) storage engine for RocksDB and others. I also know the storage engine we wrote (Radix Storage Engine) could be used to build LevelDB and Cassandra on top. Even Cockroach makes for a pretty solid storage engine. Could you give an example of why you couldn't use VoltDB to get FoundationDB like properties?
  - owaislone 8y ago
    It is possible to write code to add a compatibility layer on top of any DB to make it work like any other DB. Foundation DB's architecture specifically accounts for this. It was designed from the ground up to allow building different interfaces/layers. We can write compilers for any language to compile it to Python bytecode. That does not make python VM and .NET CLR the same thing. CLR clearly has a design goal of allowing multiple languages to run.

ex3ndr 8y ago

Probably a silly question, but does someone running it in production already? I mean not Apple, but someone from community.

manigandham 8y ago
As the article states, Snowflake Data [1] and Wavefront [2] have been using it. SkuVault [3] is another company. https://www.snowflake.net/how-foundationdb-powers-snowflake-metadata-forward/ https://www.snowflake.net/how-foundationdb-powers-snowflake-... https://www.wavefront.com/wavefront-foundationdb-open-source-project/ https://www.wavefront.com/wavefront-foundationdb-open-source... https://abdullin.com/sku-vault/foundationdb-layers/ https://abdullin.com/sku-vault/foundationdb-layers/ There are probably several other companies from the original releases that haven't posted anything publicly, but it'll be awhile before we get full deployments from the latest open-source versions.

praveenhm 8y ago

Is there any apache Spark foundationDB integration, like drivers?

gregwebs 8y ago
You might be interested in TiDB's TiSpark which is all built on top of TiKV (equivalent to FoundationDB). https://github.com/pingcap/tispark https://github.com/pingcap/tispark

whateveracct 8y ago

Another highlight: FoundationDB is in nixpkgs unstable!