5 ms·

FoundationDB Document Layer

297 points by wwilson 8y ago

monstrado 8y ago
This is massive for adoption. Can't wait to give it a try.
spullara 8y ago
Really excited that they were able to release this layer. With the changes to the license of MongoDB it could give people a real alternative.
justaaron 8y ago
That's huge. I could have used this a few years ago
andrewflnr 8y ago
Is there any reason to expect fundamental inefficiencies with this approach vs a vertically integrated approach like a conventional document store? Any other downsides? I ask because this seems like to obviously good an idea to be true, so I need to look for the flip side before I get to excited. Otherwise, this laying seems like the obvious approach, and I can only lament that it's taken this long.
- ryanworl 8y ago
  Yes, the layered approach will always consume more bandwidth unless you just happen to hit the case where your transaction is read-only and the layer process is colocated with the storage server where your data is. The upsides far outweigh this downside in my opinion. The fact that writes are buffered locally before committing the transaction (improving latency) is probably enough by itself to overcome that downside.
  - bithavoc 8y ago
    Interesting, TiDB also three components [0](KV, DB and PD), I guess this MongoDB layer in FoundationDB is the DB piece of the cluster. [0] https://github.com/pingcap/docs#tidb-server https://github.com/pingcap/docs#tidb-server
- tracker1 8y ago
  Haven't dug in, but depends on if there's a system in place to support multiple indexes on your documents and how the system uses them. This could be a relatively small or huge impact depending on your interface requirements, sorting etc.
aseipp 8y ago
Wonderful. I just updated the FoundationDB packages in NixOS to 6.0.15 at the release earlier this month. Will try packaging this up soon as well...
monstrado 8y ago
Are there any plans to extend the API compatibility of the document layer to be compatible with other APIs, such as, DynamoDB?
yoava 8y ago
Just wandering how will foindationDb compare to mongo in terms of the non functional s - performance, scale of a collection, number of collections, queries with and without indexes, aggregated, etc.
rubyn00bie 8y ago
Soooo stoked. This should make futzing with it to get started a lot easier. I've been thinking about going to the FoundationDB summit to hopefully learn more... Does anyone have any good resources for learning about it? I've tried a few times but find the documentation a bit hard to grok. Maybe I'm just slow. Also if you haven't seen it, this is one of the most amazing talks on distributed systems (because it's about testing them) I've ever seen (by Will Wilson from FoundationDB pre-Apple merger): https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Truly mind blowing how good FoundationDB is supposed to be for its use cases, and this Mongo layer is just the business when it comes to helping adoption.
- andrewflnr 8y ago
  That talk was glorious. "We want to build a database, but first we need to build a simulator for the database, but first we need to write a compiler to help us write the simulator." It would be self-parodying engineering hubris if it hadn't worked.
tinyvm 8y ago
> Document Layer has a familiar API, and is compatible with the MongoDB® protocol Was wondering how I would replace MongoDB due to the shift in Licensing become more and more agressive. Well , I think I found it. Farewell MongoDB.
- justaaron 8y ago
  more like a scaling issue, as mongo is master-slave and a distributed horizontally scaled world can't live with such a bottleneck.
  - tracker1 8y ago
    Depends on your deployment specifics and your requirements. I would say that a replica set of mongodb may be faster, but a sharded replica set in mongo may be slower than a local cluster of foundation with the document layer interface.
  - merlynn 8y ago
    Well that’s just a ridiculously uninformed statement. Of course mongodb scales horizontally. I can share many success stories of massive scale with mongodb.
- pxtail 8y ago
  It's like hard slap straight in the face. They changed license because they perceived that multibilion giant(s) feasts and earn money upon their work and gives back not that much. In return giant just slaps
  - brazzledazzle 8y ago
    I’ll never understand why we collectively wag our fingers at individuals or companies that try to keep the likes of Amazon from building a profitable service off of their hard work then contribute back little-to-nothing. Would redis, mongodb and dgraph have even considered alternate licensing if companies like Amazon had thrown them a minuscule amount of funding and patches? We can’t know because they didn’t. And then we sneer at them for having the audacity to try to stay open but stop these giants from using them and throwing them away.
rhacker 8y ago
I don't know much about foundationDB, but when they say API compatible, are they referring to the technical TCP/IP connection stack, or ALSO the concepts of collections and the query language of MongoDB (such as aggregations, find, limit, etc..) In other words can I throw a giant aggregation query with all mongo's various crazy query language syntax and it will.. work the same?
- sbr464 8y ago
  It doesn't support the aggregation framework currently. Here is a list of differences: https://foundationdb.github.io/fdb-document-layer/known-differences.html https://foundationdb.github.io/fdb-document-layer/known-diff...
  - spullara 8y ago
    Can someone who uses MongoDB at scale (many machines, single db) talk about how big gap is between what FDB supports and what their application uses?
- rubyn00bie 8y ago
  Here's what seems to me comprehensive list: https://foundationdb.github.io/fdb-document-layer/known-differences.html https://foundationdb.github.io/fdb-document-layer/known-diff... Looks like there are definitely some shortcomings, but also some wins... From the link: > Sessions > MongoDB® has introduced sessions in v3.6. The Document Layer doesn’t support sessions yet. Sessions in MongoDB® enable better consistency guarentees. It’s important to note that, even though the Document Layer doesn’t support sessions, it has better consistency guarentees by default due to FoundationDB backend.
justaaron 8y ago
i think we are going to generally see more and more object to key-value mapping layers as the latter are widely available via cloud providers, and one basically needs something in the spirit of persistent-volume-claims from kubernetes land in the database layer that can be fulfilled by some key-value backend...
sfilargi 8y ago
FoundationDB always sounded amazing, BUT, the way they disappeared when it was bought by Apple left a bitter taste in my mouth and I don't think I would ever trust them enough to build my business around their product.
- dilap 8y ago
  It's all open-source and out there now though, so I don't think you really need to trust them, right?
thefounder 8y ago
Can someone write a Google Datastore layer so that I can finish my appengine/datastore adventure once and for all?
- rckclmbr 8y ago
  Be the change you want to see
  - thefounder 8y ago
    I think it's not worth it. I rather not use Datastore for anything from now on. You have to cut your losses at some point. We are using microservices so it's not hard to make the change(though we loose some tools we designed for datastore).
- wsh91 8y ago
  Is there anything we could be doing to serve you better?
  - thefounder 8y ago
    Yes, let me run datastore on my servers. Surely I may use your cloud offer but in certain cases I need to run it on my servers. I can't build/invest in tools/drivers for a proprietary cloud service.
- merb 8y ago
  well there is https://github.com/AppScale/appscale https://github.com/AppScale/appscale which probably has some kind of layer on top of cassandra that will emulate it. but it's a really big project... (But you can run just the Datastore: https://github.com/AppScale/appscale/tree/master/AppDB https://github.com/AppScale/appscale/tree/master/AppDB)
  - thefounder 8y ago
    I already tried it...it looks good on paper but bad in practice. Think of a graveyard where appengine go.
janemanos 8y ago
Sounds kind of "old" tech to have different layers for supporting different data models. Like the native multi-model approach of ArangoDB much better. Their aggregation framework is pretty neat and the database also has full graph capabilities. Heck, I can even combine joins and graph traversal in the same query.
mikece 8y ago
I'm a little confused. Does the "layers" concept mean the data which is stored in the core can be accessed in any valid manner that the layers allow? Or stated differently: if I've got a dumpster fire of data in a MongoDB that should have been in an RDBMS but moving to SQL would be too risky/slow would this allow me to schlep my Mongo data to FoundationDB and have the app continue to use the Mongo access methods while I start switching APIs over to querying by SQL in batches instead of One Big Volatile Move?
- sigstoat 8y ago
  foundationdb is just a KV store that doesn't even authenticate clients connecting to it. the idea is you put layers in front of it to take nice structured requests (of some sort) and turn them into transactional KV manipulations, while also doing things like authentication, access control, logging, whatever else you need. so you could (hypothetically, if such software existed), have this mongo layer, and then another layer off to the side which took SQL queries, and figured out how to turn them into KV queries against the data that the mongo layer has stored. more realistically, you could have your SQL clients know how to talk to foundationdb directly (making them into the layer; presumably they're a web backend or something already), and they could know how to execute the high level operation they want to perform against the KV data stored by the mongo layer. conveniently skipping all of the SQL translation that there isn't software for.

297 points by wwilson 8y ago

This is massive for adoption. Can't wait to give it a try.

Really excited that they were able to release this layer. With the changes to the license of MongoDB it could give people a real alternative.

justaaron 8y ago

That's huge. I could have used this a few years ago

andrewflnr 8y ago

Is there any reason to expect fundamental inefficiencies with this approach vs a vertically integrated approach like a conventional document store? Any other downsides? I ask because this seems like to obviously good an idea to be true, so I need to look for the flip side before I get to excited. Otherwise, this laying seems like the obvious approach, and I can only lament that it's taken this long.

ryanworl 8y ago
Yes, the layered approach will always consume more bandwidth unless you just happen to hit the case where your transaction is read-only and the layer process is colocated with the storage server where your data is. The upsides far outweigh this downside in my opinion. The fact that writes are buffered locally before committing the transaction (improving latency) is probably enough by itself to overcome that downside.
- bithavoc 8y ago
  Interesting, TiDB also three components [0](KV, DB and PD), I guess this MongoDB layer in FoundationDB is the DB piece of the cluster. [0] https://github.com/pingcap/docs#tidb-server https://github.com/pingcap/docs#tidb-server
tracker1 8y ago
Haven't dug in, but depends on if there's a system in place to support multiple indexes on your documents and how the system uses them. This could be a relatively small or huge impact depending on your interface requirements, sorting etc.

aseipp 8y ago

Wonderful. I just updated the FoundationDB packages in NixOS to 6.0.15 at the release earlier this month. Will try packaging this up soon as well...

monstrado 8y ago

Are there any plans to extend the API compatibility of the document layer to be compatible with other APIs, such as, DynamoDB?

yoava 8y ago

Just wandering how will foindationDb compare to mongo in terms of the non functional s - performance, scale of a collection, number of collections, queries with and without indexes, aggregated, etc.

rubyn00bie 8y ago

Soooo stoked. This should make futzing with it to get started a lot easier. I've been thinking about going to the FoundationDB summit to hopefully learn more... Does anyone have any good resources for learning about it? I've tried a few times but find the documentation a bit hard to grok. Maybe I'm just slow. Also if you haven't seen it, this is one of the most amazing talks on distributed systems (because it's about testing them) I've ever seen (by Will Wilson from FoundationDB pre-Apple merger): https://www.youtube.com/watch?v=4fFDFbi3toc https://www.youtube.com/watch?v=4fFDFbi3toc Truly mind blowing how good FoundationDB is supposed to be for its use cases, and this Mongo layer is just the business when it comes to helping adoption.

andrewflnr 8y ago
That talk was glorious. "We want to build a database, but first we need to build a simulator for the database, but first we need to write a compiler to help us write the simulator." It would be self-parodying engineering hubris if it hadn't worked.

tinyvm 8y ago

> Document Layer has a familiar API, and is compatible with the MongoDB® protocol Was wondering how I would replace MongoDB due to the shift in Licensing become more and more agressive. Well , I think I found it. Farewell MongoDB.

justaaron 8y ago
more like a scaling issue, as mongo is master-slave and a distributed horizontally scaled world can't live with such a bottleneck.
- tracker1 8y ago
  Depends on your deployment specifics and your requirements. I would say that a replica set of mongodb may be faster, but a sharded replica set in mongo may be slower than a local cluster of foundation with the document layer interface.
- merlynn 8y ago
  Well that’s just a ridiculously uninformed statement. Of course mongodb scales horizontally. I can share many success stories of massive scale with mongodb.
pxtail 8y ago
It's like hard slap straight in the face. They changed license because they perceived that multibilion giant(s) feasts and earn money upon their work and gives back not that much. In return giant just slaps
- brazzledazzle 8y ago
  I’ll never understand why we collectively wag our fingers at individuals or companies that try to keep the likes of Amazon from building a profitable service off of their hard work then contribute back little-to-nothing. Would redis, mongodb and dgraph have even considered alternate licensing if companies like Amazon had thrown them a minuscule amount of funding and patches? We can’t know because they didn’t. And then we sneer at them for having the audacity to try to stay open but stop these giants from using them and throwing them away.

rhacker 8y ago

I don't know much about foundationDB, but when they say API compatible, are they referring to the technical TCP/IP connection stack, or ALSO the concepts of collections and the query language of MongoDB (such as aggregations, find, limit, etc..) In other words can I throw a giant aggregation query with all mongo's various crazy query language syntax and it will.. work the same?

sbr464 8y ago
It doesn't support the aggregation framework currently. Here is a list of differences: https://foundationdb.github.io/fdb-document-layer/known-differences.html https://foundationdb.github.io/fdb-document-layer/known-diff...
- spullara 8y ago
  Can someone who uses MongoDB at scale (many machines, single db) talk about how big gap is between what FDB supports and what their application uses?
rubyn00bie 8y ago
Here's what seems to me comprehensive list: https://foundationdb.github.io/fdb-document-layer/known-differences.html https://foundationdb.github.io/fdb-document-layer/known-diff... Looks like there are definitely some shortcomings, but also some wins... From the link: > Sessions > MongoDB® has introduced sessions in v3.6. The Document Layer doesn’t support sessions yet. Sessions in MongoDB® enable better consistency guarentees. It’s important to note that, even though the Document Layer doesn’t support sessions, it has better consistency guarentees by default due to FoundationDB backend.

justaaron 8y ago

i think we are going to generally see more and more object to key-value mapping layers as the latter are widely available via cloud providers, and one basically needs something in the spirit of persistent-volume-claims from kubernetes land in the database layer that can be fulfilled by some key-value backend...

sfilargi 8y ago

FoundationDB always sounded amazing, BUT, the way they disappeared when it was bought by Apple left a bitter taste in my mouth and I don't think I would ever trust them enough to build my business around their product.

dilap 8y ago
It's all open-source and out there now though, so I don't think you really need to trust them, right?

thefounder 8y ago

Can someone write a Google Datastore layer so that I can finish my appengine/datastore adventure once and for all?

rckclmbr 8y ago
Be the change you want to see
- thefounder 8y ago
  I think it's not worth it. I rather not use Datastore for anything from now on. You have to cut your losses at some point. We are using microservices so it's not hard to make the change(though we loose some tools we designed for datastore).
wsh91 8y ago
Is there anything we could be doing to serve you better?
- thefounder 8y ago
  Yes, let me run datastore on my servers. Surely I may use your cloud offer but in certain cases I need to run it on my servers. I can't build/invest in tools/drivers for a proprietary cloud service.
merb 8y ago
well there is https://github.com/AppScale/appscale https://github.com/AppScale/appscale which probably has some kind of layer on top of cassandra that will emulate it. but it's a really big project... (But you can run just the Datastore: https://github.com/AppScale/appscale/tree/master/AppDB https://github.com/AppScale/appscale/tree/master/AppDB)
- thefounder 8y ago
  I already tried it...it looks good on paper but bad in practice. Think of a graveyard where appengine go.

janemanos 8y ago

Sounds kind of "old" tech to have different layers for supporting different data models. Like the native multi-model approach of ArangoDB much better. Their aggregation framework is pretty neat and the database also has full graph capabilities. Heck, I can even combine joins and graph traversal in the same query.

mikece 8y ago

I'm a little confused. Does the "layers" concept mean the data which is stored in the core can be accessed in any valid manner that the layers allow? Or stated differently: if I've got a dumpster fire of data in a MongoDB that should have been in an RDBMS but moving to SQL would be too risky/slow would this allow me to schlep my Mongo data to FoundationDB and have the app continue to use the Mongo access methods while I start switching APIs over to querying by SQL in batches instead of One Big Volatile Move?

sigstoat 8y ago
foundationdb is just a KV store that doesn't even authenticate clients connecting to it. the idea is you put layers in front of it to take nice structured requests (of some sort) and turn them into transactional KV manipulations, while also doing things like authentication, access control, logging, whatever else you need. so you could (hypothetically, if such software existed), have this mongo layer, and then another layer off to the side which took SQL queries, and figured out how to turn them into KV queries against the data that the mongo layer has stored. more realistically, you could have your SQL clients know how to talk to foundationdb directly (making them into the layer; presumably they're a web backend or something already), and they could know how to execute the high level operation they want to perform against the KV data stored by the mongo layer. conveniently skipping all of the SQL translation that there isn't software for.