6 ms·

Skipping the boring parts of building a database using FoundationDB

75 points by ovaistariq 4y ago

ovaistariq 4y ago
Building distributed database systems correctly from the ground up is a notoriously hard problem. We have seen this firsthand building Docstore at Uber. This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Vendors are spending so much time rebuilding existing solutions, they end up not solving the actual end users’ problems, although ostensibly that's why they decided to create a new data platform in the first place! In this post we talk about our approach to building Tigris - the open source developer data platform. We talk about why we chose to build on top of FoundationDB, one of the most reliable distributed KV store with an amazing correctness story. We also go into detail about our experience using it.
- metadat 4y ago
  Are you Himank or Yevgeniy? Good luck with your project, FDB fucked it's users back in 2015 when it abruptly closed shop and went closed source. Hopefully some good can come of it yet.
  - ovaistariq 4y ago
    It has significant usage across large companies such as Apple, Snowflake, Epic Games, VMWare, etc. I don't see it going closed source. Besides that, here Tigris is taking the responsibility for the product as it is abstracting FoundationDB from the end-user.
  - endisneigh 4y ago
    I can assure you from FDB will not be closed sourced again. But I agree that it was pretty bad for its [open source] users that it was made closed source. Terrible move for adoption.
  - spullara 4y ago
    I mean, that was Apple. It was no small feat to get them to later open source it.
  - jen20 4y ago
    > FDB fucked it's users back in 2015 when it abruptly closed shop and went closed source This is simply untrue - it was not open source prior to acquisition either. The first point at which FDB was open-source was 2018.
    - metadat 4y ago
      FDB screwed their users in 2015 when they abruptly closed shop with no further development, support, or even a migration plan. Gotta get that Apple money.
      - jen20 4y ago
        Many of those users continued to be users, with source licenses, who are users to this day. If you had a commercial contract you were unaffected. Perhaps not if you were freeloading…
        metadat 4y ago
        Ah yes, it was my own fault for believing in them.
endisneigh 4y ago
It's a shame that FoundationDB went closed source when it did, since that was the key period (imo) for database exploration - the early 2010s. If it would've been more popularized then I imagine most people would be using it now. The nice thing about FDB is that after 3 plus nodes, you can simply add nodes using your cloud provider of choice and it scales pretty nicely while still giving your high availability and fault tolerance. It's pretty funny to me though to see this - I've been spending a few days building a simple database on top of FDB that supports indexes, secondary indexes and schema migrations backed by json-schema (very, very similar to this, totally independently!) To get into a little bit, it's not super difficult if you use FDB. FDB is a very bare key value store. It's incredibly low level. You don't even get a notion of collections. You have to implement everything yourself. what it does give you, however, is a giant hash map that will guarantee that items are in sorted order. so to build what I was describing it's easy: a collection can be a tuple to map: (your-app, your-collection, _id, your-model-id-number) => json e.g. (hn-app, users, _id, 1) => { _id: 1, username: endisneigh } (hn-app, users, _id, 2) => { _id: 1, username: reader } an index can be something like: (your-app, your-collection, your-field-to-index, index-value, _id, your-model-id-number) => json e.g. (hn-app, users, username, endisneigh _id, 1) => { _id: 1, username: endisneigh } (hn-app, users, username, reader, _id, 1) => { _id: 1, username: reader} Because FDB gives you transactions, you maintain the index by populating the keys according to the pattern above on your create* and update* operations. To do something like a schema migration, FDB gives you a get_range operation that you can use to find all keys that have a prefix. So what you'd do is store a value indicating that you're doing a migration into the database, iterate through the keys in a batch (so it's all a single transaction), update the value in the db saying what the last key you've migrated is, and continue until you've done all of the keys. A lot of stuff is pretty trivial once you assume the underlying semantics are solved. I've seen some interesting projects involving things like using FDB as a virtual file system for SQLite, but the problem with that is FDBs primitives are actually flexible, and so there are optimizations you can make if you built it using those primitives from the beginning, as opposed to using FDB as simple a key value store without taking advantage of the transactions. ------- On another note, one idea I've had (feel free to steal) is to reimplement IndexedDB using FoundationDB. IndexedDB is also a key value store which supports transactions, like FDB. Obviously IDB is not networked. The idea is that if you can semantically map IDB with FDB, then you could use FDB as a store for IDB (scoped to the user, of course). And then any app that uses IDB for its storage (like an offline app) could use FDB as the backing without having to use a different set of data structures to actually represent the storage.
asiejhfasdfhj 4y ago
Kudos! you guys tweaked the sales pitch in a single day!
- ovaistariq 4y ago
  I hope you are sold now :) I don't really have experience pitching our own product. So the feedback on HN has been enlightening.
deleted 4y ago
[deleted]
Scarbutt 4y ago
Is there a 'limits' doc somewhere? how many objects can an array hold? what is the depth limit for nested data?
- ovaistariq 4y ago
  The limitations are listed here: https://docs.tigrisdata.com/apidocs/#section/Limitations https://docs.tigrisdata.com/apidocs/#section/Limitations Apart from the size of the document, there is no limit on the size of the array or the depth of nested data. We plan on substantially increasing the document size limit.
  - frellus 4y ago
    "The maximum allowed document size is 100KB" I suppose this is due to the FDB limitations, so this obviously isn't a blobstore nor will it ever be (?). For example, we need to store video and image files which are easily 100KB - 1GB in size. Tigris or FDB are great to store metadata in (and the metadata is just as important to us as anything) but the blob storage is a bit of a problem. Would be interesting to integrate something with, for example, MinIO or S3.
    - hchaudhary 4y ago
      Thank you for explaining your use case. This is one of the features we have in mind to support use cases similar to yours for storing videos, images, etc. But this may not happen this year. It would be great if you could add something here https://github.com/tigrisdata/tigris/issues https://github.com/tigrisdata/tigris/issues that will help us track and monitor similar use cases.
      - frellus 4y ago
        Done, and thanks a lot for the invitation to add, happy to contribute https://github.com/tigrisdata/tigris/issues/562 https://github.com/tigrisdata/tigris/issues/562
ithrow 4y ago
Can typescript be limited to just creating the schema and then use javascript for the rest of your application?
- hchaudhary 4y ago
  You can use our typescript client for everything not only for Schemas. Here is a section that has examples on how to use it https://docs.tigrisdata.com/typescript/ https://docs.tigrisdata.com/typescript/. Please let us know if anything is confusing.
  - ithrow 4y ago
    Didn't explain myself well, meant I want to use Javascript only and maybe typescript only for the schemas. I understand that Tigris exposes a HTTP endpoint and is all json but wondered how well the Typescript SDK will play for pure Javascript development.
    - jmjoshi 4y ago
      I am the TypeScript client author here. Tigris exposes native HTTP apis and we have openapi YAML available at https://github.com/tigrisdata/tigris-api/blob/main/server/v1/openapi.yaml https://github.com/tigrisdata/tigris-api/blob/main/server/v1... The native Javascript SDK is on our roadmap. In the meantime if you would like you can generate Javascript client using openapi generator such as https://blog.logrocket.com/generating-integrating-openapi-services-react/ https://blog.logrocket.com/generating-integrating-openapi-se... We would be happy to accept your contribution.

75 points by ovaistariq 4y ago

Building distributed database systems correctly from the ground up is a notoriously hard problem. We have seen this firsthand building Docstore at Uber. This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Vendors are spending so much time rebuilding existing solutions, they end up not solving the actual end users’ problems, although ostensibly that's why they decided to create a new data platform in the first place! In this post we talk about our approach to building Tigris - the open source developer data platform. We talk about why we chose to build on top of FoundationDB, one of the most reliable distributed KV store with an amazing correctness story. We also go into detail about our experience using it.

metadat 4y ago
Are you Himank or Yevgeniy? Good luck with your project, FDB fucked it's users back in 2015 when it abruptly closed shop and went closed source. Hopefully some good can come of it yet.
- ovaistariq 4y ago
  It has significant usage across large companies such as Apple, Snowflake, Epic Games, VMWare, etc. I don't see it going closed source. Besides that, here Tigris is taking the responsibility for the product as it is abstracting FoundationDB from the end-user.
- endisneigh 4y ago
  I can assure you from FDB will not be closed sourced again. But I agree that it was pretty bad for its [open source] users that it was made closed source. Terrible move for adoption.
- spullara 4y ago
  I mean, that was Apple. It was no small feat to get them to later open source it.
- jen20 4y ago
  > FDB fucked it's users back in 2015 when it abruptly closed shop and went closed source This is simply untrue - it was not open source prior to acquisition either. The first point at which FDB was open-source was 2018.
  - metadat 4y ago
    FDB screwed their users in 2015 when they abruptly closed shop with no further development, support, or even a migration plan. Gotta get that Apple money.
    - jen20 4y ago
      Many of those users continued to be users, with source licenses, who are users to this day. If you had a commercial contract you were unaffected. Perhaps not if you were freeloading…
      - metadat 4y ago
        Ah yes, it was my own fault for believing in them.

endisneigh 4y ago

It's a shame that FoundationDB went closed source when it did, since that was the key period (imo) for database exploration - the early 2010s. If it would've been more popularized then I imagine most people would be using it now. The nice thing about FDB is that after 3 plus nodes, you can simply add nodes using your cloud provider of choice and it scales pretty nicely while still giving your high availability and fault tolerance. It's pretty funny to me though to see this - I've been spending a few days building a simple database on top of FDB that supports indexes, secondary indexes and schema migrations backed by json-schema (very, very similar to this, totally independently!) To get into a little bit, it's not super difficult if you use FDB. FDB is a very bare key value store. It's incredibly low level. You don't even get a notion of collections. You have to implement everything yourself. what it does give you, however, is a giant hash map that will guarantee that items are in sorted order. so to build what I was describing it's easy: a collection can be a tuple to map: (your-app, your-collection, _id, your-model-id-number) => json e.g. (hn-app, users, _id, 1) => { _id: 1, username: endisneigh } (hn-app, users, _id, 2) => { _id: 1, username: reader } an index can be something like: (your-app, your-collection, your-field-to-index, index-value, _id, your-model-id-number) => json e.g. (hn-app, users, username, endisneigh _id, 1) => { _id: 1, username: endisneigh } (hn-app, users, username, reader, _id, 1) => { _id: 1, username: reader} Because FDB gives you transactions, you maintain the index by populating the keys according to the pattern above on your create* and update* operations. To do something like a schema migration, FDB gives you a get_range operation that you can use to find all keys that have a prefix. So what you'd do is store a value indicating that you're doing a migration into the database, iterate through the keys in a batch (so it's all a single transaction), update the value in the db saying what the last key you've migrated is, and continue until you've done all of the keys. A lot of stuff is pretty trivial once you assume the underlying semantics are solved. I've seen some interesting projects involving things like using FDB as a virtual file system for SQLite, but the problem with that is FDBs primitives are actually flexible, and so there are optimizations you can make if you built it using those primitives from the beginning, as opposed to using FDB as simple a key value store without taking advantage of the transactions. ------- On another note, one idea I've had (feel free to steal) is to reimplement IndexedDB using FoundationDB. IndexedDB is also a key value store which supports transactions, like FDB. Obviously IDB is not networked. The idea is that if you can semantically map IDB with FDB, then you could use FDB as a store for IDB (scoped to the user, of course). And then any app that uses IDB for its storage (like an offline app) could use FDB as the backing without having to use a different set of data structures to actually represent the storage.

asiejhfasdfhj 4y ago

Kudos! you guys tweaked the sales pitch in a single day!

ovaistariq 4y ago
I hope you are sold now :) I don't really have experience pitching our own product. So the feedback on HN has been enlightening.

deleted 4y ago

[deleted]

Scarbutt 4y ago

Is there a 'limits' doc somewhere? how many objects can an array hold? what is the depth limit for nested data?

ovaistariq 4y ago
The limitations are listed here: https://docs.tigrisdata.com/apidocs/#section/Limitations https://docs.tigrisdata.com/apidocs/#section/Limitations Apart from the size of the document, there is no limit on the size of the array or the depth of nested data. We plan on substantially increasing the document size limit.
- frellus 4y ago
  "The maximum allowed document size is 100KB" I suppose this is due to the FDB limitations, so this obviously isn't a blobstore nor will it ever be (?). For example, we need to store video and image files which are easily 100KB - 1GB in size. Tigris or FDB are great to store metadata in (and the metadata is just as important to us as anything) but the blob storage is a bit of a problem. Would be interesting to integrate something with, for example, MinIO or S3.
  - hchaudhary 4y ago
    Thank you for explaining your use case. This is one of the features we have in mind to support use cases similar to yours for storing videos, images, etc. But this may not happen this year. It would be great if you could add something here https://github.com/tigrisdata/tigris/issues https://github.com/tigrisdata/tigris/issues that will help us track and monitor similar use cases.
    - frellus 4y ago
      Done, and thanks a lot for the invitation to add, happy to contribute https://github.com/tigrisdata/tigris/issues/562 https://github.com/tigrisdata/tigris/issues/562

ithrow 4y ago

Can typescript be limited to just creating the schema and then use javascript for the rest of your application?

hchaudhary 4y ago
You can use our typescript client for everything not only for Schemas. Here is a section that has examples on how to use it https://docs.tigrisdata.com/typescript/ https://docs.tigrisdata.com/typescript/. Please let us know if anything is confusing.
- ithrow 4y ago
  Didn't explain myself well, meant I want to use Javascript only and maybe typescript only for the schemas. I understand that Tigris exposes a HTTP endpoint and is all json but wondered how well the Typescript SDK will play for pure Javascript development.
  - jmjoshi 4y ago
    I am the TypeScript client author here. Tigris exposes native HTTP apis and we have openapi YAML available at https://github.com/tigrisdata/tigris-api/blob/main/server/v1/openapi.yaml https://github.com/tigrisdata/tigris-api/blob/main/server/v1... The native Javascript SDK is on our roadmap. In the meantime if you would like you can generate Javascript client using openapi generator such as https://blog.logrocket.com/generating-integrating-openapi-services-react/ https://blog.logrocket.com/generating-integrating-openapi-se... We would be happy to accept your contribution.