3 ms·

Actually, Apple does this for iCloud! They use FoundationDB[1] to store billions of databases, one for each user (plus shared or global databases). See: https:

Actually, Apple does this for iCloud! They use FoundationDB[1] to store billions of databases, one for each user (plus shared or global databases). See: https://read.engineerscodex.com/p/how-apple-built-icloud-to-store-billions https://read.engineerscodex.com/p/how-apple-built-icloud-to-... Discussed on HN at the time: https://news.ycombinator.com/item?id=39028672 https://news.ycombinator.com/item?id=39028672 [1]: https://github.com/apple/foundationdb https://github.com/apple/foundationdb https://en.wikipedia.org/wiki/FoundationDB https://en.wikipedia.org/wiki/FoundationDB

danpalmer 3y ago
> store billions of databases This is sort of true and sort of false. When you think of a "database", if you're thinking of a Postgres database, you're way off the reality of what "database" means here. FoundationDB has a concept called "layers", and essentially they have created a layer that looks like a separate database on top of a layer that is separately encrypted groups of keys. They don't have billions of FoundationDB clusters or machines, and at the infra level, i.e. the instances of the FoundationDB server software, it's unaware of individual "databases". A closer analogy would be like having billions of tables, but even that isn't accurate because in relational databases a table is usually a notably more static concept than data in a table. The closest analogy would be that each of the billions of users has a bunch of rows with a user-id field on them, and there's a proxy that filters everything such that you can view the table as if it only had one user's data in it. To be clear, FoundationDB is awesome and Apple have done some really cool stuff with it, but it's less crazy/impressive than it sounds.
- dexwiz 3y ago
  This sounds like a pretty standard multitenant datastore. Everything has a user/group Id on it, and a logical layer that locks a connection to a specific group.
  - danpalmer 3y ago
    Yeah, the advantage or difference here is that these "layers" are a common design pattern with FoundationDB, several ship in FDB by default, and you're encouraged to make more, so the database certainly has better support than just adding a column for TenantID, but still you're right that it's not too out there.
  - deleted 3y ago
    [deleted]
  - pests 3y ago
    FoundationDB is really just a set of structures and tools to build any type of database you want on top of a solid foundation. "FoundationDB decouples its data storage technology from its data model. FoundationDB’s core ordered key-value storage technology can be efficiently adapted and remapped to a broad array of rich data models. Using indexing as an example, FoundationDB’s core provides no indexing and never will. Instead, a layer provides indexing by storing two kinds of key-values, one for the data and one for the index." https://apple.github.io/foundationdb/layer-concept.html https://apple.github.io/foundationdb/layer-concept.html Then existing standard layers like the Record layer, providing "(very) roughly equivalent to a simple relational database" providing structured types, index, complex types, queries, etc. https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer Or one for documents, which speaks the MongoDB wire protocol https://github.com/FoundationDB/fdb-document-layer https://github.com/FoundationDB/fdb-document-layer

danpalmer 3y ago

> store billions of databases This is sort of true and sort of false. When you think of a "database", if you're thinking of a Postgres database, you're way off the reality of what "database" means here. FoundationDB has a concept called "layers", and essentially they have created a layer that looks like a separate database on top of a layer that is separately encrypted groups of keys. They don't have billions of FoundationDB clusters or machines, and at the infra level, i.e. the instances of the FoundationDB server software, it's unaware of individual "databases". A closer analogy would be like having billions of tables, but even that isn't accurate because in relational databases a table is usually a notably more static concept than data in a table. The closest analogy would be that each of the billions of users has a bunch of rows with a user-id field on them, and there's a proxy that filters everything such that you can view the table as if it only had one user's data in it. To be clear, FoundationDB is awesome and Apple have done some really cool stuff with it, but it's less crazy/impressive than it sounds.

dexwiz 3y ago
This sounds like a pretty standard multitenant datastore. Everything has a user/group Id on it, and a logical layer that locks a connection to a specific group.
- danpalmer 3y ago
  Yeah, the advantage or difference here is that these "layers" are a common design pattern with FoundationDB, several ship in FDB by default, and you're encouraged to make more, so the database certainly has better support than just adding a column for TenantID, but still you're right that it's not too out there.
- deleted 3y ago
  [deleted]
- pests 3y ago
  FoundationDB is really just a set of structures and tools to build any type of database you want on top of a solid foundation. "FoundationDB decouples its data storage technology from its data model. FoundationDB’s core ordered key-value storage technology can be efficiently adapted and remapped to a broad array of rich data models. Using indexing as an example, FoundationDB’s core provides no indexing and never will. Instead, a layer provides indexing by storing two kinds of key-values, one for the data and one for the index." https://apple.github.io/foundationdb/layer-concept.html https://apple.github.io/foundationdb/layer-concept.html Then existing standard layers like the Record layer, providing "(very) roughly equivalent to a simple relational database" providing structured types, index, complex types, queries, etc. https://github.com/FoundationDB/fdb-record-layer https://github.com/FoundationDB/fdb-record-layer Or one for documents, which speaks the MongoDB wire protocol https://github.com/FoundationDB/fdb-document-layer https://github.com/FoundationDB/fdb-document-layer