17 ms·

Databases is an extremely slow-maturing area, similar to programming languages, but are all deviations from Postgres shiny and hipster? The idea of colocating

by klabb3 2y ago

Databases is an extremely slow-maturing area, similar to programming languages, but are all deviations from Postgres shiny and hipster? The idea of colocating data and behavior is really a quantifiable reduction in complexity. It removes latency and bandwidth concerns, which means both operational concerns and development concerns (famously the impact of the N+1 problem is greatly reduced). You can absolutely argue that networked Postgres is better for other reasons (and you may be right) but SQLite is about as boring and predictable as you can get, with known strong advantages. This is the reason it’s getting popular on the server. That said, I don’t like the idea of creating many small databases very much - as they suggest with Durable Objects. That gives noSQL nightmares - breaking all kinds of important invariants of relational dbs. I think it’s much preferable to use SQLite as a monolithic database like it’s done in their D1 product.

crabmusket 2y ago
> That gives noSQL nightmares - breaking all kinds of important invariants of relational dbs IMO Durable Objects map well to use cases where there actually are documents. Think of Figma. There is a ton of data that lives inside the literal Figma document. It would be awful to have a relational table for like "shapes" with one row per rectangle across Figma's entire customer base. That's just not an appropriate use of a relational database. So let's say I built Figma on MongoDB, where each Figma document is a Mongo document. That corresponds fairly straightforwardly to each Figma document being a Durable Object instance, using either the built-in noSQL storage that Durable Objects already have, or a small Sqlite relational database which does have a "shapes" table, but only containing the shapes in this one document.
- jchanimal 2y ago
  We are wrestling with questions like this on the new document database we’re building. A database should correspond to some administrative domain object. Today in Fireproof a database is a unit of sharing, but we are working toward a broader model where a database corresponds to an individual application’s state. So one database is all the shared documents not just a single unit of sharing. These small changes early on can have big impact later. If you’re interested in these sort of design questions, the Fireproof Discord is where we are hashing out the v0.20 api. (I was an early contributor to Apache CouchDB. Damien Katz, creator of CouchDB, is helping with engineering and raised these questions recently, along with other team members.)
- klabb3 2y ago
  > Durable Objects map well to use cases where there actually are documents Right. I wouldn’t dispute this. This is akin to a file format from software back in the day (like say photoshop but now with multiplayer). What this means is that you get different compatibility boundaries and you relinquish centralized control and ability to do transparent migrations and analysis. For all intents and purposes, the documents should be more or less opaque and self-contained. I personally like this, but I also recognize that most web engineers of our current generation are not used to think in this disciplined and defensive way upfront.
deleted 2y ago
[deleted]
masterj 2y ago
If you adopt a wide-column db like Cassandra or DynamoDB, don’t you have to pick a shard for your table? The idea behind Durable Objects seems similar
- simpsond 2y ago
  You have a row key, which gets consistently hashed to a shard / node on the ring.
8n4vidtmkvmk 2y ago
N+1 problem is also reduced if you keep your one and only server next to your one and only database. This was actually the solution we came up with at a very big global company. Well, not 1 server, but 1 data center. If your write leaders are all in one place it apparently doesn't matter that everything else is global, for certain write requests at least.

by klabb3 2y ago

crabmusket 2y ago

> That gives noSQL nightmares - breaking all kinds of important invariants of relational dbs IMO Durable Objects map well to use cases where there actually are documents. Think of Figma. There is a ton of data that lives inside the literal Figma document. It would be awful to have a relational table for like "shapes" with one row per rectangle across Figma's entire customer base. That's just not an appropriate use of a relational database. So let's say I built Figma on MongoDB, where each Figma document is a Mongo document. That corresponds fairly straightforwardly to each Figma document being a Durable Object instance, using either the built-in noSQL storage that Durable Objects already have, or a small Sqlite relational database which does have a "shapes" table, but only containing the shapes in this one document.

jchanimal 2y ago
We are wrestling with questions like this on the new document database we’re building. A database should correspond to some administrative domain object. Today in Fireproof a database is a unit of sharing, but we are working toward a broader model where a database corresponds to an individual application’s state. So one database is all the shared documents not just a single unit of sharing. These small changes early on can have big impact later. If you’re interested in these sort of design questions, the Fireproof Discord is where we are hashing out the v0.20 api. (I was an early contributor to Apache CouchDB. Damien Katz, creator of CouchDB, is helping with engineering and raised these questions recently, along with other team members.)
klabb3 2y ago
> Durable Objects map well to use cases where there actually are documents Right. I wouldn’t dispute this. This is akin to a file format from software back in the day (like say photoshop but now with multiplayer). What this means is that you get different compatibility boundaries and you relinquish centralized control and ability to do transparent migrations and analysis. For all intents and purposes, the documents should be more or less opaque and self-contained. I personally like this, but I also recognize that most web engineers of our current generation are not used to think in this disciplined and defensive way upfront.

deleted 2y ago

[deleted]

masterj 2y ago

If you adopt a wide-column db like Cassandra or DynamoDB, don’t you have to pick a shard for your table? The idea behind Durable Objects seems similar

simpsond 2y ago
You have a row key, which gets consistently hashed to a shard / node on the ring.

8n4vidtmkvmk 2y ago

N+1 problem is also reduced if you keep your one and only server next to your one and only database. This was actually the solution we came up with at a very big global company. Well, not 1 server, but 1 data center. If your write leaders are all in one place it apparently doesn't matter that everything else is global, for certain write requests at least.