3 ms·
You can use a scalable database that gives you serializable transactions whilst not requiring you to represent your document in the relational model. A good ex
by mike_hearn 3y ago
You can use a scalable database that gives you serializable transactions whilst not requiring you to represent your document in the relational model.
A good example of this architecture would be using FoundationDB [1] with Permazen [2]. In this design there are three layers:
1. A horizontally scaling sorted transactional K/V store. This is provided by FoundationDB. Transactions are automatically ordered within the cluster.
2. A network protocol that can serialize database transactions. Permazen has the ability to do this, you can do things like cache reads, HTTP POST transactions and so on. Stuff you can't easily do with SQL databases.
3. A way to map in-memory objects to/from key/value pairs, with schema migration, indexing and other DB-like features. Permazen also does this.
Permazen can be thought of as an ORM for KV stores. It's a library intended to execute on trusted servers (because the KV store can't do any business logic validation). However, for something like Figma where it's basically trusting the client anyway that doesn't matter. Additionally you can do some tricks with the architecture to support untrusted clients; I've explored these topics with Archie (the Permazen designer) in the past.
The nice thing about this design is that it doesn't require sharding by "file", can scale to large numbers of simultaneous co-authors, and results in a very natural coding model. However, Permazen is a Java library. To use it from a browser would be awkward. That said it has fairly minimal reliance on the JDK. You could probably auto-convert it to Kotlin and then use Kotlin/JS or Kotlin/WASM. But frankly it'd be easier to do that architecture as a real desktop app where you aren't boxed in by the browser's limitations. And of course the design ideas can be implemented in any language, it's just a lot of work.
The writeup mentions a couple of reasons for not using a database:
1. Relational/object mismatch. Permazen+FDB solves this.
2. Cost of a database vs S3. This is mostly an artifact of cloud pricing. Cloud is highly profitable but most of the margin comes from managed databases and other very high level services, not commodity byte storage. Given that FDB is free you could eliminate the cost gap by just running the database yourself, and especially, running it on your own metal.
Because Permazen has a pluggable KV backend and because there are backends that write to files, you can have both worlds - a scalable DB in the cloud and also write to files for individual cases where people don't want to store data on your backend.
https://www.foundationdb.org/ https://www.foundationdb.org/ [1]
https://permazen.io/ https://permazen.io/ [2]