3 ms·

Did yall build all the database implementation components yourself, or are any of them open source / off the shelf? It sounds like Kafka could do the transactio

by jitl 3y ago

Did yall build all the database implementation components yourself, or are any of them open source / off the shelf? It sounds like Kafka could do the transaction log job, and overall the architecture sounds similar to XTDBv1 which does use Kafka for that component. https://v1-docs.xtdb.com/concepts/what-is-xtdb/ https://v1-docs.xtdb.com/concepts/what-is-xtdb/ I’m also curious if you support unique constraints on non-id columns, or if having writer actions close to the database kind of takes that role.

xixixao 3y ago
We open-sourced a single-machine version of the backend: https://github.com/get-convex/convex-backend https://github.com/get-convex/convex-backend It doesn’t use Kafka, the transaction logic is custom implemented in Rust. Convex does not have special support for unique constraints, as they are easy to implement in your own code (like you wrote, the mutations are happening “inside” the database). This follows the philosophy of making it clear what the database primitives are, and hence what performance you can expect from the database. Of course since your code is all JavaScript/TypeScript, it’s easy to layer in constraints via a library, like in https://labs.convex.dev/convex-ents/schema#unique-fields https://labs.convex.dev/convex-ents/schema#unique-fields
james_cowling 3y ago
Convex cofounder here. Yep the database transaction logic, timestamp assignment, concurrency control, etc is all custom. This is a pretty key requirement in our ability to perform dynamic subscriptions and automatic caching in real time. Internally the very bottom of the stack on the hosted product is actually just a write ahead log on RDS and in the open source product it's just sqlite. We'll likely eventually build our own durable write ahead log, and have plenty of experience doing so, but this hasn't been needed so far.

by jitl 3y ago

xixixao 3y ago

We open-sourced a single-machine version of the backend: https://github.com/get-convex/convex-backend https://github.com/get-convex/convex-backend It doesn’t use Kafka, the transaction logic is custom implemented in Rust. Convex does not have special support for unique constraints, as they are easy to implement in your own code (like you wrote, the mutations are happening “inside” the database). This follows the philosophy of making it clear what the database primitives are, and hence what performance you can expect from the database. Of course since your code is all JavaScript/TypeScript, it’s easy to layer in constraints via a library, like in https://labs.convex.dev/convex-ents/schema#unique-fields https://labs.convex.dev/convex-ents/schema#unique-fields

james_cowling 3y ago

Convex cofounder here. Yep the database transaction logic, timestamp assignment, concurrency control, etc is all custom. This is a pretty key requirement in our ability to perform dynamic subscriptions and automatic caching in real time. Internally the very bottom of the stack on the hosted product is actually just a write ahead log on RDS and in the open source product it's just sqlite. We'll likely eventually build our own durable write ahead log, and have plenty of experience doing so, but this hasn't been needed so far.