3 ms·
That's definitely a good question. A few thoughts here (I am one of the authors). The "bring your own data layer" has several goals: (1) it is really helpful i
by sewen 2y ago
That's definitely a good question. A few thoughts here (I am one of the authors). The "bring your own data layer" has several goals:
(1) it is really helpful in getting good latencies.
(2) it makes it self-contained, so easy to start and run anywhere
(3) There is a simplicity in the deeply integrated architecture, where consensus of the log, fencing of the state machine leaders, etc. goes hand in hand. It removes the need to coordinate between different components with different paradigms (pub-sub-logs, SQL databases, etc) that each have their own consistency/transactions. And coordination avoidance is probably the best one can do in distributed systems. This ultimately leads also to an easier to understand behavior when running/operating the system.
(4) The storage is actually pluggable, because the internal architecture uses virtual consensus. So if the biggest ask from users would be "let me use Kafka or SQS FIFO" then that's doable.
We'd love to go about this the following way: We aim to provide an experience than is users would end up preferring to maintaining multiple clusters of storage systems (like Cassandra + ElasticSearch + X server and Y queues) though this integrated design. If that turns out to not be what anyone wants, we can still relatively easily work with other systems.