4 ms·
Distributed SQLite with Elixir
- deleted 4y ago[deleted]
- Groxx 4y agoSeems hugged to death: https://web.archive.org/web/20230215152735/https://silbernagel.dev/posts/distributed-sqlite-with-elixir https://web.archive.org/web/20230215152735/https://silbernag...
- mattfrommars 4y ago> This is because we are not replicating the data between the instances. This is a problem Elixir is built for… What feature is this and does it not really exist in C# or Java?
- silbermm 4y agoThe feature is distributed message passing (basically an Actor model) https://www.erlang.org/doc/reference_manual/distributed.html https://www.erlang.org/doc/reference_manual/distributed.html You can get this in other languages with the help of middleware like RabbitMQ or Kafka, but not many languages have it built in.
- nesarkvechnep 4y agoYou can’t get it in C# or Java. The BEAM virtual machine is build upon the idea of lightweight processes. You might get an imitation in other mainstream places but not the real thing. Take a look at the Actor Model if you haven’t heard of it.
- mattfrommars 4y ago> might get an imitation in other mainstream places but not the real thing. Could you further elaborate on this? I am familiar with Actor Model. I had work related to Akka in Scala that works on the principles of Actor Model.
- weatherlight 4y agoErlang uses a preemptive scheduler for process scheduling, allowing for parallelization on multiple CPUs, while Scala distinguishes between thread-based and event-based actors. Thread-based actors can perform long-running computations without hindering other actors, but use more memory and scheduling overhead. Event-based actors are more lightweight but not suitable for parallelism as they execute on the same thread.
- nl 4y agoThis is such a weird post. They build a distributed counter that is consistent across a cluster using the Elixir message passing thing, but there's this whole thing about using the Fly Distributed SQLite thing as well. But if they wanted to build a distributed counter they could choose to use Postgres at the first step and then it will just work.
- Joel_Mckay 4y agoFor elixir+phoenix a cockroachDB store abstraction is natively supported. I like the design in many ways, but still need to find something noncritical to test the solution on before clearing it for production. =)
- qaq 4y agocockroachDB uses Postgres wire protocol and SQL
- Joel_Mckay 4y agoCross compatibility is a huge plus from my perspective. However, there are some designs one should avoid... like auto increment indices etc. =)
- ammmir 4y agoManually replicating writes to all nodes seems pretty fragile. At some point you're going to run into classic distributed systems issues, like netsplits, and reconciling the latest count. Maybe a CRDT implementation would avoid these issues alltogether? But it sounds like LiteFS (with a single writer?) might be a pretty nice setup if it handles replication for you, as you probably don't want to build a Raft or Paxos system yourself.
- im_down_w_otp 4y agoLitestream isn't for those kinds of use cases. It's for DR, not availability and/or scale.
- geysersam 4y agoReplicating writes to all nodes does not sound scalable. Is it?
- simonw 4y agoLitestream is different from LiteFS: same author, different projects. LiteFS provides replication with the ability to serve read traffic from those replicas. Litestream provides disaster recovery by streaming a constant backup to an S3 bucket (or similar).
- avinassh 4y agoWhats DR?
- flowcont 4y agoDisaster Recovery
- pstuart 4y agoI think the cleaner thing would have been doing event sourcing utilizing OTP and rendered to local SQLite instances.
- TimWillebrands 4y agoDoesn't event-sourcing require a central datastore where all events are persisted before building the sqlite readmodels?