3 ms·
This a great and interesting offering! I think this fits well with fly.io and their model of computing. I now wish that I had engaged with this idea that was v
by no_wizard 5y ago
This a great and interesting offering! I think this fits well with fly.io and their model of computing.
I now wish that I had engaged with this idea that was very similar to litestream that I had about a year and half ago. I always thought SQLite just needed a distribution layer to be extremely effective as a distributed database of sorts. Its flat file architecture means its easy to provision, restore and backup. SQLite also has incremental snapshotting and re-producible WAL logs that can be used to do incremental backups, restores, writes etc. It just needs a "frontend" to handle those bits. Latency has gotten to the point where you can replicate a database by its continued snapshots (which is, on a high level, what litestream appears to be doing) being propagated out to object / blob storage. You could even achieve brute force consensus with this approach if you ran it in a truly distributed way (though RAFT is probably more efficient).
Reason I didn't do this? I thought to myself - why in the world in 2020 would someone choose to use SQLite at scale instead of something like Firebase, Spanner, Fauna, or even Postgres? So after I did an initial prototype (long gone, never pushed it to GitHub) I just felt like...there was no appetite for it.
Now I regret!
Just a long winded way of saying, congrats! This is awesome! Thanks for doing exactly what I wanted to do but didn't have the guts to follow through with.
- epilys 5y agoI implemented exactly this setup, in Rust, last year for a client. Distributed WAL with write locks on a RAFT scheme. Custom VFS in Rust for sqlite3 to handle the IO. I asked the client to opensource it but it's probably not gonna happen... It's definitely doable though.
- ComputerGuru 5y agoDid you write your own rust raft implementation or reuse something already available?
- epilys 5y agoReused a well known library that uses raft. I don't know if I should mention any more details since it was a private project.
- evntdrvn 5y agothere’s some stuff out there: - https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite - https://github.com/chiselstrike/chiselstore https://github.com/chiselstrike/chiselstore - https://dqlite.io/ https://dqlite.io/ I’m sure there’s more, those are just the ones I remember.