3 ms·
Databases are really just data structures and algorithms along with some notion of durability. Client/Server databases are just remote data structures. (E.g. R
by phamilton 3y ago
Databases are really just data structures and algorithms along with some notion of durability.
Client/Server databases are just remote data structures. (E.g. Redis is short for "Remote Dictionary Server")
Sometimes you want your data structures and algorithms to run locally. Could be performance, privacy, cost, or any number of reasons.
Local, in-memory data structures hit a few bottlenecks. First, they may not fit in memory. A mechanism for keeping the dataset in larger storage (e.g. disk) and paging in the necessary bits as needed extends the range of datasets one can comfortably work with locally by quite a bit. That's standard SQLite.
A second potential bottleneck to local data structures is distribution. We carry computers in our pockets, on our watches, in our cars. Delivering large datasets to each of those locations may be impractical. Cloud based VFS allows the benefits of local data structures on the subset they need without requiring them to fetch the entire dataset. That can be a huge win if there's a specific subset they need.
It always depends on the use case, but when the case fits there are a lot of big wins here.
- eterevsky 3y agoOne obvious problem that I see with this approach is that it will break if there is any change in the storage format. With client-server architecture, the server code owns the data format, while in this storage-level remote access case, you have to ensure that all of your clients are updated simultaneously. Depending on your architecture it might or might not be feasible.
- vidarh 3y agoFor the typical uses for this, you'd tend to serve up a version of Sqlite compiled to wasm or JS to the frontend, so you can be sure it's the same one. Sqlite's storage format is also unusually stable: https://www.sqlite.org/formatchng.html https://www.sqlite.org/formatchng.html
- zX41ZdbW 3y agoIt's better to ensure 100% compatibility with the data format so the new server versions can read the old format without conversions. For example, in ClickHouse, you can install the latest version over a version from 7 years ago, and it will work just fine with the old MergeTree tables.