3 ms·
Any recommendations from HN for a write-once (literally once), data storage format that's suitable for network storage? sqlite docs recommend avoiding using it
by kayson 1y ago
Any recommendations from HN for a write-once (literally once), data storage format that's suitable for network storage?
sqlite docs recommend avoiding using it on network storage, though from what I can gather, it's less of an issue if you're truly only doing reads (meaning I could create it locally and then copy it to network storage). Apache Parquet seems promising, and it seems to support indexing now which is an important requirement.
- mcculley 1y agoSQLite works fine over read-only NFS, in my experience. Just only work on an immutable copy and restart your application if ever changing it. If your application is short lived and can only ever see an immutable copy on the path, then it is a great solution.
- nasretdinov 1y agoSQLite does work on NFS even in read-write scenario. Discovered by accident, but my statement still holds. The WAL mode is explicitly not supported over network filesystems, but I guess you don't expect it to :)
- kayson 1y agoMy experience has been the opposite... Lots of db lock and corruption issues. The FAQ doesn't call out WAL specifically, just says don't do it at all: https://www.sqlite.org/faq.html#q5 https://www.sqlite.org/faq.html#q5
- kirici 1y agoI've had multiple flaky issues with SQLite (e.g. non-HA Grafana) on Azure Files using NFS v4.1 leading to locked DBs. Perhaps some implementations work, I'm not gonna rely on it or advise others to do so.
- nasretdinov 1y agoYeah trying to write from several hosts will certainly fail if you don't have advisory locks working, which is not a given, so you are right of course
- kirici 1y agoThese were singular containers, let alone hosts.
- simlevesque 1y agoParquet files are what I use.
- pstuart 1y agoMultiple writers on network storage is the issue. Reading should be totally fine.
- usefulcat 1y agoOrdinary files inside squashfs? https://www.kernel.org/doc/html/latest/filesystems/squashfs.html https://www.kernel.org/doc/html/latest/filesystems/squashfs....
- heavyset_go 1y agoSQLite over NFS works if you have one writer and many readers.