3 ms·

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being acces

Tangentially related: I'd like to use litestream but my SQLite files are several gigabytes, is there a way to lazily download the db only once it's being accessed? (using something like userfaultfd maybe? just thinking out loud)

benbjohnson 4y ago
I'm not sure how that could be implemented with Litestream since it runs as a separate process. It could be possible with LiteFS where it just pages in data on the fly. That's on the roadmap but it's probably still a ways out.
mandeepj 4y ago
> is there a way to lazily download the db only once it's being accessed? Not sure what's your full scenario, because you mentioned "lazily download" so thought you might have a luck here https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630 tl;dr: using http range and a smaller page size - might be the way to go
pkhuong 4y ago
Verneuil (https://github.com/backtrace-labs/verneuil https://github.com/backtrace-labs/verneuil) offers that for read replicas, because backtrace as well has some multi-GB sqlite DBs. It's a VFS (loadable at runtime as a sqlite extension), instead of a filesystem. I don't remember if that's in the docs; commit that adds the configuration field https://github.com/backtrace-labs/verneuil/commit/027318ba743b6d6fd6089b9a57ceba6880abfaea#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R40-R62 https://github.com/backtrace-labs/verneuil/commit/027318ba74... and commit for the enum https://github.com/backtrace-labs/verneuil/blob/e6697498f3baf9d67a1269e926ed4ffa465976a9/src/snapshot.rs#L42-L53 https://github.com/backtrace-labs/verneuil/blob/e6697498f3ba... (the actual implementation is nothing special https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bcd413ab48d5d52eeb07d42ad0e811a1 https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bc...) Issue for the feature: https://github.com/backtrace-labs/verneuil/issues/12 https://github.com/backtrace-labs/verneuil/issues/12

benbjohnson 4y ago

I'm not sure how that could be implemented with Litestream since it runs as a separate process. It could be possible with LiteFS where it just pages in data on the fly. That's on the roadmap but it's probably still a ways out.

mandeepj 4y ago

> is there a way to lazily download the db only once it's being accessed? Not sure what's your full scenario, because you mentioned "lazily download" so thought you might have a luck here https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630 tl;dr: using http range and a smaller page size - might be the way to go

pkhuong 4y ago

Verneuil (https://github.com/backtrace-labs/verneuil https://github.com/backtrace-labs/verneuil) offers that for read replicas, because backtrace as well has some multi-GB sqlite DBs. It's a VFS (loadable at runtime as a sqlite extension), instead of a filesystem. I don't remember if that's in the docs; commit that adds the configuration field https://github.com/backtrace-labs/verneuil/commit/027318ba743b6d6fd6089b9a57ceba6880abfaea#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R40-R62 https://github.com/backtrace-labs/verneuil/commit/027318ba74... and commit for the enum https://github.com/backtrace-labs/verneuil/blob/e6697498f3baf9d67a1269e926ed4ffa465976a9/src/snapshot.rs#L42-L53 https://github.com/backtrace-labs/verneuil/blob/e6697498f3ba... (the actual implementation is nothing special https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bcd413ab48d5d52eeb07d42ad0e811a1 https://github.com/backtrace-labs/verneuil/commit/b6bdfcf7bc...) Issue for the feature: https://github.com/backtrace-labs/verneuil/issues/12 https://github.com/backtrace-labs/verneuil/issues/12