4 ms·
> What is the advantage of moving around .sqlite-files, over just loading the (compressed) WARCs into sqlite databases when you need them? The .warc spec is id
by fforflo 4y ago
> What is the advantage of moving around .sqlite-files, over just loading the (compressed) WARCs into sqlite databases when you need them?
The .warc spec is ideal. I'm not saying we replace it (ref. xkcd: standards).
On top of what uniqueid said, "loading" is much slower and more cumbersome than it sounds.
I'm not saying SQLite will replace text (maybe my aphorism sounded too firm). I'm saying that maybe along with the .warc.gz archives at rest, one could have .sql.gz files at rest as well.
In other words: why not move ACID-compliant archives moving around?
- marginalia_nu 4y agoSeems like the benefit of sqlite is the sort of usecases where you maybe don't want to load everything you've crawled into a search engine, but want to be able to cherrypick the data and retrieve specific documents for further processing. Which is certainly a use case that exists, and indeed not really what WARC is designed for.
- traverseda 4y agoGreat for most end user facing applications though.
- marginalia_nu 4y agoEnd-user facing applications usually don't consume website crawls, do they? That's impractical for many reasons, the sheer size alone being perhaps the biggest obstacle. If you want to do something like have an offline copy of a website, ZIM[1] is a far more suitable format as it's extremely space-efficient and also fast. [1] https://docs.fileformat.com/compression/zim/ https://docs.fileformat.com/compression/zim/