4 ms·
It would be nice if compressed HTML was on a similar order to filesystem block sizes, but most pages are much larger than 4k bytes even compressed. May depend o
by toast0 4y ago
It would be nice if compressed HTML was on a similar order to filesystem block sizes, but most pages are much larger than 4k bytes even compressed. May depend on the specific site that's being scraped.
- marginalia_nu 4y agoThis isn't a hypothetical. I've used this exact workflow for this exact use case. I'm telling you it's no good. I run a search engine crawler and my average across 100M docs is about 7 Kb when compressed with zstd (fs block size is typically 4 Kb). Some much larger than that of course, but many smaller still. HTML in general compresses absurdly well.