3 ms·
You're going to incur insane overhead doing this, since each file's actual size is a multiple of the filesystem's block size, which is similar order of magnitud
by marginalia_nu 4y ago
You're going to incur insane overhead doing this, since each file's actual size is a multiple of the filesystem's block size, which is similar order of magnitude as a compressed HTML file.
- PaulHoule 4y agoGranted. It's fair to stuff them into SQLite or some other kind of composite file but you trade one problem for another. The same content-addressible indexing still makes sense.
- Someone 4y agoThat depends on the file system. It may do block sub allocation (https://en.wikipedia.org/wiki/Block_suballocation https://en.wikipedia.org/wiki/Block_suballocation) (Blocks also need not be large, but in this context that’s a theoretical issue)
- toast0 4y agoIt would be nice if compressed HTML was on a similar order to filesystem block sizes, but most pages are much larger than 4k bytes even compressed. May depend on the specific site that's being scraped.
- marginalia_nu 4y agoThis isn't a hypothetical. I've used this exact workflow for this exact use case. I'm telling you it's no good. I run a search engine crawler and my average across 100M docs is about 7 Kb when compressed with zstd (fs block size is typically 4 Kb). Some much larger than that of course, but many smaller still. HTML in general compresses absurdly well.