3 ms·
Common Crawl's data is a bit inflated due to how it's stored. WARC is the opposite of a compression format ;-) I think you are correct in that one of the prim
by marginalia_nu 3y ago
Common Crawl's data is a bit inflated due to how it's stored. WARC is the opposite of a compression format ;-)
I think you are correct in that one of the primary problems faced in search is selecting interesting documents. I think I throw out like 75% of what I crawl because it's not very likely to ever be a good search result. You can probably filter it even more aggressively.
It's a smart avenue to approach search not in the sense of "how can I make this computer cluster big enough to fit the data" but "how can I make this data small enough to fit my computer"..