5 ms·
Is there any value in starting with Common Crawl? https://hn.algolia.com/?query=author%3Amarginalia_nu%20common%20crawl&sort=byDate&type=comment https://hn.alg
by password4321 4y ago
Is there any value in starting with Common Crawl?
https://hn.algolia.com/?query=author%3Amarginalia_nu%20common%20crawl&sort=byDate&type=comment https://hn.algolia.com/?query=author%3Amarginalia_nu%20commo... > https://news.ycombinator.com/item?id=32205535#32211292 https://news.ycombinator.com/item?id=32205535#32211292
> It's simply too unwieldy. It's far easier (and cheaper) to do my own crawling at a manageable scale, than it is to work with CC's datasets.
Is there any way to contribute to Common Crawl beyond donating?
https://commoncrawl.org/big-picture/what-you-can-do/ https://commoncrawl.org/big-picture/what-you-can-do/
- marginalia_nu 4y agoYeah I stand by that. I just don't see what Common Crawl would actually help me with, other than making my own data more stale given it would take about as long to download the CC dataset as my own crawl takes to perform (i.e. ~200h). As it stands, crawling isn't the hard part of building a search engine. Don't get me wrong, if you're doing data science and want to access a crawl data set, Common Crawl is amazing.