2 ms·
Yeah I stand by that. I just don't see what Common Crawl would actually help me with, other than making my own data more stale given it would take about as lon
by marginalia_nu 4y ago
Yeah I stand by that.
I just don't see what Common Crawl would actually help me with, other than making my own data more stale given it would take about as long to download the CC dataset as my own crawl takes to perform (i.e. ~200h).
As it stands, crawling isn't the hard part of building a search engine. Don't get me wrong, if you're doing data science and want to access a crawl data set, Common Crawl is amazing.