4 ms·
Does CC publish the methodology for how they determine what to crawl. More particularly, how do they determine what not to crawl.
by 1vuio0pswjnm7 2y ago
Does CC publish the methodology for how they determine what to crawl. More particularly, how do they determine what not to crawl.
- arnaudsm 2y agoYes a few big sites are missing, notably reddit. Most of CC is spam though, the real useful content is really small. I'm experimenting my own search engine at the moment, and am considering to make it public at some point. It's not that impossible of a task!