5 ms·
Why aren't more of them using Common Crawl? Is it flawed?
by mavhc 4y ago
Why aren't more of them using Common Crawl? Is it flawed?
- snowwrestler 4y agoI found it weird that the author references the Common Crawl a couple of times but it does not have its own entry on the page. I’m not familiar with what it is… guess I’ll just Google it.
- marginalia_nu 4y agoCan only speak for my self, but I use my own dinky crawler because CC doesn't really solve any problems I have operating my search engine, and its unwieldy size creates new problems I didn't have doing my own crawling. Crawling just isn't the hard part of building a search engine. Sure there are pitfalls and obstacles, but they're all fairly solvable.