3 ms·
Can only speak for my self, but I use my own dinky crawler because CC doesn't really solve any problems I have operating my search engine, and its unwieldy size
by marginalia_nu 4y ago
Can only speak for my self, but I use my own dinky crawler because CC doesn't really solve any problems I have operating my search engine, and its unwieldy size creates new problems I didn't have doing my own crawling.
Crawling just isn't the hard part of building a search engine. Sure there are pitfalls and obstacles, but they're all fairly solvable.