5 ms·
Are there any good FOSS projects that would let me run my own crawler?
by Lammy 6y ago
Are there any good FOSS projects that would let me run my own crawler?
- devmunchies 6y agoisn't that what Searx is? https://asciimoo.github.io/searx/ https://asciimoo.github.io/searx/
- chromedev 6y agoNo, it is a metasearch engine but not a crawler.
- tobias2014 6y agoYaCy is the only one I'm aware of. See https://www.yacy.net/demonstration_tutorial_screenshot/ https://www.yacy.net/demonstration_tutorial_screenshot/
- surround 6y agoYaCy is a decentralized search engine
- lovelearning 6y agoTry Apache Nutch.
- Lammy 6y agoLooks like Common Crawl switched to this from their homegroan crawler. That's high praise!