4 ms·
We need more links indexed. The crawler that I wrote is pretty slow and right now we don't have a lot of documents indexed. I will be replacing the crawler I wr
by brentadamson 8y ago
We need more links indexed. The crawler that I wrote is pretty slow and right now we don't have a lot of documents indexed. I will be replacing the crawler I wrote with a faster one soon. Until then I have been using !bangs.
- ericst 8y agoHave you ever heard of https://commoncrawl.org/ https://commoncrawl.org/? Maybe it helps you to bootstrap. It's cool to see some new things happening in the search engine world.
- brentadamson 8y agoI've considered that but didn't really know anything about the organization and how long they'll be around. We're gonna give Colly a try: https://github.com/gocolly/colly https://github.com/gocolly/colly.