3 ms·
I think the indexing needs some work. For the query "Kozai", Google, DDG, and Bing all return Wikipedia's page on the Kozai mechanism as either the top result
by antognini 9y ago
I think the indexing needs some work. For the query "Kozai", Google, DDG, and Bing all return Wikipedia's page on the Kozai mechanism as either the top result or the second result. The lone result I get from Jive is a link to what appears to be a Japanese porn site. I can't even upvote the Wikipedia page because it doesn't show up.
- brentadamson 9y agoWe need more links indexed. The crawler that I wrote is pretty slow and right now we don't have a lot of documents indexed. I will be replacing the crawler I wrote with a faster one soon. Until then I have been using !bangs.
- ericst 9y agoHave you ever heard of https://commoncrawl.org/ https://commoncrawl.org/? Maybe it helps you to bootstrap. It's cool to see some new things happening in the search engine world.
- brentadamson 8y agoI've considered that but didn't really know anything about the organization and how long they'll be around. We're gonna give Colly a try: https://github.com/gocolly/colly https://github.com/gocolly/colly.