4 ms·

A homegrown crawler is used but most search results are interspersed with existing search engine results.

andai 10y ago
May i ask what is the benefit of using your own as well as an outside solution?
Ed10101 10y ago
I did a couple of searches for various terms and you had a huge index for the various terms. Surely such an index is only possible by hooking into search results from another engine, eventually this will lead to some sort of issue? What's really made it difficult for competitors is being able to match Google's index. Bing crawls a lot of pages but is much more picky about what it index? Also can you say where you are pulling the results from the Yandex API, Bing, Yahoo or Google itself?
trishume 10y ago
Do you really do your own crawling for purposes other than image retrieval? Is it really any good? I highly doubt that your queries are primarily coming from your own engine. The results are too good. It has good ranking, spelling correction, super large indices, and is fast. For example searching "GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA" works even though it only occurs on a few pages and as the blog post that string came from explains, you need super fancy indexing techniques to handle things like that quickly. You also talk about this as if it is a single-person project, which makes it even less likely you made all this from scratch. I like the concept and the parts that you undoubtedly make yourself like the UI, image retrieval and caching, are really good. This is a great site don't get me wrong. I just think you should be more forthcoming about where your results are coming from.
- fivesigma 10y ago
  Where did I imply that queries primarily come from my own engine? Right now only 15-20% of them are. Your search was routed through Bing.
  - trishume 10y ago
    Oh sorry for not being clearer, I didn't mean to imply you misinformed people. I just thought you should be more transparent about where results are coming from. For example it would have been nice to see a "results from Bing" message somewhere on the search page, or an item on the about page saying that you use another search engine's results. It would actually have increased my confidence and impression of your project. I know that you can't have made a fantastic query engine like Bing/Google's without a ton of engineers so you must be using someone else's, and it would have given me a better impression if you said that up front instead of my having to infer that.
- jdc0589 10y ago
  Totally unrelated: oh man. My absolute favorite assignment in undergrad computer science classes involved searching ~50GB of compressed text files containing protein sequences, to see how many times "ATG" or something occurred. It was a ton of fun. Minimum requirements were to get the correct counts. Then it turned in to a competition to see who could make the fastest solution. You had like 8 machines at your disposal, each with the full dataset, to distribute whatever you wanted. I think we did it in 3 languages or so (java, erlang, something else...).
  - mmsimanga 10y ago
    You got my interest, care to share a link with the write up?

andai 10y ago

May i ask what is the benefit of using your own as well as an outside solution?

I did a couple of searches for various terms and you had a huge index for the various terms. Surely such an index is only possible by hooking into search results from another engine, eventually this will lead to some sort of issue? What's really made it difficult for competitors is being able to match Google's index. Bing crawls a lot of pages but is much more picky about what it index? Also can you say where you are pulling the results from the Yandex API, Bing, Yahoo or Google itself?

trishume 10y ago

Do you really do your own crawling for purposes other than image retrieval? Is it really any good? I highly doubt that your queries are primarily coming from your own engine. The results are too good. It has good ranking, spelling correction, super large indices, and is fast. For example searching "GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA" works even though it only occurs on a few pages and as the blog post that string came from explains, you need super fancy indexing techniques to handle things like that quickly. You also talk about this as if it is a single-person project, which makes it even less likely you made all this from scratch. I like the concept and the parts that you undoubtedly make yourself like the UI, image retrieval and caching, are really good. This is a great site don't get me wrong. I just think you should be more forthcoming about where your results are coming from.

fivesigma 10y ago
Where did I imply that queries primarily come from my own engine? Right now only 15-20% of them are. Your search was routed through Bing.
- trishume 10y ago
  Oh sorry for not being clearer, I didn't mean to imply you misinformed people. I just thought you should be more transparent about where results are coming from. For example it would have been nice to see a "results from Bing" message somewhere on the search page, or an item on the about page saying that you use another search engine's results. It would actually have increased my confidence and impression of your project. I know that you can't have made a fantastic query engine like Bing/Google's without a ton of engineers so you must be using someone else's, and it would have given me a better impression if you said that up front instead of my having to infer that.
jdc0589 10y ago
Totally unrelated: oh man. My absolute favorite assignment in undergrad computer science classes involved searching ~50GB of compressed text files containing protein sequences, to see how many times "ATG" or something occurred. It was a ton of fun. Minimum requirements were to get the correct counts. Then it turned in to a competition to see who could make the fastest solution. You had like 8 machines at your disposal, each with the full dataset, to distribute whatever you wanted. I think we did it in 3 languages or so (java, erlang, something else...).
- mmsimanga 10y ago
  You got my interest, care to share a link with the write up?