3 ms·

That would be me. I thought it would be a cool thing to have - previews of a website before you visit it - so I built it. It does not make money but its monthly

That would be me. I thought it would be a cool thing to have - previews of a website before you visit it - so I built it. It does not make money but its monthly cost is minimal at the moment. It is too early to discuss "the agenda" behind it because there is none. If it catches on and becomes popular enough, I have a few ideas on how to make it sustainable without sacrificing any of the privacy features.

drKarl 10y ago
Have you built your own crawler or do you rely on something like YaCy or Searx, or in other existing search engines?
- fivesigma 10y ago
  A homegrown crawler is used but most search results are interspersed with existing search engine results.
  - andai 10y ago
    May i ask what is the benefit of using your own as well as an outside solution?
  - Ed10101 10y ago
    I did a couple of searches for various terms and you had a huge index for the various terms. Surely such an index is only possible by hooking into search results from another engine, eventually this will lead to some sort of issue? What's really made it difficult for competitors is being able to match Google's index. Bing crawls a lot of pages but is much more picky about what it index? Also can you say where you are pulling the results from the Yandex API, Bing, Yahoo or Google itself?
  - trishume 10y ago
    Do you really do your own crawling for purposes other than image retrieval? Is it really any good? I highly doubt that your queries are primarily coming from your own engine. The results are too good. It has good ranking, spelling correction, super large indices, and is fast. For example searching "GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA" works even though it only occurs on a few pages and as the blog post that string came from explains, you need super fancy indexing techniques to handle things like that quickly. You also talk about this as if it is a single-person project, which makes it even less likely you made all this from scratch. I like the concept and the parts that you undoubtedly make yourself like the UI, image retrieval and caching, are really good. This is a great site don't get me wrong. I just think you should be more forthcoming about where your results are coming from.
    - fivesigma 10y ago
      Where did I imply that queries primarily come from my own engine? Right now only 15-20% of them are. Your search was routed through Bing.
      - trishume 10y ago
        Oh sorry for not being clearer, I didn't mean to imply you misinformed people. I just thought you should be more transparent about where results are coming from. For example it would have been nice to see a "results from Bing" message somewhere on the search page, or an item on the about page saying that you use another search engine's results. It would actually have increased my confidence and impression of your project. I know that you can't have made a fantastic query engine like Bing/Google's without a ton of engineers so you must be using someone else's, and it would have given me a better impression if you said that up front instead of my having to infer that.
    - jdc0589 10y ago
      Totally unrelated: oh man. My absolute favorite assignment in undergrad computer science classes involved searching ~50GB of compressed text files containing protein sequences, to see how many times "ATG" or something occurred. It was a ton of fun. Minimum requirements were to get the correct counts. Then it turned in to a competition to see who could make the fastest solution. You had like 8 machines at your disposal, each with the full dataset, to distribute whatever you wanted. I think we did it in 3 languages or so (java, erlang, something else...).
      - mmsimanga 10y ago
        You got my interest, care to share a link with the write up?
cr0sh 10y ago
Based on only spending a couple of minutes with it, I have to say your search engine is the first one that has made me "take notice" in a long time - well, since Google came on the scene. I definitely plan to check it out more fully when I have some free time - but so far, I'm impressed with it. I like it!
eevilspock 10y ago
First, I'm impressed. But I'm having trouble believing that it doesn't cost much. Isn't that lot of bandwidth (getting pages and generating previews), especially if you are doing that on the fly? Or does your crawler generate and cache preview images? If the latter how large is your index? Is the monthly cost minimal simply because you don't yet have much traffic?

drKarl 10y ago

Have you built your own crawler or do you rely on something like YaCy or Searx, or in other existing search engines?

fivesigma 10y ago
A homegrown crawler is used but most search results are interspersed with existing search engine results.
- andai 10y ago
  May i ask what is the benefit of using your own as well as an outside solution?
- Ed10101 10y ago
  I did a couple of searches for various terms and you had a huge index for the various terms. Surely such an index is only possible by hooking into search results from another engine, eventually this will lead to some sort of issue? What's really made it difficult for competitors is being able to match Google's index. Bing crawls a lot of pages but is much more picky about what it index? Also can you say where you are pulling the results from the Yandex API, Bing, Yahoo or Google itself?
- trishume 10y ago
  Do you really do your own crawling for purposes other than image retrieval? Is it really any good? I highly doubt that your queries are primarily coming from your own engine. The results are too good. It has good ranking, spelling correction, super large indices, and is fast. For example searching "GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA" works even though it only occurs on a few pages and as the blog post that string came from explains, you need super fancy indexing techniques to handle things like that quickly. You also talk about this as if it is a single-person project, which makes it even less likely you made all this from scratch. I like the concept and the parts that you undoubtedly make yourself like the UI, image retrieval and caching, are really good. This is a great site don't get me wrong. I just think you should be more forthcoming about where your results are coming from.
  - fivesigma 10y ago
    Where did I imply that queries primarily come from my own engine? Right now only 15-20% of them are. Your search was routed through Bing.
    - trishume 10y ago
      Oh sorry for not being clearer, I didn't mean to imply you misinformed people. I just thought you should be more transparent about where results are coming from. For example it would have been nice to see a "results from Bing" message somewhere on the search page, or an item on the about page saying that you use another search engine's results. It would actually have increased my confidence and impression of your project. I know that you can't have made a fantastic query engine like Bing/Google's without a ton of engineers so you must be using someone else's, and it would have given me a better impression if you said that up front instead of my having to infer that.
  - jdc0589 10y ago
    Totally unrelated: oh man. My absolute favorite assignment in undergrad computer science classes involved searching ~50GB of compressed text files containing protein sequences, to see how many times "ATG" or something occurred. It was a ton of fun. Minimum requirements were to get the correct counts. Then it turned in to a competition to see who could make the fastest solution. You had like 8 machines at your disposal, each with the full dataset, to distribute whatever you wanted. I think we did it in 3 languages or so (java, erlang, something else...).
    - mmsimanga 10y ago
      You got my interest, care to share a link with the write up?

cr0sh 10y ago

Based on only spending a couple of minutes with it, I have to say your search engine is the first one that has made me "take notice" in a long time - well, since Google came on the scene. I definitely plan to check it out more fully when I have some free time - but so far, I'm impressed with it. I like it!

eevilspock 10y ago

First, I'm impressed. But I'm having trouble believing that it doesn't cost much. Isn't that lot of bandwidth (getting pages and generating previews), especially if you are doing that on the fly? Or does your crawler generate and cache preview images? If the latter how large is your index? Is the monthly cost minimal simply because you don't yet have much traffic?