5 ms·

I’m working on a search engine crawled by users themselves. If I think about it, I'm not interested in 99% of the web. Therefore I don't need to deal with exhau

I’m working on a search engine crawled by users themselves. If I think about it, I'm not interested in 99% of the web. Therefore I don't need to deal with exhaustively crawling the entire web. Hypothetically, if I could access only the sites visited by the HN crowd, that would be fine by me. Somebody in an art community might feel similarly. If there were some metric the crawlers could use to rate a page (time spent on page, manually downvote, etc), then the signal/noise ratio would be really good. Obviously if I wanted to do a really broad search, I'd use a typical search engine, but if it's tech related, I wouldn't mind pages that have been browsed by HN.

dominotw 5y ago
this is such a cool idea. i am interested.
- desertraven 5y ago
  Thank you!
another_story 5y ago
You want something where users voluntarily (and anonymously) submit their search histories and they can be compared? That sounds interesting.
dpifke 5y ago
I'd be pretty happy with a search engine that lived on my local machine, just for pages I've already visited. That might be a good first step towards what you're planning. Step two would then be figuring out how to do federated queries across users with similar interests. From a privacy standpoint, I'd be less uncomfortable letting random strangers run (sanitized) queries against my browsing history (stored locally), than I would be uploading my browser history to a third party.
- desertraven 5y ago
  Great points, thank you. I've been wondering how to do it in a privacy-preserving way. Also the current implementation is a browser extension, I wonder if there may be a better architecture.
  - wanderingmind 5y ago
    Can you point to the extension for people to test
premek 5y ago
do you have anything we can see ?
- desertraven 5y ago
  Still pretty early stages. And I still have some decisions to make regarding the architecture of it. I'd like to do a ShowHN (my first), in the near future.
xeromal 5y ago
I kind of dig this idea.
- desertraven 5y ago
  Thank you! It's been slow progress because I'm a freelancer in a quiet period, so that's taken precedent. But will start posting progress soon.
  - powersurge360 5y ago
    I suspect this might be a tongue in cheek reference to digg, a site that does a very similar thing. It was popular for a while before being eaten alive by Reddit. Don’t let that dissuade you, though, there is room to add a new dimension to it or do things differently. https://en.wikipedia.org/wiki/Digg https://en.wikipedia.org/wiki/Digg
matt_heimer 5y ago
Are you old enough to remember https://en.wikipedia.org/wiki/Webring https://en.wikipedia.org/wiki/Webring ? Take that idea and evolve it into a federated search service. If your site is in the "fishing" ring then your pages get indexed into the fishing index and the search widget on your site would potentially return results from all the sites in the same ring. I guess I want SearchRing.
asicsp 5y ago
>if it's tech related https://notrashsearch.github.io/about.html https://notrashsearch.github.io/about.html might help (https://news.ycombinator.com/item?id=29774456 https://news.ycombinator.com/item?id=29774456) There's also https://search.marginalia.nu/ https://search.marginalia.nu/ (https://news.ycombinator.com/item?id=28550764 https://news.ycombinator.com/item?id=28550764)
numeri 5y ago
YaCy [1] already exists, but I've never tried it. I've been wanting to set it up (locally at first, not participating in the decentralized network, unless I really like it) to crawl only sites I choose to add to the list – I was thinking perhaps just blogs and documentation that I like, along with anything I bookmark. That being said, I think YaCy looks dated, the website has several typos, and I get the impression that not many people are still active in the network. I feel that a new project with similar technology in a "hotter" language (just for popularity's sake) could get a lot more traction. [1] https://yacy.net/ https://yacy.net/
challenger-derp 5y ago
Are there more compelling use-cases than creating public-good kind of whitelists? What incentives encourage sharing of websites? I can list some undesirable incentives: marketing, "shilling" products, MLM promotions. Don't get me wrong, i'm not a cynic, rather, i'm trying to play the part of devil's advocate here. Overall, i'm excited about the idea as I'm quite certain that altruism will prevail and the community will develop well curated website recommendations that make navigating the web a better experience.
husamia 5y ago
check out https://kagi.com/faq https://kagi.com/faq

dominotw 5y ago

this is such a cool idea. i am interested.

desertraven 5y ago
Thank you!

another_story 5y ago

You want something where users voluntarily (and anonymously) submit their search histories and they can be compared? That sounds interesting.

dpifke 5y ago

I'd be pretty happy with a search engine that lived on my local machine, just for pages I've already visited. That might be a good first step towards what you're planning. Step two would then be figuring out how to do federated queries across users with similar interests. From a privacy standpoint, I'd be less uncomfortable letting random strangers run (sanitized) queries against my browsing history (stored locally), than I would be uploading my browser history to a third party.

desertraven 5y ago
Great points, thank you. I've been wondering how to do it in a privacy-preserving way. Also the current implementation is a browser extension, I wonder if there may be a better architecture.
- wanderingmind 5y ago
  Can you point to the extension for people to test

premek 5y ago

do you have anything we can see ?

desertraven 5y ago
Still pretty early stages. And I still have some decisions to make regarding the architecture of it. I'd like to do a ShowHN (my first), in the near future.

xeromal 5y ago

I kind of dig this idea.

desertraven 5y ago
Thank you! It's been slow progress because I'm a freelancer in a quiet period, so that's taken precedent. But will start posting progress soon.
- powersurge360 5y ago
  I suspect this might be a tongue in cheek reference to digg, a site that does a very similar thing. It was popular for a while before being eaten alive by Reddit. Don’t let that dissuade you, though, there is room to add a new dimension to it or do things differently. https://en.wikipedia.org/wiki/Digg https://en.wikipedia.org/wiki/Digg

matt_heimer 5y ago

Are you old enough to remember https://en.wikipedia.org/wiki/Webring https://en.wikipedia.org/wiki/Webring ? Take that idea and evolve it into a federated search service. If your site is in the "fishing" ring then your pages get indexed into the fishing index and the search widget on your site would potentially return results from all the sites in the same ring. I guess I want SearchRing.

asicsp 5y ago

>if it's tech related https://notrashsearch.github.io/about.html https://notrashsearch.github.io/about.html might help (https://news.ycombinator.com/item?id=29774456 https://news.ycombinator.com/item?id=29774456) There's also https://search.marginalia.nu/ https://search.marginalia.nu/ (https://news.ycombinator.com/item?id=28550764 https://news.ycombinator.com/item?id=28550764)

numeri 5y ago

YaCy [1] already exists, but I've never tried it. I've been wanting to set it up (locally at first, not participating in the decentralized network, unless I really like it) to crawl only sites I choose to add to the list – I was thinking perhaps just blogs and documentation that I like, along with anything I bookmark. That being said, I think YaCy looks dated, the website has several typos, and I get the impression that not many people are still active in the network. I feel that a new project with similar technology in a "hotter" language (just for popularity's sake) could get a lot more traction. [1] https://yacy.net/ https://yacy.net/

challenger-derp 5y ago

Are there more compelling use-cases than creating public-good kind of whitelists? What incentives encourage sharing of websites? I can list some undesirable incentives: marketing, "shilling" products, MLM promotions. Don't get me wrong, i'm not a cynic, rather, i'm trying to play the part of devil's advocate here. Overall, i'm excited about the idea as I'm quite certain that altruism will prevail and the community will develop well curated website recommendations that make navigating the web a better experience.

husamia 5y ago

check out https://kagi.com/faq https://kagi.com/faq