8 ms·

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google a

There have been a few search engines out recently. I'm curious how people evaluate them quickly. I've realized my searching is basically optimized for google and the web that has grown up around it. Also, in 1998 I wasn't as aware of what was out there as I am now. It's pretty rare (even if its possible) that I do a search and come across a completely new site that I haven't heard of before, for anything nontrivial. That was different when search began. Google is now almost a convenience. If I have a coding question, I search for "turn list of tensors into tensor" or whatever but I'm really looking for SO or the pytorch documentation, and I'll ignore the geeksforgeeks and other seo spam that finds it's way it. It's almost like google is a statistical "portal" page, like Yahoo or one of those old cluttered sites were, that lets me quickly get through the menus by just searching. That's different from a blank slate search like we might have done 25 years ago. I think what's really lacking now is uncorrupted search for anything that can be monetized. Like I tried to search for a frying pan once on google and it was unusable. I'm not sure any better search engine can fix that, that's why everyone appends "reddit" to queries they are looking for a real opinion on, again, because they are optimizing for the current state of the web. Anyway, all that to say I think there are a lot of problems with (google dominated) search, but they are basically reflected in the current web overall, so just a better search engine, outside of stripping out the ads, can only do so much. Real improved search efforts need to somehow change the content that's out there at the same time as they improve they experience, and let us know how to, in a simple way, get the most out of it. I think google has a much deeper moat than most people realize

danuker 5y ago
> because they are optimizing for the current state of the web. I believe people will at least start looking for alternatives. For example, I have been collecting search engines, and whenever I encounter a page with too many commercial-laden SEO-porked results, I use a different search engine in Firefox. I have enabled the Search Bar, I can do Alt+D, Tab, Tab, enter my query, then click a different search engine, which searches instantly, unlike the main bar, where you have to press Enter once more after clicking. I just added this one also. See my collection: https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH97UUJ6h5+AWSeXxS https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH9...
- coverband 5y ago
  Could you paste again as text links? Thanks.
  - danuker 5y ago
    Firefox uses a custom format for compressing the SEARCH ENGINES. I am appalled. I had to do a `pip install lz4` then apply and run this change: https://gist.github.com/Tblue/62ff47bef7f894e92ed5?permalink_comment_id=2315586#gistcomment-2315586 https://gist.github.com/Tblue/62ff47bef7f894e92ed5?permalink... And it did not work. I hit a brick wall. I completely lost trust in Firefox. I want a browser created by a non-profit. Thank you Google for corrupting everything you touch. I then found this blessed soul: https://www.jeffersonscher.com/ffu/searchjson.html https://www.jeffersonscher.com/ffu/searchjson.html And here are the ones not built-in: https://metager.org/meta/meta.ger3?eingabe={searchTerms} https://metager.org/meta/meta.ger3?eingabe={searchTerms} https://wiki.archlinux.org/index.php?title=Special:Search&search={searchTerms} https://wiki.archlinux.org/index.php?title=Special:Search&se... https://www.mojeek.com/search?q={searchTerms} https://www.mojeek.com/search?q={searchTerms} https://www.openstreetmap.org/search https://www.openstreetmap.org/search https://pypi.org/search/?q={searchTerms} https://pypi.org/search/?q={searchTerms} https://www.qwant.com/?q={searchTerms}&client=opensearch https://www.qwant.com/?q={searchTerms}&client=opensearch https://searx.xyz/ https://searx.xyz/ https://swisscows.com/opensearch.xml https://swisscows.com/opensearch.xml https://yandex.com/search/?text={searchTerms}&from=os&clid=1836587 https://yandex.com/search/?text={searchTerms}&from=os&clid=1... https://whoogle.sdf.org/search https://whoogle.sdf.org/search https://lazyweb.ai/nav/ https://lazyweb.ai/nav/ https://en.wiktionary.org/w/api.php?action=opensearch&format=xml&search={searchTerms}&namespace=0 https://en.wiktionary.org/w/api.php?action=opensearch&format... https://github.com/search?q={searchTerms}&ref=opensearch https://github.com/search?q={searchTerms}&ref=opensearch http://www.urbandictionary.com/define.php?term={searchTerms} http://www.urbandictionary.com/define.php?term={searchTerms} http://alternativeto.net/browse/search/?q={searchTerms} http://alternativeto.net/browse/search/?q={searchTerms} https://search.marginalia.nu/search?query={searchTerms}&ref=opensearch https://search.marginalia.nu/search?query={searchTerms}&ref=... https://kagi.com/search?q={searchTerms} https://kagi.com/search?q={searchTerms} https://you.com/search?q={searchTerms} https://you.com/search?q={searchTerms} https://millionshort.com/search?keywords={searchTerms}&remove=0&shopping=&advertising=&chat=&country= https://millionshort.com/search?keywords={searchTerms}&remov... https://search.f-droid.org/?q={searchTerms}&opensearch=1 https://search.f-droid.org/?q={searchTerms}&opensearch=1 https://www.gnod.com/search/ https://www.gnod.com/search/ http://teclis.com:1333/?topics={searchTerms} http://teclis.com:1333/?topics={searchTerms} https://wiby.me/?q={searchTerms} https://wiby.me/?q={searchTerms} https://www.alexandria.org/?q={searchTerms} https://www.alexandria.org/?q={searchTerms}
outcoldman 5y ago
> I've realized my searching is basically optimized for google Is it just me, or I feel like Google does not provide anymore good results for me. Like every time I search something completely out of my knowledge, like "How to purchase a property in Mexico", it will give me 100+ results of some results with autogenerated content like "10 best places to buy property in Mexico". And the only way to fix that would be to add something like `site:reddit.com`
- daptaq 5y ago
  Might be an instance of Goodhart's law: https://en.m.wikipedia.org/wiki/Goodhart's_law https://en.m.wikipedia.org/wiki/Goodhart's_law If all websites try to optimise for SEO, they undermine the assumption that the evaluation of a search engine is the pure consequence of how well a site satisfies a query.
  - hinkley 5y ago
    I really think than one, we are going to have to end up with search engines managing a curated list of 'roots', and two, those roots are going to end up consisting substantially of a mix of more 'human' sites and, let's be honest with ourselves, a certain amount of content that is paying for favoritism. I think it's very possible that we have effectively raised the noise floor so high that there is no signal, but also likely that perverse incentives from trying to profit off of search engines have made them our enemies instead of our friends. For instance, does Google favor sites that run their own tools on them? I've stopped paying attention but recall hearing mutterings to that effect. If so then running the tools is a protection racket. For other perverse incentives: if you try to rank sites by how long someone stays on them before backing out, or searching again, then you end up favoring rabbit-hole sites, that either string you along or suck you into a tangent. "Oh, this must have answered their question about keeping bees," no, they're reading gossip about the Queen of England and have forgotten all about beekeeping.
    - IAmEveryone 5y ago
      Oh, look! It’s the same „flaw“ people have been explaining each other for a decade. We’ve all seen it dozens of times, yet Google apparently remains in the dark.
vishnugupta 5y ago
> I'm curious how people evaluate them quickly. Speaking about myself; I cold turkey migrated to DDG ~2 months ago. So far I've had to resort to Google search 10 times or so. One thing I miss though is Google's nice visualisation of fast changing results e.g., match scores. For example: https://imgur.com/a/Q5nZkjo https://imgur.com/a/Q5nZkjo
- Seirdy 5y ago
  DDG's organic link results are from Bing, sans personalization. DuckDuckGo advertises using "over 400 sources", which means that at least 399 sources only power infoboxes ("instant answers") and non-generalist search, such as the Video search.
karmab 5y ago
I sEeU
ffhhj 5y ago
> There have been a few search engines out recently I'd like to try them out, could you mention which?
- version_five 5y ago
  you.com and kagi.com off the top of my head
  - Seirdy 5y ago
    I've been keeping my eye on You.com, tracking a few SERPs over time compared to other Bing- and Google-based engines. So far, the results don't seem independent. Try comparing results with a Bing-based engine (e.g. DuckDuckGo) or a Google-based one (e.g. StartPage, GMX) to see if they differ. (Don't use Google or Bing directly, since results will be personalized based on factors like location, device, your fingerprint, etc.).
- lolinder 5y ago
  I've been using the Kagi beta for a few months now, and it's awesome: https://kagi.com/ https://kagi.com/ The biggest thing I've found is that when doing technical searches it always turns up the sources I'm actually looking for, actively filtering all the GitHub/StackOverflow copycat sites. It also seems to up-weight official docs compared to Google. For example, "how to read a json file in python" turns up the Python docs as the second result, where in Google they're nowhere to be found.
- Seirdy 5y ago
  I listed a bunch over at https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe..., and I'm always adding more. I first discovered Alexandria in early February: https://git.sr.ht/~seirdy/seirdy.one/commit/935b55f10f9024ee660d5b0f6c9f7e546db159d8 https://git.sr.ht/~seirdy/seirdy.one/commit/935b55f10f9024ee... Around the same time, I also discovered sengine.info, Artado, Entfer, and Siik. By sheer coincidence they all were mentioned to me or decided to crawl my site within the same couple weeks. So yes, from my perspective there have been more than a few new smaller engines getting active on the heels of bigger names like Neeva, Kagi, Brave Search, etc.
chris123 5y ago
I'd love to be able to add a tag to a search to have it exclude sites with any kind of monetization, I know that's not realistic cuz that's where Google makes most of its money (or do they make most of their money somewhere other than advertising these days?). Anyways, yeah, I'm sick of SEO optimized, click optimized, advertising optimized, affiliate link optimized crap.
- vivekraghunatha 5y ago
  (Founder of Neeva here) -- Chris -- I love the idea. Neeva does this in a contextually relevant manner to the intent of the query. For example, on health queries, we label all health sites as "trusted", "ad-supported" etc. and allow you to filter down to the appropriate subset of results. For programming queries, we label sites as "official sites", "forums", "blogs", "code repos", "programming websites" (the SEO-ed ones). We work with human raters to do this. Would love to hear if you find it useful and what other labels would be of use.
zeta0134 5y ago
I've been wondering for a while now about building a search engine for the ad free web. That is, penalize or outright refuse to index any recognized advertising network, letting through only those sites which don't perform invasive tracking with third party services. Mostly as a curiosity: what would be left? What would rise to the top when you filter all of that out?
- aghilmort 5y ago
  exploring adding that search / similar as breeze filter at https://breezethat.com/- https://breezethat.com/- not sure can do with curation + bing or google, may have to wait until we bring our scraper out of testing
- version_five 5y ago
  I've thought about something similar, basically "the good internet" that would be a hand curated list of sites that are not there just as a pretext for ads. I think a lot of software project documentation qualifies, lots of stuff on university sites like lectures notes for example. I assume that across different niches there is other stuff like this. I think the key would be something that can't be gamed, like it has to be legitimate content that is online for an existing purpose and not as a pretext.
- not2b 5y ago
  Wikipedia.
  - not2b 5y ago
    Evidently someone disagrees, but no ads or trackers except on the home page and its pages rank highly on current search engines, so if you exclude trackers and ads that's what you're going to get.
- randomsilence 5y ago
  Check the 'Small or non-commercial Web' search engines on this overview page: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe...
amelius 5y ago
> I'm curious how people evaluate them quickly. Are there search benchmarks to be found somewhere? There must be. If you want to write a search engine, you need a way to validate the results.
- marginalia_nu 5y ago
  There are benchmarks within the adjacent field of information retrieval, but in general it's hard to properly validate a search engine because real data is so noisy and misbehaved, and sample data is so different from real data.
  - kreeben 5y ago
    Sure, the problem of information retrieval is not exactly that of web search but they're pretty close. So, from such a knowledgeable person such as yourself, when it comes to this topic, could you remind us, what are some of those benchmarks?
    - marginalia_nu 5y ago
      https://en.m.wikipedia.org/wiki/Precision_and_recall https://en.m.wikipedia.org/wiki/Precision_and_recall For some standard corpus.
      - kreeben 5y ago
        >> Precision and recall The phrase I was looking for. Thx a bunch! Gonna marginalia that now.
        marginalia_nu 5y ago
        Haha, ironically it lacks both precision and recall for that topic.
- jll29 5y ago
  The "Web track" task at the annual US NIST TREC conference ( https://trec.nist.gov/ https://trec.nist.gov/ ) is an open innovation benchmark that everyone can contribute; participants get a set of queries that they have to run on exactly the same corpus. Then they return the top-k results to a team that evaluates them. Here is an example (2014) Web track paper of the 23th TREC: https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf (TREC has a plentitude of difference benchmark tasks and you can submit your own: https://trec.nist.gov/pubs/trec29/trec2020.html https://trec.nist.gov/pubs/trec29/trec2020.html - recent TREC 2020 papers)
demopathos 5y ago
I'm surprised you are not a fan of geeksforgeeks. While each of their webpages have substantially less content than the pytorch docs or SO result, I find that they get to the point instantly. My mean time to solution from G4G is definitely smaller than SO.
- RosanaAnaDana 5y ago
  Geeksforgeeks is toxic garbage poisoning the well of good solutions for common problems.
- _bohm 5y ago
  I've found that their content is often inaccurate or written by people who come across as novices. I actually emailed them to correct an inaccuracy in one of their articles once, which they did, so kudos to them for that.
- version_five 5y ago
  I guess everyone has their go-to sites and their pet peeves. Geeksforgeeks may be less spammy than some, but I still think of it as that annoying site that got in the way of either the SO or documentation answer that I was looking for. Just to expand, if I want the api reference, say I search for defaultdict (for some reason I like using them but always have to look at the reference), I want the python documentation. I definitely don't want a third party telling me about it. And if I search a "make list of tensor into tensor" type question, I want SO where someone had asked the same question and got "tensor.stack" as the reply, so I can understand the answer and follow up by looking at the tensor.stack pytorch reference it I want. Anything else is wasting my time, I think most users with similarly specific queries are not looking for tutorials, they are looking for the names of functions they hypothesize exist, or references. That's why intermediary sites that try to give an explanation are annoying, at least for me.
- Seirdy 5y ago
  I generally find that sites like SO, GFG, etc. often play the role of "Reading the Docs as a Service". I prefer using them only after official documentation or specifications fail me. When I want an opinionated answer, I just ping some people I already know or check the personal websites of the developers of the language/tool I'm using. If I have further questions, I check the relevant IRC channel. Sites like SO are a "last resort" for me. In other words, I'd rather see these at the bottom of the SERP than the top, but I wouldn't want to completely eliminate them.
Seirdy 5y ago
> I'm curious how people evaluate them quickly. To paint with a broad brush, I look at three criteria: 1. Infoboxes ("instant answers") should focus on site previews rather than trying to intelligently answer my question. Most DuckDuckGo infoboxes are good examples of this; Bing and Google ones are too "clever". 2. Organic results should be unique; most engines are powered by a commercial Bing API or use Google Custom Search. Compare results with a Bing or Google proxy (duckduckgo, startpage, etc) to avoid personalized results. Monitor queries over time to see if SERPs change in ways that diverge from Google/Bing/Yandex. 3. "other" stuff. Common features I find appealing include area-specific search (Kagi has a "non-commercial lens" mostly powered by its Teclis index; Brave is rolling out "goggles"), displaying additional info about each result (Marginalia and Kagi highlight results with heavy JS or tracking), user-driven SERP personalization (Neeva and Kagi allow promoting/demoting domains), etc. And always check privacy policies, TOS, GDPR/CCPA compliance, etc. > Google is now almost a convenience. If I have a coding question, I search for "turn list of tensors into tensor" or whatever but I'm really looking for SO or the pytorch documentation, and I'll ignore the geeksforgeeks and other seo spam that finds it's way it. It's almost like google is a statistical "portal" page, I like engines like Neeva and Kagi that allow customizing SERPs by demoting irrelevant results; I demote crap like GFG, w3schools, tutorialspoint, dev(.)to, etc. and promote official documentation. Alternatively, you can use an adblocker to block results matching a pattern: https://reddit.com/hgqi5o https://reddit.com/hgqi5o
8n4vidtmkvmk 5y ago
maybe the solution is to make google itself reddit style. let users downvote the seo spam websites and allow them to be downranked. sure it opens the door for a different kind of abuse... but maybe that problem is more fixable?
- seltzered_ 5y ago
  I feel like this was tried a decade ago: https://developers.google.com/search/blog/2011/03/introducing-1-button https://developers.google.com/search/blog/2011/03/introducin... ("Introducing the +1 button", Google, 2011)

danuker 5y ago

> because they are optimizing for the current state of the web. I believe people will at least start looking for alternatives. For example, I have been collecting search engines, and whenever I encounter a page with too many commercial-laden SEO-porked results, I use a different search engine in Firefox. I have enabled the Search Bar, I can do Alt+D, Tab, Tab, enter my query, then click a different search engine, which searches instantly, unlike the main bar, where you have to press Enter once more after clicking. I just added this one also. See my collection: https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH97UUJ6h5+AWSeXxS https://0bin.net/paste/ZSCRYVx1#sxD+jBIpScJismXBYwaoJPh75TH9...

coverband 5y ago
Could you paste again as text links? Thanks.
- danuker 5y ago
  Firefox uses a custom format for compressing the SEARCH ENGINES. I am appalled. I had to do a `pip install lz4` then apply and run this change: https://gist.github.com/Tblue/62ff47bef7f894e92ed5?permalink_comment_id=2315586#gistcomment-2315586 https://gist.github.com/Tblue/62ff47bef7f894e92ed5?permalink... And it did not work. I hit a brick wall. I completely lost trust in Firefox. I want a browser created by a non-profit. Thank you Google for corrupting everything you touch. I then found this blessed soul: https://www.jeffersonscher.com/ffu/searchjson.html https://www.jeffersonscher.com/ffu/searchjson.html And here are the ones not built-in: https://metager.org/meta/meta.ger3?eingabe={searchTerms} https://metager.org/meta/meta.ger3?eingabe={searchTerms} https://wiki.archlinux.org/index.php?title=Special:Search&search={searchTerms} https://wiki.archlinux.org/index.php?title=Special:Search&se... https://www.mojeek.com/search?q={searchTerms} https://www.mojeek.com/search?q={searchTerms} https://www.openstreetmap.org/search https://www.openstreetmap.org/search https://pypi.org/search/?q={searchTerms} https://pypi.org/search/?q={searchTerms} https://www.qwant.com/?q={searchTerms}&client=opensearch https://www.qwant.com/?q={searchTerms}&client=opensearch https://searx.xyz/ https://searx.xyz/ https://swisscows.com/opensearch.xml https://swisscows.com/opensearch.xml https://yandex.com/search/?text={searchTerms}&from=os&clid=1836587 https://yandex.com/search/?text={searchTerms}&from=os&clid=1... https://whoogle.sdf.org/search https://whoogle.sdf.org/search https://lazyweb.ai/nav/ https://lazyweb.ai/nav/ https://en.wiktionary.org/w/api.php?action=opensearch&format=xml&search={searchTerms}&namespace=0 https://en.wiktionary.org/w/api.php?action=opensearch&format... https://github.com/search?q={searchTerms}&ref=opensearch https://github.com/search?q={searchTerms}&ref=opensearch http://www.urbandictionary.com/define.php?term={searchTerms} http://www.urbandictionary.com/define.php?term={searchTerms} http://alternativeto.net/browse/search/?q={searchTerms} http://alternativeto.net/browse/search/?q={searchTerms} https://search.marginalia.nu/search?query={searchTerms}&ref=opensearch https://search.marginalia.nu/search?query={searchTerms}&ref=... https://kagi.com/search?q={searchTerms} https://kagi.com/search?q={searchTerms} https://you.com/search?q={searchTerms} https://you.com/search?q={searchTerms} https://millionshort.com/search?keywords={searchTerms}&remove=0&shopping=&advertising=&chat=&country= https://millionshort.com/search?keywords={searchTerms}&remov... https://search.f-droid.org/?q={searchTerms}&opensearch=1 https://search.f-droid.org/?q={searchTerms}&opensearch=1 https://www.gnod.com/search/ https://www.gnod.com/search/ http://teclis.com:1333/?topics={searchTerms} http://teclis.com:1333/?topics={searchTerms} https://wiby.me/?q={searchTerms} https://wiby.me/?q={searchTerms} https://www.alexandria.org/?q={searchTerms} https://www.alexandria.org/?q={searchTerms}

outcoldman 5y ago

> I've realized my searching is basically optimized for google Is it just me, or I feel like Google does not provide anymore good results for me. Like every time I search something completely out of my knowledge, like "How to purchase a property in Mexico", it will give me 100+ results of some results with autogenerated content like "10 best places to buy property in Mexico". And the only way to fix that would be to add something like `site:reddit.com`

daptaq 5y ago
Might be an instance of Goodhart's law: https://en.m.wikipedia.org/wiki/Goodhart's_law https://en.m.wikipedia.org/wiki/Goodhart's_law If all websites try to optimise for SEO, they undermine the assumption that the evaluation of a search engine is the pure consequence of how well a site satisfies a query.
- hinkley 5y ago
  I really think than one, we are going to have to end up with search engines managing a curated list of 'roots', and two, those roots are going to end up consisting substantially of a mix of more 'human' sites and, let's be honest with ourselves, a certain amount of content that is paying for favoritism. I think it's very possible that we have effectively raised the noise floor so high that there is no signal, but also likely that perverse incentives from trying to profit off of search engines have made them our enemies instead of our friends. For instance, does Google favor sites that run their own tools on them? I've stopped paying attention but recall hearing mutterings to that effect. If so then running the tools is a protection racket. For other perverse incentives: if you try to rank sites by how long someone stays on them before backing out, or searching again, then you end up favoring rabbit-hole sites, that either string you along or suck you into a tangent. "Oh, this must have answered their question about keeping bees," no, they're reading gossip about the Queen of England and have forgotten all about beekeeping.
  - IAmEveryone 5y ago
    Oh, look! It’s the same „flaw“ people have been explaining each other for a decade. We’ve all seen it dozens of times, yet Google apparently remains in the dark.

vishnugupta 5y ago

> I'm curious how people evaluate them quickly. Speaking about myself; I cold turkey migrated to DDG ~2 months ago. So far I've had to resort to Google search 10 times or so. One thing I miss though is Google's nice visualisation of fast changing results e.g., match scores. For example: https://imgur.com/a/Q5nZkjo https://imgur.com/a/Q5nZkjo

Seirdy 5y ago
DDG's organic link results are from Bing, sans personalization. DuckDuckGo advertises using "over 400 sources", which means that at least 399 sources only power infoboxes ("instant answers") and non-generalist search, such as the Video search.

karmab 5y ago

I sEeU

ffhhj 5y ago

> There have been a few search engines out recently I'd like to try them out, could you mention which?

version_five 5y ago
you.com and kagi.com off the top of my head
- Seirdy 5y ago
  I've been keeping my eye on You.com, tracking a few SERPs over time compared to other Bing- and Google-based engines. So far, the results don't seem independent. Try comparing results with a Bing-based engine (e.g. DuckDuckGo) or a Google-based one (e.g. StartPage, GMX) to see if they differ. (Don't use Google or Bing directly, since results will be personalized based on factors like location, device, your fingerprint, etc.).
lolinder 5y ago
I've been using the Kagi beta for a few months now, and it's awesome: https://kagi.com/ https://kagi.com/ The biggest thing I've found is that when doing technical searches it always turns up the sources I'm actually looking for, actively filtering all the GitHub/StackOverflow copycat sites. It also seems to up-weight official docs compared to Google. For example, "how to read a json file in python" turns up the Python docs as the second result, where in Google they're nowhere to be found.
Seirdy 5y ago
I listed a bunch over at https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe..., and I'm always adding more. I first discovered Alexandria in early February: https://git.sr.ht/~seirdy/seirdy.one/commit/935b55f10f9024ee660d5b0f6c9f7e546db159d8 https://git.sr.ht/~seirdy/seirdy.one/commit/935b55f10f9024ee... Around the same time, I also discovered sengine.info, Artado, Entfer, and Siik. By sheer coincidence they all were mentioned to me or decided to crawl my site within the same couple weeks. So yes, from my perspective there have been more than a few new smaller engines getting active on the heels of bigger names like Neeva, Kagi, Brave Search, etc.

chris123 5y ago

I'd love to be able to add a tag to a search to have it exclude sites with any kind of monetization, I know that's not realistic cuz that's where Google makes most of its money (or do they make most of their money somewhere other than advertising these days?). Anyways, yeah, I'm sick of SEO optimized, click optimized, advertising optimized, affiliate link optimized crap.

vivekraghunatha 5y ago
(Founder of Neeva here) -- Chris -- I love the idea. Neeva does this in a contextually relevant manner to the intent of the query. For example, on health queries, we label all health sites as "trusted", "ad-supported" etc. and allow you to filter down to the appropriate subset of results. For programming queries, we label sites as "official sites", "forums", "blogs", "code repos", "programming websites" (the SEO-ed ones). We work with human raters to do this. Would love to hear if you find it useful and what other labels would be of use.

zeta0134 5y ago

I've been wondering for a while now about building a search engine for the ad free web. That is, penalize or outright refuse to index any recognized advertising network, letting through only those sites which don't perform invasive tracking with third party services. Mostly as a curiosity: what would be left? What would rise to the top when you filter all of that out?

aghilmort 5y ago
exploring adding that search / similar as breeze filter at https://breezethat.com/- https://breezethat.com/- not sure can do with curation + bing or google, may have to wait until we bring our scraper out of testing
version_five 5y ago
I've thought about something similar, basically "the good internet" that would be a hand curated list of sites that are not there just as a pretext for ads. I think a lot of software project documentation qualifies, lots of stuff on university sites like lectures notes for example. I assume that across different niches there is other stuff like this. I think the key would be something that can't be gamed, like it has to be legitimate content that is online for an existing purpose and not as a pretext.
not2b 5y ago
Wikipedia.
- not2b 5y ago
  Evidently someone disagrees, but no ads or trackers except on the home page and its pages rank highly on current search engines, so if you exclude trackers and ads that's what you're going to get.
randomsilence 5y ago
Check the 'Small or non-commercial Web' search engines on this overview page: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

amelius 5y ago

> I'm curious how people evaluate them quickly. Are there search benchmarks to be found somewhere? There must be. If you want to write a search engine, you need a way to validate the results.

marginalia_nu 5y ago
There are benchmarks within the adjacent field of information retrieval, but in general it's hard to properly validate a search engine because real data is so noisy and misbehaved, and sample data is so different from real data.
- kreeben 5y ago
  Sure, the problem of information retrieval is not exactly that of web search but they're pretty close. So, from such a knowledgeable person such as yourself, when it comes to this topic, could you remind us, what are some of those benchmarks?
  - marginalia_nu 5y ago
    https://en.m.wikipedia.org/wiki/Precision_and_recall https://en.m.wikipedia.org/wiki/Precision_and_recall For some standard corpus.
    - kreeben 5y ago
      >> Precision and recall The phrase I was looking for. Thx a bunch! Gonna marginalia that now.
      - marginalia_nu 5y ago
        Haha, ironically it lacks both precision and recall for that topic.
jll29 5y ago
The "Web track" task at the annual US NIST TREC conference ( https://trec.nist.gov/ https://trec.nist.gov/ ) is an open innovation benchmark that everyone can contribute; participants get a set of queries that they have to run on exactly the same corpus. Then they return the top-k results to a team that evaluates them. Here is an example (2014) Web track paper of the 23th TREC: https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf (TREC has a plentitude of difference benchmark tasks and you can submit your own: https://trec.nist.gov/pubs/trec29/trec2020.html https://trec.nist.gov/pubs/trec29/trec2020.html - recent TREC 2020 papers)

demopathos 5y ago

I'm surprised you are not a fan of geeksforgeeks. While each of their webpages have substantially less content than the pytorch docs or SO result, I find that they get to the point instantly. My mean time to solution from G4G is definitely smaller than SO.

RosanaAnaDana 5y ago
Geeksforgeeks is toxic garbage poisoning the well of good solutions for common problems.
_bohm 5y ago
I've found that their content is often inaccurate or written by people who come across as novices. I actually emailed them to correct an inaccuracy in one of their articles once, which they did, so kudos to them for that.
version_five 5y ago
I guess everyone has their go-to sites and their pet peeves. Geeksforgeeks may be less spammy than some, but I still think of it as that annoying site that got in the way of either the SO or documentation answer that I was looking for. Just to expand, if I want the api reference, say I search for defaultdict (for some reason I like using them but always have to look at the reference), I want the python documentation. I definitely don't want a third party telling me about it. And if I search a "make list of tensor into tensor" type question, I want SO where someone had asked the same question and got "tensor.stack" as the reply, so I can understand the answer and follow up by looking at the tensor.stack pytorch reference it I want. Anything else is wasting my time, I think most users with similarly specific queries are not looking for tutorials, they are looking for the names of functions they hypothesize exist, or references. That's why intermediary sites that try to give an explanation are annoying, at least for me.
Seirdy 5y ago
I generally find that sites like SO, GFG, etc. often play the role of "Reading the Docs as a Service". I prefer using them only after official documentation or specifications fail me. When I want an opinionated answer, I just ping some people I already know or check the personal websites of the developers of the language/tool I'm using. If I have further questions, I check the relevant IRC channel. Sites like SO are a "last resort" for me. In other words, I'd rather see these at the bottom of the SERP than the top, but I wouldn't want to completely eliminate them.

Seirdy 5y ago

> I'm curious how people evaluate them quickly. To paint with a broad brush, I look at three criteria: 1. Infoboxes ("instant answers") should focus on site previews rather than trying to intelligently answer my question. Most DuckDuckGo infoboxes are good examples of this; Bing and Google ones are too "clever". 2. Organic results should be unique; most engines are powered by a commercial Bing API or use Google Custom Search. Compare results with a Bing or Google proxy (duckduckgo, startpage, etc) to avoid personalized results. Monitor queries over time to see if SERPs change in ways that diverge from Google/Bing/Yandex. 3. "other" stuff. Common features I find appealing include area-specific search (Kagi has a "non-commercial lens" mostly powered by its Teclis index; Brave is rolling out "goggles"), displaying additional info about each result (Marginalia and Kagi highlight results with heavy JS or tracking), user-driven SERP personalization (Neeva and Kagi allow promoting/demoting domains), etc. And always check privacy policies, TOS, GDPR/CCPA compliance, etc. > Google is now almost a convenience. If I have a coding question, I search for "turn list of tensors into tensor" or whatever but I'm really looking for SO or the pytorch documentation, and I'll ignore the geeksforgeeks and other seo spam that finds it's way it. It's almost like google is a statistical "portal" page, I like engines like Neeva and Kagi that allow customizing SERPs by demoting irrelevant results; I demote crap like GFG, w3schools, tutorialspoint, dev(.)to, etc. and promote official documentation. Alternatively, you can use an adblocker to block results matching a pattern: https://reddit.com/hgqi5o https://reddit.com/hgqi5o

8n4vidtmkvmk 5y ago

maybe the solution is to make google itself reddit style. let users downvote the seo spam websites and allow them to be downranked. sure it opens the door for a different kind of abuse... but maybe that problem is more fixable?

seltzered_ 5y ago
I feel like this was tried a decade ago: https://developers.google.com/search/blog/2011/03/introducing-1-button https://developers.google.com/search/blog/2011/03/introducin... ("Introducing the +1 button", Google, 2011)