10 ms·

A look at search engines with their own indexes (2021)

271 points by tintedfireglass 4y ago

deleted 4y ago
[deleted]
mavhc 4y ago
Why aren't more of them using Common Crawl? Is it flawed?
- snowwrestler 4y ago
  I found it weird that the author references the Common Crawl a couple of times but it does not have its own entry on the page. I’m not familiar with what it is… guess I’ll just Google it.
- marginalia_nu 4y ago
  Can only speak for my self, but I use my own dinky crawler because CC doesn't really solve any problems I have operating my search engine, and its unwieldy size creates new problems I didn't have doing my own crawling. Crawling just isn't the hard part of building a search engine. Sure there are pitfalls and obstacles, but they're all fairly solvable.
aberoham 4y ago
No DDG?
- therealdrag0 4y ago
  It’s there under bing
- Kiro 4y ago
  They don't have their own index so they are listed under Bing.
mg 4y ago
Yay, another person who keeps an eye on the landscape of search engines and tries different ones. I thought I am the only one with this hobby. I made this page to be able to use a different search engine for every search I do: https://www.gnod.com/search https://www.gnod.com/search (Click on "more engines" to see the full list and chose the ones for which buttons are displayed. But you can also search right from the list by first entering a search query and then clicking on an entry in the list.)
- dhc02 4y ago
  Suggestions for additions: Kagi.com Neeva.com
  - mg 4y ago
    Do you use these? I tried Kagi and liked the design. Having to log into a search engine to use it is a no-go to me, though. But if there is "demand" for it, I can surely add it. So far, I think I am the only user of Gnod Search. Neeva does not let me search, no matter what: "We'll get in touch as soon as Neeva is available in your region". Maybe they don't want to deal with users in Europe? I think for a list of search engines to try, it is best to stick with the open ones. Otherwise, the experience will become cumbersome if every other engine you want to try tells you that you cannot use it for some reason.
    - tbassetto 4y ago
      I switched to Kagi recently (and became a paid user!). Somehow DDG never really clicked for me, I find results on Kagi better.
    - memorable 4y ago
      You can acually use Neeva without an account by setting your browser's search query to 'neeva.com/search?q=%s'. I use Neeva without an account and not in the US and it works just fine.
    - Semaphor 4y ago
      I’m a former DDG user (maybe for 2 years?) who switched to Kagi in December (and have paid for my first month already). I preferred DDG over Google (for most searches, Google was better at rare searches; in addition, DDG got worse over time), but prefer Kagi over both. I almost always get what I need, and the personalization features (specifically ranking sites higher/lower linked to your account) are great. Really need to find the time to play around with lenses, but I have heard good things ;)
slimsag 4y ago
I think 'general purpose' search engines are often doomed to fail, and this extends to 'web search engines' (which, in reality, do not search the full web and are NOT general purpose.) This list makes a great list of attempts at general purpose search engines (I would include DDG and Bing) which ultimately fail and are not what the masses want. There's also precedence for how only domain-specific search engines are valuable / make sense these days: Google itself is shifting to ML-based answers (attempting to build the holy-grail of general-purpose search engines), and whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet. Examples of real-world domain-specific search engines today: * Image, YouTube, Maps search * Amazon search * Code search (Sourcegraph, cs.github.com, cs.opensource.google, etc.) * Facebook, Twitter, Reddit, TikTok, LinkedIn search * Documents search (iCloud, Drive, etc.) * Messenger search (Discord, Slack, Messenger, etc.) It's really telling how Google has largely only achieved general-purpose search in their domains of data (a Google search turns up YouTube videos, Maps locations) but isn't even complete in that area (Drive, images, etc. don't show up in Google results) General-purpose search is simply non-viable these days due to (a) data silos and (b) you need to have domain specific search to provide an edge (e.g. regex for code search, or drag-n-drop an image for image similarity search)
- PestoDiRucola 4y ago
  The issue is that some of these domain-specific search engines kinda suck on their own, for example reddit search is notoriously bad. This is often overcome by searching what you want in a general-purpose search engine and appending "reddit" to your search query.
- exmadscientist 4y ago
  So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish. It is shocking how many searches I do which are well described by "show me every single thing on the entire Internet which matches this string". (I'd love to match a simple regex, since part numbers have stupid trivial variations, but I'll take what's actually feasible.) Or, I often want to match only PDFs, but that hurdle seems well solved these days. There aren't that many results for a few of these things (and those are often the ones I need information about the most), so I sometimes really do want to see every result. I don't see any other way to do that than a full-text, full-web index. I just don't. And my job gets a lot harder if that class of search isn't possible anymore. So I'm desperately hoping these indices remain at least somewhat viable. And yes, I am noticing that Google is lovingly correcting my long part numbers that do actually match a fair bit of results to crap that is useless. Fuck that bullshit, Google.
ddtaylor 4y ago
I'm glad others are interested in this topic and there is ongoing research. The current state of search engines is abysmal. My "plan" was to support DuckDuckGo as the alternative to Google, since they declined in quality so rapidly in the last few years with most results just pushing their own services or funneling you into product ads. But, DDG has been making a lot of misteps recently as well. Despite all the chest pounding they do about privacy, they still load tracking scripts on their domain and I feel their communication about it has been really poor and gives of scummy vibes.
- tintedfireglass 4y ago
  DDG is completely dependent on Microsoft for both the search results and their ad revenue. This causes a lot of problems as whenever Microsoft stops supporting DDG,the company is basically dead. In my opinion I think this is why DDG is taking a lot of "missteps" as they don't want to sour their ties with Microsoft. This is just my theory.
  - wiz21c 4y ago
    I was about to report the note 2 of the article: > DuckDuckGo’s help pages claim that the engine uses over 400 sources; my interpretation is that at least 398 sources don’t impact organic results. ouch and I thought DDG was clean... But yet, it doesn't mean that DDG tracks us, does it ?
    - tintedfireglass 4y ago
      Your reply made me read DDG privacy policy. > > Similarly, we may add an affiliate code to some eCommerce sites (e.g. Amazon & eBay) that results in small commissions being paid back to DuckDuckGo when you make purchases at those sites. Why was there a big backlash against brave for doing this but no one bats an eye when DDG does it?
      - smotched 4y ago
        theres a massive ddg bias on hw, see this thread for example https://news.ycombinator.com/item?id=31492505 https://news.ycombinator.com/item?id=31492505
uniqueuid 4y ago
There's one important aspect of search engine evaluations that many people forget. A key part in user satisfaction are re-searches (up to 50%, if I recall correctly from a paper from Teevan et al.) This means that people expect a search engine to find the same results they found in the past for a given query. That's a huge part of lock-in in the ecosystem.
boyter 4y ago
Alas my poor entry into the ring https://bonzamate.com.au https://bonzamate.com.au is missing. It is Australia specific though which might explain that. It is running its own index though, so might be interesting to some.
- cookie_monsta 4y ago
  Sweet as.
- marginalia_nu 4y ago
  Sort of have unfortunate similarities to BonziBuddy with that name.
  - boyter 4y ago
    Never even considered that. Although as an Australian it’s not even close.
  - deleted 4y ago
    [deleted]
  - metadat 4y ago
    I thought it was intentional, as a half joke / nod to the BonzaiBuddy of old.
- Seirdy 4y ago
  Thanks, added. Diff: https://git.sr.ht/~seirdy/seirdy.one/commit/ae3e26e5928a9f0f48eabda959d4012eead0f81a https://git.sr.ht/~seirdy/seirdy.one/commit/ae3e26e5928a9f0f...
hubraumhugo 4y ago
Google won because they started early and had the right algorithm. Back in 1998, Google's PageRank was an innovative algorithm that calculated relevance based on counting backlinks instead of parsing the word counts in embedded HTML text like other search engines. This made Google way better than any other available search engine back then (Lycos, Yahoo, AltaVista, etc.), and within weeks, everyone was switching to Google. Google then had the scale to grow with the internet.
- celdon25 4y ago
  Definitely the right algorithm if you want search results that are driven by marketing teams.
- flatdog 4y ago
  PageRank was and still is mostly marketing, there's no way Google was so much better due to PageRank. It's more likely due to their method of indexing inlinks as pseudo terms within documents that gave them the initial edge.
  - marginalia_nu 4y ago
    It's a one-two punch. PageRank does undeniably help the underspecified case, indexing anchor texts adds more keywords and makes more searches underspecified.
tintedfireglass 4y ago
Inspired by this I have started making a list of alternative search engines and plan to review them once count reaches 100. https://github.com/Tintedfireglass/search-engines https://github.com/Tintedfireglass/search-engines
verisimi 4y ago
I'm using presearch - https://presearch.io/about https://presearch.io/about and I like it.
- tintedfireglass 4y ago
  Presearch is nice but the crypto token is stopping me from using it.
  - verisimi 4y ago
    I don't use the token stuff (I don't even log in). It has decent results, and all the main search engines are available, if needed, on the side.
jfoster 4y ago
For most software, there is an open source attempt aimed at producing something more or less equivalent. Is there any such thing trying to attempt the same level of scalability as Google or Bing? I'm aware of Apache Lucene, Elasticsearch, etc. The thing is, those all seem aimed toward indexing a lot fewer orders of magnitude less stuff than Google does. I'm guessing that perhaps anyone capable & attempting to implement such a thing just automatically gets fast-tracked into a job at Google (or similar company). Is that why it doesn't exist?
- native_samples 4y ago
  How would you even test that? Open source projects typically have a budget of near zero. There are lots of things that the open source approach can't handle but commercial development can, large scale search engines are one.
  - marginalia_nu 4y ago
    I guess hypothetically you could go for a wikimedia-style model and set up a foundation that operates the thing, but that would require a lot of funding, which would require decent utility up front. Vicious circle there. I'm struggling with how to run even a small scale search engine as an open source project. I haven't really found any good projects to model for how to go about an open development process for a system that actually requires decent hardware to run, so right now it's just me developing in public, which is fine I guess, but not much different from how it was before I open sourced it.
    - psychoslave 4y ago
      Wikimedia foundation actually did try to launch such a thing: https://en.wikipedia.org/wiki/Knowledge_Engine_%28Wikimedia_Foundation%29 https://en.wikipedia.org/wiki/Knowledge_Engine_%28Wikimedia_... https://www.newsweek.com/wikipedia-takes-google-new-transparent-search-engine-427028 https://www.newsweek.com/wikipedia-takes-google-new-transpar... Spoiler: it failed.
tintedfireglass 4y ago
Previous discussion post by original author (March 2021): https://news.ycombinator.com/item?id=26429942 https://news.ycombinator.com/item?id=26429942
danskeren 4y ago
Surprised to see my site removed from the list: > Ask.moe uses Google Custom Search now, so it's not a search engine anymore; it's a search client. Not quite sure what a "search client" is, but I don't see why a Bing proxy (using the Bing Search API) is any more of a search engine than a Google proxy (using the Programmable Search Engine).
Linda703 4y ago
[dead]
marginalia_nu 4y ago
I think a big part of why Google has such sticking power is that on the one hand they actually are pretty good at some things (even though they visibly flounder with others), but web browsers are heavily designed around leveraging the particular tasks Google is really good at. This is manifested by turning the URL bar into a search bar by default, making URLs difficult to manually edit (especially on mobile), making bookmarks inaccessible and difficult to manage by requiring multiple clicks to access, and having an interface that makes it easy to accidentally bookmark the wrong websites. I don't believe there's some big conspiracy where Google has orchestrated this, but it's probably more like an effort for every web browser to mimic Chrome without realizing why that has problematic knock-on effects. I really sort of wish there were good alternative web browsers that weren't ostensibly waging war on user agency, or at least gave more than token customization abilities.
- tintedfireglass 4y ago
  Yes,These days most people think that GOOGLE=INTERNET and look on GOOGLE for the nearest hotel or ask GOOGLE today's weather. Google has basically become so big that for most people Google is the manifestation of the internet. This reliance on web search as a gateway to the internet also started with google as in the days of IE we had separate URL bars and search bars, toolbars and bookmark tabs were the norm and there were many different ways people used to obtain their news,talk to friends and in general consume content on the internet. But Google basically turned the browser into a funnel that sends all internet users to https://google.com https://google.com and as chrome has most market-share in both desktop and mobile(thanks to android). Most browsers adopted the SEARCH-ENGINE route in their browsers leading to the position we are in. Today the average person visits a website only in two ways 1.Auto-fill on their URL...sorry search bar 2.Type the name of the website on the google search engine. I have no clue why the feds don't care about big tech monopolies the way Microsoft was attacked back then for IE which is basically child's play compared to what companies(even Microsoft with Edge) are doing today.
  - bil7 4y ago
    > basically turned the browser into a funnel that sends all internet users to https://google.com https://google.com Exactly right and completely intentional. I saw they also pay Apple $15bn to be the default search provider on iOS [0]. Annually. [0] https://www.macrumors.com/2021/08/27/google-could-pay-apple-15-billion-default-search/ https://www.macrumors.com/2021/08/27/google-could-pay-apple-...
kristopolous 4y ago
The realization I made recently is the financial incentive of the page view is the corrupting force that ruins search The content producer is incentivized to make content as quickly as possible in as little effort as possible and cover it with as many ads as possible. On the other end, it's within GBYs interest to be slightly confused; by not nailing it, the user engages with the search using different words, potentially hitting an adword and clicking. GBY could be convincing the user that they're just using it wrong and need to execute more searches. So the financial cost/reward model is configured in a way that gives high margin to spammy trash covered in irritating obstructive ads and search engines that are mysteriously confused by what we ask for. And that's the modern web. Finally we are taught to tremble in fear if we dare consider touching the sacredness of The Market. So here we are stuck with trash technology to search avalanches of garbage.* Luckily there appears to be a few search engines that focus on sites without advertising and trackers and they seem to produce nice results --- * occasionally people like to claim this is just freedom or something so let me clear that field now: being forced to focus on maximizing profit isn't a free society. A free society would permit various missions to be pursued without fear of being squeezed out of existence by those focused exclusively on profit maximization. If priorities are imposed by a p&l tyranny, that's not freedom
- dontcontactme 4y ago
  What are these search engines that focus on sites without advertising and trackers?
  - kristopolous 4y ago
    Try https://teclis.com https://teclis.com and https://search.marginalia.nu https://search.marginalia.nu My test queries are things like "souffle recipe" or "worst celebrity hairstyles" - if it can survive that it can survive anything
  - tintedfireglass 4y ago
    https://wiby.me https://wiby.me is a good one. I actually found this blog on wiby
  - 4y ago
langsoul-com 4y ago
Do Chinese search engines have their own index engines? They're censored, but if you could remove that, would they be as good as Google, Bing, yandex?
- Seirdy 4y ago
  Yes, I listed several. Scroll down a bit.
mysterydip 4y ago
I know a common complaint is search engine x fails at finding info for knowledge domain y, usually something technical that in the overall lexicon is ambiguous with something more generic/popular. I wonder if a good solution to this is to use one of the smaller open engines in a private instance and just submit the handful of sites you know have the relevant info?
ta8903 4y ago
>Two engines were excluded from this list for having a far-right focus. Does this mean like extremist terrorism or regular right wing stuff? Would be interesting to see.
kjrose 4y ago
Does anyone know what the search engines that were excluded for "reasons" are? I like having thorough lists and I am curious what as many as possible are regardless of political silliness or differences in using financial systems. My guess is the crypto one is presearch, but the others I have no idea.
- prophesi 4y ago
  They list out their reasons here https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#exclusions https://seirdy.one/posts/2021/03/10/search-engines-with-own-... Sounds like two were excluded for having a far-right focus, one for crypto payments, and the rest for being small/simple proof-of-concepts That said, I would also be interested in what engines in particular were excluded, with a small blurb as to why it wasn't included.
  - kjrose 4y ago
    Precisely. I don't care the reasons. I would want to know what they are. I absolutely hate when people claim to be providing a "complete" list of something and then skip things because 'reasons'.
LukeEF 4y ago
I bounce from default engine to engine (most recently startpage as a 'better' google) but so far always find myself back with google search. Last time I went elsewhere, I recorded the occasions when I appended the URL bar search with @google. Quick info on a business (address, telephone # etc.) and maps dominated. I haven't used another engine that can deliver similar. I also realized how much time I spend looking at maps and street viewer... The knowledge graph and maps are a massive search moat.
freediver 4y ago
By the looks of it, we are witnessing the "Cambrian explosion" of web search. Search engines will evolve and those that do not adapt might be up for a nasty surprise by the end of Mesozoic Era, no matter how big they are.
kypro 4y ago
As an English language user Yandex is awful for typical use in my experience. I'd say at least a third of the results it serves on its English site are Russian. That said, I find myself using Yandex quite a bit simply because I can assume it has a Russian bias, or at the very least it doesn't have the same biases you might expect from a search engine operated in the West. I think a lot of people would be surprised at the extent at which Google and other Western search engines filter their search results today. A lot of content which might be labelled "extreme" or politically problematic I find is now either missing entirely from Google search or pushed so far down in the results that it's practically inaccessible. On Yandex I find I don't have this problem. As an example during the pandemic Yandex was pretty useful for finding conspiratorial content about Covid-19 which was censored by Google. Not that this is necessarily a good thing for the average user, but it has its uses if you're trying to come to your own conclusions about things rather than relying on the narrative provided by Western media companies. It's obviously also been useful when trying to better understand the Russian side of the ongoing RU/UA conflict which has either been completely absent or very unfairly portrayed by most Western media sources (again, perhaps for good reason, but still).

deleted 4y ago

[deleted]

mavhc 4y ago

Why aren't more of them using Common Crawl? Is it flawed?

snowwrestler 4y ago
I found it weird that the author references the Common Crawl a couple of times but it does not have its own entry on the page. I’m not familiar with what it is… guess I’ll just Google it.
marginalia_nu 4y ago
Can only speak for my self, but I use my own dinky crawler because CC doesn't really solve any problems I have operating my search engine, and its unwieldy size creates new problems I didn't have doing my own crawling. Crawling just isn't the hard part of building a search engine. Sure there are pitfalls and obstacles, but they're all fairly solvable.

aberoham 4y ago

No DDG?

therealdrag0 4y ago
It’s there under bing
Kiro 4y ago
They don't have their own index so they are listed under Bing.

mg 4y ago

Yay, another person who keeps an eye on the landscape of search engines and tries different ones. I thought I am the only one with this hobby. I made this page to be able to use a different search engine for every search I do: https://www.gnod.com/search https://www.gnod.com/search (Click on "more engines" to see the full list and chose the ones for which buttons are displayed. But you can also search right from the list by first entering a search query and then clicking on an entry in the list.)

dhc02 4y ago
Suggestions for additions: Kagi.com Neeva.com
- mg 4y ago
  Do you use these? I tried Kagi and liked the design. Having to log into a search engine to use it is a no-go to me, though. But if there is "demand" for it, I can surely add it. So far, I think I am the only user of Gnod Search. Neeva does not let me search, no matter what: "We'll get in touch as soon as Neeva is available in your region". Maybe they don't want to deal with users in Europe? I think for a list of search engines to try, it is best to stick with the open ones. Otherwise, the experience will become cumbersome if every other engine you want to try tells you that you cannot use it for some reason.
  - tbassetto 4y ago
    I switched to Kagi recently (and became a paid user!). Somehow DDG never really clicked for me, I find results on Kagi better.
  - memorable 4y ago
    You can acually use Neeva without an account by setting your browser's search query to 'neeva.com/search?q=%s'. I use Neeva without an account and not in the US and it works just fine.
  - Semaphor 4y ago
    I’m a former DDG user (maybe for 2 years?) who switched to Kagi in December (and have paid for my first month already). I preferred DDG over Google (for most searches, Google was better at rare searches; in addition, DDG got worse over time), but prefer Kagi over both. I almost always get what I need, and the personalization features (specifically ranking sites higher/lower linked to your account) are great. Really need to find the time to play around with lenses, but I have heard good things ;)

slimsag 4y ago

I think 'general purpose' search engines are often doomed to fail, and this extends to 'web search engines' (which, in reality, do not search the full web and are NOT general purpose.) This list makes a great list of attempts at general purpose search engines (I would include DDG and Bing) which ultimately fail and are not what the masses want. There's also precedence for how only domain-specific search engines are valuable / make sense these days: Google itself is shifting to ML-based answers (attempting to build the holy-grail of general-purpose search engines), and whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet. Examples of real-world domain-specific search engines today: * Image, YouTube, Maps search * Amazon search * Code search (Sourcegraph, cs.github.com, cs.opensource.google, etc.) * Facebook, Twitter, Reddit, TikTok, LinkedIn search * Documents search (iCloud, Drive, etc.) * Messenger search (Discord, Slack, Messenger, etc.) It's really telling how Google has largely only achieved general-purpose search in their domains of data (a Google search turns up YouTube videos, Maps locations) but isn't even complete in that area (Drive, images, etc. don't show up in Google results) General-purpose search is simply non-viable these days due to (a) data silos and (b) you need to have domain specific search to provide an edge (e.g. regex for code search, or drag-n-drop an image for image similarity search)

PestoDiRucola 4y ago
The issue is that some of these domain-specific search engines kinda suck on their own, for example reddit search is notoriously bad. This is often overcome by searching what you want in a general-purpose search engine and appending "reddit" to your search query.
exmadscientist 4y ago
So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish. It is shocking how many searches I do which are well described by "show me every single thing on the entire Internet which matches this string". (I'd love to match a simple regex, since part numbers have stupid trivial variations, but I'll take what's actually feasible.) Or, I often want to match only PDFs, but that hurdle seems well solved these days. There aren't that many results for a few of these things (and those are often the ones I need information about the most), so I sometimes really do want to see every result. I don't see any other way to do that than a full-text, full-web index. I just don't. And my job gets a lot harder if that class of search isn't possible anymore. So I'm desperately hoping these indices remain at least somewhat viable. And yes, I am noticing that Google is lovingly correcting my long part numbers that do actually match a fair bit of results to crap that is useless. Fuck that bullshit, Google.

ddtaylor 4y ago

I'm glad others are interested in this topic and there is ongoing research. The current state of search engines is abysmal. My "plan" was to support DuckDuckGo as the alternative to Google, since they declined in quality so rapidly in the last few years with most results just pushing their own services or funneling you into product ads. But, DDG has been making a lot of misteps recently as well. Despite all the chest pounding they do about privacy, they still load tracking scripts on their domain and I feel their communication about it has been really poor and gives of scummy vibes.

tintedfireglass 4y ago
DDG is completely dependent on Microsoft for both the search results and their ad revenue. This causes a lot of problems as whenever Microsoft stops supporting DDG,the company is basically dead. In my opinion I think this is why DDG is taking a lot of "missteps" as they don't want to sour their ties with Microsoft. This is just my theory.
- wiz21c 4y ago
  I was about to report the note 2 of the article: > DuckDuckGo’s help pages claim that the engine uses over 400 sources; my interpretation is that at least 398 sources don’t impact organic results. ouch and I thought DDG was clean... But yet, it doesn't mean that DDG tracks us, does it ?
  - tintedfireglass 4y ago
    Your reply made me read DDG privacy policy. > > Similarly, we may add an affiliate code to some eCommerce sites (e.g. Amazon & eBay) that results in small commissions being paid back to DuckDuckGo when you make purchases at those sites. Why was there a big backlash against brave for doing this but no one bats an eye when DDG does it?
    - smotched 4y ago
      theres a massive ddg bias on hw, see this thread for example https://news.ycombinator.com/item?id=31492505 https://news.ycombinator.com/item?id=31492505

uniqueuid 4y ago

There's one important aspect of search engine evaluations that many people forget. A key part in user satisfaction are re-searches (up to 50%, if I recall correctly from a paper from Teevan et al.) This means that people expect a search engine to find the same results they found in the past for a given query. That's a huge part of lock-in in the ecosystem.

boyter 4y ago

Alas my poor entry into the ring https://bonzamate.com.au https://bonzamate.com.au is missing. It is Australia specific though which might explain that. It is running its own index though, so might be interesting to some.

cookie_monsta 4y ago
Sweet as.
marginalia_nu 4y ago
Sort of have unfortunate similarities to BonziBuddy with that name.
- boyter 4y ago
  Never even considered that. Although as an Australian it’s not even close.
- deleted 4y ago
  [deleted]
- metadat 4y ago
  I thought it was intentional, as a half joke / nod to the BonzaiBuddy of old.
Seirdy 4y ago
Thanks, added. Diff: https://git.sr.ht/~seirdy/seirdy.one/commit/ae3e26e5928a9f0f48eabda959d4012eead0f81a https://git.sr.ht/~seirdy/seirdy.one/commit/ae3e26e5928a9f0f...

hubraumhugo 4y ago

Google won because they started early and had the right algorithm. Back in 1998, Google's PageRank was an innovative algorithm that calculated relevance based on counting backlinks instead of parsing the word counts in embedded HTML text like other search engines. This made Google way better than any other available search engine back then (Lycos, Yahoo, AltaVista, etc.), and within weeks, everyone was switching to Google. Google then had the scale to grow with the internet.

celdon25 4y ago
Definitely the right algorithm if you want search results that are driven by marketing teams.
flatdog 4y ago
PageRank was and still is mostly marketing, there's no way Google was so much better due to PageRank. It's more likely due to their method of indexing inlinks as pseudo terms within documents that gave them the initial edge.
- marginalia_nu 4y ago
  It's a one-two punch. PageRank does undeniably help the underspecified case, indexing anchor texts adds more keywords and makes more searches underspecified.

tintedfireglass 4y ago

Inspired by this I have started making a list of alternative search engines and plan to review them once count reaches 100. https://github.com/Tintedfireglass/search-engines https://github.com/Tintedfireglass/search-engines

verisimi 4y ago

I'm using presearch - https://presearch.io/about https://presearch.io/about and I like it.

tintedfireglass 4y ago
Presearch is nice but the crypto token is stopping me from using it.
- verisimi 4y ago
  I don't use the token stuff (I don't even log in). It has decent results, and all the main search engines are available, if needed, on the side.

jfoster 4y ago

For most software, there is an open source attempt aimed at producing something more or less equivalent. Is there any such thing trying to attempt the same level of scalability as Google or Bing? I'm aware of Apache Lucene, Elasticsearch, etc. The thing is, those all seem aimed toward indexing a lot fewer orders of magnitude less stuff than Google does. I'm guessing that perhaps anyone capable & attempting to implement such a thing just automatically gets fast-tracked into a job at Google (or similar company). Is that why it doesn't exist?

native_samples 4y ago
How would you even test that? Open source projects typically have a budget of near zero. There are lots of things that the open source approach can't handle but commercial development can, large scale search engines are one.
- marginalia_nu 4y ago
  I guess hypothetically you could go for a wikimedia-style model and set up a foundation that operates the thing, but that would require a lot of funding, which would require decent utility up front. Vicious circle there. I'm struggling with how to run even a small scale search engine as an open source project. I haven't really found any good projects to model for how to go about an open development process for a system that actually requires decent hardware to run, so right now it's just me developing in public, which is fine I guess, but not much different from how it was before I open sourced it.
  - psychoslave 4y ago
    Wikimedia foundation actually did try to launch such a thing: https://en.wikipedia.org/wiki/Knowledge_Engine_%28Wikimedia_Foundation%29 https://en.wikipedia.org/wiki/Knowledge_Engine_%28Wikimedia_... https://www.newsweek.com/wikipedia-takes-google-new-transparent-search-engine-427028 https://www.newsweek.com/wikipedia-takes-google-new-transpar... Spoiler: it failed.

tintedfireglass 4y ago

Previous discussion post by original author (March 2021): https://news.ycombinator.com/item?id=26429942 https://news.ycombinator.com/item?id=26429942

danskeren 4y ago

Surprised to see my site removed from the list: > Ask.moe uses Google Custom Search now, so it's not a search engine anymore; it's a search client. Not quite sure what a "search client" is, but I don't see why a Bing proxy (using the Bing Search API) is any more of a search engine than a Google proxy (using the Programmable Search Engine).

Linda703 4y ago

[dead]

marginalia_nu 4y ago

I think a big part of why Google has such sticking power is that on the one hand they actually are pretty good at some things (even though they visibly flounder with others), but web browsers are heavily designed around leveraging the particular tasks Google is really good at. This is manifested by turning the URL bar into a search bar by default, making URLs difficult to manually edit (especially on mobile), making bookmarks inaccessible and difficult to manage by requiring multiple clicks to access, and having an interface that makes it easy to accidentally bookmark the wrong websites. I don't believe there's some big conspiracy where Google has orchestrated this, but it's probably more like an effort for every web browser to mimic Chrome without realizing why that has problematic knock-on effects. I really sort of wish there were good alternative web browsers that weren't ostensibly waging war on user agency, or at least gave more than token customization abilities.

tintedfireglass 4y ago
Yes,These days most people think that GOOGLE=INTERNET and look on GOOGLE for the nearest hotel or ask GOOGLE today's weather. Google has basically become so big that for most people Google is the manifestation of the internet. This reliance on web search as a gateway to the internet also started with google as in the days of IE we had separate URL bars and search bars, toolbars and bookmark tabs were the norm and there were many different ways people used to obtain their news,talk to friends and in general consume content on the internet. But Google basically turned the browser into a funnel that sends all internet users to https://google.com https://google.com and as chrome has most market-share in both desktop and mobile(thanks to android). Most browsers adopted the SEARCH-ENGINE route in their browsers leading to the position we are in. Today the average person visits a website only in two ways 1.Auto-fill on their URL...sorry search bar 2.Type the name of the website on the google search engine. I have no clue why the feds don't care about big tech monopolies the way Microsoft was attacked back then for IE which is basically child's play compared to what companies(even Microsoft with Edge) are doing today.
- bil7 4y ago
  > basically turned the browser into a funnel that sends all internet users to https://google.com https://google.com Exactly right and completely intentional. I saw they also pay Apple $15bn to be the default search provider on iOS [0]. Annually. [0] https://www.macrumors.com/2021/08/27/google-could-pay-apple-15-billion-default-search/ https://www.macrumors.com/2021/08/27/google-could-pay-apple-...

kristopolous 4y ago

The realization I made recently is the financial incentive of the page view is the corrupting force that ruins search The content producer is incentivized to make content as quickly as possible in as little effort as possible and cover it with as many ads as possible. On the other end, it's within GBYs interest to be slightly confused; by not nailing it, the user engages with the search using different words, potentially hitting an adword and clicking. GBY could be convincing the user that they're just using it wrong and need to execute more searches. So the financial cost/reward model is configured in a way that gives high margin to spammy trash covered in irritating obstructive ads and search engines that are mysteriously confused by what we ask for. And that's the modern web. Finally we are taught to tremble in fear if we dare consider touching the sacredness of The Market. So here we are stuck with trash technology to search avalanches of garbage.* Luckily there appears to be a few search engines that focus on sites without advertising and trackers and they seem to produce nice results --- * occasionally people like to claim this is just freedom or something so let me clear that field now: being forced to focus on maximizing profit isn't a free society. A free society would permit various missions to be pursued without fear of being squeezed out of existence by those focused exclusively on profit maximization. If priorities are imposed by a p&l tyranny, that's not freedom

dontcontactme 4y ago
What are these search engines that focus on sites without advertising and trackers?
- kristopolous 4y ago
  Try https://teclis.com https://teclis.com and https://search.marginalia.nu https://search.marginalia.nu My test queries are things like "souffle recipe" or "worst celebrity hairstyles" - if it can survive that it can survive anything
- tintedfireglass 4y ago
  https://wiby.me https://wiby.me is a good one. I actually found this blog on wiby
- 4y ago

langsoul-com 4y ago

Do Chinese search engines have their own index engines? They're censored, but if you could remove that, would they be as good as Google, Bing, yandex?

Seirdy 4y ago
Yes, I listed several. Scroll down a bit.

mysterydip 4y ago

I know a common complaint is search engine x fails at finding info for knowledge domain y, usually something technical that in the overall lexicon is ambiguous with something more generic/popular. I wonder if a good solution to this is to use one of the smaller open engines in a private instance and just submit the handful of sites you know have the relevant info?

ta8903 4y ago

>Two engines were excluded from this list for having a far-right focus. Does this mean like extremist terrorism or regular right wing stuff? Would be interesting to see.

kjrose 4y ago

Does anyone know what the search engines that were excluded for "reasons" are? I like having thorough lists and I am curious what as many as possible are regardless of political silliness or differences in using financial systems. My guess is the crypto one is presearch, but the others I have no idea.

prophesi 4y ago
They list out their reasons here https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#exclusions https://seirdy.one/posts/2021/03/10/search-engines-with-own-... Sounds like two were excluded for having a far-right focus, one for crypto payments, and the rest for being small/simple proof-of-concepts That said, I would also be interested in what engines in particular were excluded, with a small blurb as to why it wasn't included.
- kjrose 4y ago
  Precisely. I don't care the reasons. I would want to know what they are. I absolutely hate when people claim to be providing a "complete" list of something and then skip things because 'reasons'.

LukeEF 4y ago

I bounce from default engine to engine (most recently startpage as a 'better' google) but so far always find myself back with google search. Last time I went elsewhere, I recorded the occasions when I appended the URL bar search with @google. Quick info on a business (address, telephone # etc.) and maps dominated. I haven't used another engine that can deliver similar. I also realized how much time I spend looking at maps and street viewer... The knowledge graph and maps are a massive search moat.

freediver 4y ago

By the looks of it, we are witnessing the "Cambrian explosion" of web search. Search engines will evolve and those that do not adapt might be up for a nasty surprise by the end of Mesozoic Era, no matter how big they are.

kypro 4y ago

As an English language user Yandex is awful for typical use in my experience. I'd say at least a third of the results it serves on its English site are Russian. That said, I find myself using Yandex quite a bit simply because I can assume it has a Russian bias, or at the very least it doesn't have the same biases you might expect from a search engine operated in the West. I think a lot of people would be surprised at the extent at which Google and other Western search engines filter their search results today. A lot of content which might be labelled "extreme" or politically problematic I find is now either missing entirely from Google search or pushed so far down in the results that it's practically inaccessible. On Yandex I find I don't have this problem. As an example during the pandemic Yandex was pretty useful for finding conspiratorial content about Covid-19 which was censored by Google. Not that this is necessarily a good thing for the average user, but it has its uses if you're trying to come to your own conclusions about things rather than relying on the narrative provided by Western media companies. It's obviously also been useful when trying to better understand the Russian side of the ongoing RU/UA conflict which has either been completely absent or very unfairly portrayed by most Western media sources (again, perhaps for good reason, but still).