30 ms·

Ask HN: Why doesn't anyone create a search engine comparable to 2005 Google?

398 points by syedkarim 5y ago

I seem to recall that Google consistently produced relevant results and strictly respected search operators in 2005 (?), unlike the modern Google. And back then, I think search results were the same for everyone, rather than being customized for each user.

keddad 5y ago
While I feel that Google has become worse in last couple of years, I'm pretty sure it is still better now when 15 years ago. Maybe it is just some kind of nostalgia?
- micromacrofoot 5y ago
  the internet has changed, partially due to google's influence instead of discussion forums and Q&A sites, everyone's on facebook/twitter/discord/slack/snapchat/tiktok/etc... none of that is really very google friendly online marketing and SEO is a much larger industry now, so with less (by % of total) searchable content generated by people (which is on social media) a lot of the high-ranking content that appears in search is highly optimized marketing then you have other kind of weird things like... half of all internet traffic being bots
abhaynayar 5y ago
I'm probably the only person who doesn't think Google search has deteriorated. I play security CTFs, so a lot of times I have to search for peculiar technical details on various software. Also, like any other human being, I also make generic queries. In both cases, I feel like I almost always get to the desired webpage within the top few results.
- jeffbee 5y ago
  Well no, it’s you and me and the whole google search quality evaluation team and everyone who works on google search and like 99% of the general public as well. The meme of falling search quality infects only HN. Mostly what people are complaining about is that the quality of the web itself is in free-fall.
- Ellipse0934 5y ago
  It honestly depends on what you are searching for. Case 1: You just want the name of the website, or an article, example "Facebook" -> fb.com, "Gordan Ramsay" -> Wiki/official website/Celeb gossip website you are good. Not much competition here. Case 2: You are looking for something technical like "GNU rnano CVE-abcde"/"OpenBSD ARM64 Qualcomm Wifi driver not working", you are again in the fine territory, not much if any money to be made here so very less competition. There will be the official forums, websites, maybe some conference websites in this category. Case 3: "Chicken potpie recipe", "How to be more organised": This is the category where people are trying to game the SEO algos. How the hell do recipe websites with 27 popups, 12000 word essay on the secret family history ends up on top ? There are a huge number of passionately made simple recipe websites but they have to be "found" by us. For the second query I mention about being more organized I think most people are looking for some sort of a review article which looks at some various schools of thoughts regarding discipline, cleanliness pointing to further resources and exploring the why and what to do for this. Here the search engine needs to determine the context of the query which is fairly abstract and then the internal heuristics it uses are supposed to drive it to a meaningful list of websites. Maybe the average joe would like to click cosmopolitan's article but I would never do that. Based on my previous click history maybe google should determine what I kind of links am I looking for. But when they figure that out they'd much faster use this behavioral insight for advertisers. A great search engine is basically a primitive personal librarian, I'd pay a yearly subscription for one. The internet is vast and it has stuff that I don't know about. How my 7 word abstract query is gonna get me there is the question mark. Also, for a lot of queries the top results can be plagued by spammy/fraud results which are on top because they managed to trick the SEO algos. These bad actors were not as prevalent for 2005 google.
- tsian2 5y ago
  It's always worked fine for me when it comes to finding simple things with a search. I think it's deteriorated in some ways though. I don't find the advanced search operators reliable anymore (eg. give me all the news about a topic published between certain dates) and I think it caps collections of things very early now, rather than returning the "billion" of results it says it has (eg. give me more than the 1000 most popular cat memes that I've seen before, or all the books about beaches).
est 5y ago
Because today's web are full of walled gardens, and most content are going mobile , in streaming, and SPA rendered, which is no longer plain text based.
dave333 5y ago
Does Gigablast ignore or downrate stuff on .info domains? Seems to like https://www.fiendishsudoku.com/ https://www.fiendishsudoku.com/ for "fiendish sudoku" search but doesn't know about https://www.extremesudoku.info/ https://www.extremesudoku.info/ for "extreme sudoku" search.
Hakashiro 5y ago
What is your gripe with DuckDuckGo?
ofou 5y ago
An end-to-end machine learning system should compress all the web into a good search engine soon.
rasengan 5y ago
This is how Private Search [1] works since it decouples the search from the user. This means nobody knows both who searched and what they searched for. This is a huge leap for privacy in search. [1] https://private.sh https://private.sh
- Lucasoato 5y ago
  Tried it but it just says: "Something went wrong. Please try again."
  - ZetaZero 5y ago
    same here
    - rasengan 5y ago
      It should be working now! Thanks for the heads up. There was a traffic issue.
- jaywalk 5y ago
  Looks like your comment here caused enough curiosity to take the service down.
- gbmatt 5y ago
  and i work with rasengan on private.sh so yes there's some issue there. one of the back end servers is returning a max capacity error of sorts... we are checking into it.
- snarkypixel 5y ago
  Is it a proxy to other search engines or are they building their own?
  - rasengan 5y ago
    It's a multi part partnership with Gigablast. Gigablast sees the searches, but not who searches. Private.sh sees who searches, but not what they search for.
BitwiseFool 5y ago
Natural Language Processing is a pox on modern search engines. I suspect that Google et. al. wanted to transform their product into an answer engine that powers voice assistants like Siri and just assumed everyone would naturally like the new way better. I can't stand how Google is always trying to guess what I want, rather than simply returning non-personalized results solely based on exactly what I typed in the textbox. While that may be good for most people, there is still a lot of power and utility in simple keyword-driven searches. Sadly, it seems like every major search engine has to follow Google's lead.
- marginalia_nu 5y ago
  I think some NLP is strictly beneficial for a search engine. You may think "grep for the web" sounds like a good idea, but let me tell you, having tried this, manually going through every permutation of plural forms of words and manually iterating the order of words to find a result is a chore and a half. Like, instead of trying PDP11 emulator PDP-11 emulator "PDP 11" emulator PDP11 emulators PDP-11 emulators "PDP 11" emulators PDP11 emulation PDP-11 emulation "PDP 11" emulation Basic NLP can do that a lot faster without introducing a lot of problems. I do think Google currently goes way overboard with the NLP. It often feels like the query parser is an adversary you need to outsmart to get to the good results, rather than something that's actually helpful. That's not a great vibe. However, I think the big problem isn't what they are doing, but how little control you have over the process.
  - JPKab 5y ago
    Semantic search requires NLP. So does the Q&A format the OP is complaining about. People conflate all things NLP to the latter, and forget about the former.
  - BitwiseFool 5y ago
    Maybe I'm not using the right qualifiers around the term NLP. The kind of NLP I was referring to is something like "Hey google, what is natural language processing?" and orienting the search around people asking questions in standard(ish) English like they would to another person.
nickpp 5y ago
Because we're not having a 2005-Web anymore. More to the point, SEO & Google have evolved together. To have barely relevant results today you need to be good. That takes stellar talent which costs huge amounts of money. Thus, the Google of today, which is optimized to extract that money from us.
- thisisnotatest 5y ago
  Yes, I think you'd call it a Red Queen Problem: “Here, you see, it takes all the running you can do to keep in the same place.” -Lewis Carroll's Through the Looking Glass
- ginko 5y ago
  But shouldn't all the blogspam be so hyperoptimized for Google's algorithm that is should be straightforward to detect and ignore/downrank it?
  - elcomet 5y ago
    It's not that easy, they are optimized for many metrics..
  - beingflo 5y ago
    No because google's algorithm is not well known publicly. Also, if it was straightforward to detect then google could downrank it as well.
    - ginko 5y ago
      The point is that SEO would just immediately adapt to Google's changes. If a smaller search engine filtered these out then it would likely stay under the radar.
      - thefreeman 5y ago
        you know that legitimate sites perform SEO as well, right?
wodenokoto 5y ago
The web has changed drastically. I’d imagine 2005-google engine today would be nothing but abandoned Wordpress blogs with comment spam.
- ghaff 5y ago
  And even more copy-pasted spam than already exists. The early Google (and other even earlier search engines) were invented for an Internet world which, if not pristine and pure, was at least mostly fairly legit content. Today's Internet is probably 90% deliberate spammers and scammers.
- warning26 5y ago
  I suspect this is exactly it—a lot of what made 2005-era-Google good wasn't necessarily Google's own doing. It was that SEO people hadn't yet fully figured out how to game the system yet. If you took an exact copy of Google circa-2005 and had it crawl today's web, you'd probably get mostly "SEO optimized" irrelevant blogspam.
vincent_s 5y ago
Some people try: https://www.mojeek.com/ https://www.mojeek.com/ https://fireball.com/ https://fireball.com/ https://search.brave.com/ https://search.brave.com/
- 7373737373 5y ago
  Time for an https://github.com/sindresorhus/awesome https://github.com/sindresorhus/awesome search engines?
- ColinHayhurst 5y ago
  Mojeek founder story here: https://blog.mojeek.com/2021/03/to-track-or-not-to-track.html https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm... No-tracking and independent from the start. Now at 4.6 billion pages with own infrastructure and IP. Went to market in 2020 with contextual ads and API. Self-disclosure: CEO
  - snovv_crash 5y ago
    HN is wild: 30m after something is mentiond, the CEO chimes in.
    - kingcharles 5y ago
      Now, if we could just get that on the Facebook thread... ;)
  - prox 5y ago
    Never heard of Mojeek. I will try it for a month and see how it works. Currently using DDG 99% of the time.
  - bullen 5y ago
    Do you use some sort of PageRank?
    - ColinHayhurst 5y ago
llaolleh 5y ago
Everyone runs in the other direction anytime a search engine is mentioned. The thought of competing with Google turns people off. Even in 2021, despite how bad it's become, it's still miles ahead of other competitors.
- prox 5y ago
  I disagree. A lot of people I know already switched to Duckduckgo. Google’s ability to get relevant results is dropping like a brick, while the quality of DDG has been improving slowly but steadily.
  - datenarsch 5y ago
    I wish I could agree but from my experience, DDG's search results aren't really that great. Often even worse than Google's. And another private company is not the answer I believe. We need something more drastic, an open-source search engine organized as a genuine non-profit organization. Something like that. Otherwise, whatever replaces Google will just turn into another Google as soon as it gets any momentum.
    - llaolleh 5y ago
      I think open source will be tough because you're going to need a lot of saints to work on a search engine of Google's caliber. Maybe an alternative revenue model instead of ads.
      - onecommentman 5y ago
        Consortium of universities, perhaps? Every top school (globally) kicks in some design and development time. It seems odd that the most critical link to access information on the planet is not the product of academia. With a country’s skin in the global game, there may be better leverage to keep it free and open for their citizens.
gbmatt 5y ago
Ha, yes, I've done that at https://gigablast.com/ https://gigablast.com/ . The biggest problems now are the following: 1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages. 2) Hardware costs are too high. It's much more expensive now to build a large index (50B+ pages) to be competitive. I believe my algorithms are decent, but the biggest problem for Gigablast is now the index size. You do a search on Gigablast and say, well, why didn't it get this result that Google got. And that's because the index isn't big enough because I don't have the cash for the hardware. btw, I've been working on this engine for over 20 years and have coded probably 1-2M lines of code on it.
- justinzollars 5y ago
  This is great! I found something other engines do not pick up! apparently I signed an agile manifesto in 2010 https://agilemanifesto.org/display/000000190.html https://agilemanifesto.org/display/000000190.html
- collin128 5y ago
  Have you ever looked at the Amazon file? I'll see if I can track down the link but I remember somebody sharing a dump with me from Amazon that apparently was a recent scrape. Edit: https://registry.opendata.aws/commoncrawl/ https://registry.opendata.aws/commoncrawl/
  - web007 5y ago
    That's Common Crawl, they do the spidering of some billions of webpages but that's still a tiny percentage of the web versus Google or Bing.
    - cschmidt 5y ago
      Do you have any stats on that? I've always wondered about the coverage of Common Crawl, if you include all the historical crawl files too.
    - 5y ago
chilling 5y ago
Yesterday there was a discussion[1] about it and someone suggested yandex.com. I'm using it since than and really love it. It's like going back to 2003 where everything was just plain and simple. [1]: https://news.ycombinator.com/item?id=29393467 https://news.ycombinator.com/item?id=29393467
pydry 5y ago
Early 2000s google index ran in a garage. The current google index has dedicated power stations. It's a bit like the car industry - you could run a startup from your garage in the early days but you need titanic amounts of capital to compete now thanks to vertical integration. Major governments and billionaires can compete but everybody else is locked out of the market (most "startups" use bings index).
- mleonhard 5y ago
  Google's datacenters are huge because they save user behavior data, not because their web search index is particularly big. Also, Google Search wastes a lot of resources on the "search as you type" feature. Running a search engine in your garage is feasible today because hardware and connectivity have improved much faster than the size of the WWW.
  - pydry 5y ago
    It's the frequency of updates that chews power. Also, that user data is used to improve search results and mitigate webspam that didnt exist in 2005.
- R0b0t1 5y ago
  I was thinking about exactly that. If they used simpler index would they be getting better results? There's not a lot of selective pressure so they just keep adding to the index algorithm.
- abdel_nasser 5y ago
  whatsapp was run out of a single cabinet.
mrkramer 5y ago
They do[0] but nobody cares anymore. Google controls web distribution through Google Chrome. I think we are at the point of no return. There won't be any competition anytime soon no matter what US government does. Only innovation can displace Google. [0] https://search.marginalia.nu/ https://search.marginalia.nu/
- BbzzbB 5y ago
  Marginalia is great to find blog posts, personal sites and other long form content, but it's not a replacement for Google nor intends to.
  - marginalia_nu 5y ago
    It does operate on a scale and principle fairly similar to early 2000s google, so the comparison isn't that far off, but yeah, it's quite some way before it's viable for general search. Dunno if I'll ever get there, but it does consistently seem to get better so who knows.
    - BbzzbB 5y ago
      Isn't it's familiarity to early Google a side-effect of the early Internet being text-heavy sites in the first place rather than a similarity in the search engine? Unless I am misunderstanding your site's intent, even if you reach the dream engine you are trying to achieve, I won't be using it to search answers for coding questions on SO, how-tos for car repair, sites to stream movies, governmental page for X need, transcript for earnings calls, etc. In my experience it is better than Google at what it does if I'm looking for long-form texts (exception being scientific/peer-reviewed articles, Google tends to shoot me those for the type of queries I make on Marginalia), but is very much complementary rather than a replacement.
      - marginalia_nu 5y ago
        I guess it depends on what you are looking for on the Internet I guess. Right now the biggest problem with Marginalia is that it has a fairly uneven quality level. For some queries it's absolutely incredible. For others, it doesn't really provide much useful results at all. I do think it's possible to even that out a considerable bit, to make it more viable for general queries. It's never going to be able to answer every query, but it probably could answer a lot more than it does.
BbzzbB 5y ago
No mention of DDG in the comments? Is there a reason I'm not seeing or it's just not the preferred alt-search on HN? Seems to have been working fine for me when I struggle to get past the funnels and content mills on Google.
- pantulis 5y ago
  I dont find search results to be too relevant (at least for me, also Spaniard here). It is my default search engine only for the bang commands.
- KennyBlanken 5y ago
  For me, DDG results are even worse than Google. It's set as my default and I'd say at least half of my searches in DDG generate completely useless results...pages of obviously SEO'd garbage. DDG also doesn't support showing a site's basic structure in the search results (ie, the card of a company's website with Products, Contact Us, Support, etc) and the preview text is garbo as well...it reminds me of 1990's era electronic card catalog search excerpts. I look at the first page or two, give up, search google. While I have to hunt a bit in the results, I do eventually get what I wanted.
  - infinitezest 5y ago
    Every time this comes up I'll see a few people talk about how the results aren't relevant but it has not been my experience. I've been using DDG as my main search engine for a few years and never have to go beyond the first page. I really curious why that is.
    - JohnFen 5y ago
      My experience is like yours -- DDG is legitimately better than Google. My hypothesis is that it's related to how you construct searches. I expect Google probably does better if you learn how to talk to it, since it seems to want to interpret your query rather than take it literally. My searches tend to be keyword-oriented rather than natural language. I think DDG does better with those.
rovingEngine 5y ago
I think Google was “better” from a users point of view in 2005 because it wasn’t that good at selling ads yet. I still remember the epiphany of the first time I used Google in 1999. It was amazing. I’ve thought the same about pre-ad Twitter and Facebook. Early on, startups with free services look a lot like non-profits and just maximize user benefit to grow. The problem is they’re not non-profits, and have to make money at some point. That has tended to mean ads. I’d easily pay, say, $9/mo to have access to an ad-free search engine that made me feel the way 1999 Google did.
- mmmmmbop 5y ago
  $9/mo is not going to cut it. Google's domestic annual revenue per user in 2019 was $256. [0] That's $21.33 per month. Not all of Google's revenue is from Ads, of course, but the vast majority is. (Let's ignore for now the valid counterpoint that Ads are increasingly served on other Google properties than Search.) But even charging users $21.33/mo for an ad-free search experience most likely wouldn't be enough. By providing such an option, you'd greatly reduce the value of the remaining Ads pool. The optimistic perspective on this is that if you are one of the users with disposable income, you're essentially subsidizing a great search engine and a suite of other tools for the less well-off ones. [0] https://miro.medium.com/max/6545/0*YTqXb-F5UiVhtlIS https://miro.medium.com/max/6545/0*YTqXb-F5UiVhtlIS
  - rovingEngine 5y ago
    Let’s say ads will always make more money (I have no reason to believe they won’t), and that’s required to be the dominant search engine because the web is big and expensive to organize. I’d bet there’s some way to characterize what I and others liked about the earlier web and create a search engine that just worries about that stuff. I’d pay $9/mo for whatever 1/3 of Google’s spend per user would get me. That’s not to say this thing would “beat” Google, but it could profitably exist.
    - themacguffinman 5y ago
      I doubt it, because 1/3 of Google's spend per user isn't enough when you can't attract many paying users in the first place, because you would charge much more than $9/mo, because almost no one wants to pay for a search engine so your revenue will have to make up for those people too, and then even fewer people are willing to pay more than $9/mo for 1/3 of the quality. And then I'd guess the 20 remaining users will still complain because 1999 Google is a nostalgic memory impossible to recreate without a 1999 internet for a 1999 self to live in and has little to do with raw search quality.
jakub_g 5y ago
Cliqz wanted to build new search engine but failed. It's just too difficult to operate at that scale and break the existing monopoly of big G. https://www.burda.com/en/news/cliqz-closes-areas-browser-and-search-technologies/ https://www.burda.com/en/news/cliqz-closes-areas-browser-and... https://news.ycombinator.com/item?id=23031520 https://news.ycombinator.com/item?id=23031520 https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch.html https://0x65.dev/blog/2019-12-06/building-a-search-engine-fr...
- hunterb123 5y ago
  And then Brave bought them and it succeeded. Cliqz is now Brave Search, I use it for all my devices, it's great. Works better than DDG and sometimes better than Google. I only hash bang every 100 searches or so, most of the time Google doesn't have it either. It's just to make sure. http://search.brave.com/ http://search.brave.com/
- arthur_sav 5y ago
  What if we didn't try to replicate google. Smaller and niche search engines would probably work better in this new world of vast information.
amelius 5y ago
Also, where are the books about writing a search engine? Knuth's "Searching and Sorting" volume desperately needs an update.
- mindcrime 5y ago
  I don't even know if anybody has written a book specifically about search at "web scale" (no MongoDB jokes here, please). But about the closest things I know of would be something like: https://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia-Information/dp/1558605703 https://www.amazon.com/Managing-Gigabytes-Compressing-Multim... https://www.amazon.com/Information-Retrieval-Implementing-Evaluating-Engines/dp/0262528878 https://www.amazon.com/Information-Retrieval-Implementing-Ev... https://www.amazon.com/Introduction-Information-Retrieval-Christopher-Manning/dp/0521865719 https://www.amazon.com/Introduction-Information-Retrieval-Ch...
indymike 5y ago
Brave's new search engine seems to work pretty well. Have been using it as my primary for about 10 days, and so far, I've only had to revert to Google once, and when I did the results were chock full of spam.
- travisgriggs 5y ago
  Can echo this. About 30 days going all devices. I’d say about once a day I do a !g, and rarely do I actually find something there, it usually just ends up being a confirmation search.
- concinds 5y ago
  The nice thing about Brave Search is that they're trying to create an index completely independent from Bing/Google, and they seem to be trying to innovate on ways to get there as well with their Web Discovery Project[0], unlike DuckDuckGo. They've announced Brave Search will get ads soon, with a premium version without ads, which I think is acceptable given the costs of running an independent index sustainably. [0]: https://brave.com/privacy/browser/#web-discovery-project https://brave.com/privacy/browser/#web-discovery-project
swframe2 5y ago
Have a look at gpt-3 if you want to see what the future dominant search engine will be. It will not find relevant results, it will write it on the fly customized for exactly what you want to read. (Maybe products will just ship to your door and be auto paid because the future ad targeting AIs will know you so well.)
- marginalia_nu 5y ago
  What if you are looking for something written by a human?
  - swframe2 5y ago
    You can always go to a library or bookstore.
    - marginalia_nu 5y ago
      Let's imagine I want to talk to the author of the content. How can I do that if it's just a souped up markov chain?
      - throwawayffffas 5y ago
        The markov chain can also power a chat bot.
        marginalia_nu 5y ago
        But then they would need to know that the person sending the email is the same person that read a specific article.
andrewclunn 5y ago
What about a search engine that only indexed information and technology "alternative" sites, specifically to give you the results most likely to be purged or demoted from Google's results? Would be simple enough in scope and have a built in market and use.
criddell 5y ago
Why don't you want personalized results? If I search for "subaru service" I want to find Austin Subaru, not Thorp Subaru in Cape Town.
- arthur_sav 5y ago
  I pretty much hate "personalized" search recommendations. If i'm looking for something it's usually not in relation to me but in relation to the world. If i wanted something more relevant to me, then i would specify what aspect of relevance (country, gender, age etc...) i would like instead of playing the guessing game.
  - criddell 5y ago
    > If i'm looking for something it's usually not in relation to me but in relation to the world. If that's true, then I don't think you are a typical search engine user. The personalization should just be used for defaults. You can always make a more specific query to focus on aspects you are interested in.
- vikingerik 5y ago
  Why didn't you just search "austin subaru service"? If you want a query narrowed down by location, that's your job to say so. Sure, it feels great when the engine guesses something like that correctly -- but it comes out worse overall for the plentiful cases where you have to try to compensate for it guessing wrong.
  - criddell 5y ago
    Why should I have to do all that work? I want the machine to do it for me. I can only think of examples where I want personalization. What's an example query where it interferes?
    - jeffbee 5y ago
      Amazing that the same site that thinks copilot will just generate programs for us also thinks it is literally a crime for a search engine to infer anything.
nfriedly 5y ago
I think DuckDuckGo is closer to what you want. Same results for everyone, better privacy, and they're proactive about improving their results. https://duckduckgo.com/ https://duckduckgo.com/ Part of the problem is that there's a lot more low-quality content to wade through now than there was in 2005. I think the Google of 2005 would have trouble delivering quality results today also.
- Kiro 5y ago
  Try searching for the same thing from your computer and your phone and you will get different results. Also, their results come from Bing so any improvement happens at Microsoft.
  - JohnFen 5y ago
    They do use Bing, but not solely Bing. DDG isn't just a frontend to a different search engine.
    - bla3 5y ago
      It's a bing frontend with a few special cases handled differently. For most queries, you get bing results. Easy to check by comparing results.
      - kevin_thibedeau 5y ago
        I see Russian sites from Yandex all the time on DDG.
- Sunspark 5y ago
  This. DDG is my primary search engine now and has been for awhile. I don't use Google anymore to search unless I really need to. The algos they use today are not the same classic ones that actually returned results.
  - kspacewalk2 5y ago
dragonwriter 5y ago
> Why doesn't anyone create a search engine comparable to 2005-Google? Because the universe being searched isn't the internet of 2005 and earlier, and because user expectations have moved on, too. Plus the index expense.
michaelyuan2012 5y ago
here is Drop Side Trailer information, it should be helpful for those logistic company. https://www.dreamtruegroup.com/drop-side-trailer/ https://www.dreamtruegroup.com/drop-side-trailer/
michaelyuan2012 5y ago
for the people in logistic business area who search in Google, Flatbed Container Semi Trailer, it should be good for your reference. https://www.dreamtruegroup.com/flatbed-container-semi-trailer/ https://www.dreamtruegroup.com/flatbed-container-semi-traile...
greyman 5y ago
1) Google is better at AI, for example let's take this sloppy search: "some joke where you can't tell if it is serious or joke" It is called Poe's law, and Google returned it at #4. Bing or Duckduckgo don't have a clue... 2) They have a years of user's data, like for specific term, they see what users clicked most, so they see which results were perceived as most relevant. It is hard to catch up if you dont have such data. 3) They developed anti-spamming tools during the years of fighting against SEO-spammers.
- lolpython 5y ago
  > 1) Google is better at AI, for example let's take this sloppy search: "some joke where you can't tell if it is serious or joke" > It is called Poe's law, and Google returned it at #4. Bing or Duckduckgo don't have a clue... Interesting, I was looking for a good benchmark like this. For me Google returned it at #5 with an image/related terms carousel before it which places it physically more around #7 on the page. Brave Search (never tried it before today) puts Poe's Law at #8. So Google is still better. But the other results are mostly worse (IMO) on Google. Here are the first 8 results: - 175 Bad Jokes That You Can't Help But Laugh At - Reader's (rd.com) - 57 Hilarious, Silly Jokes No One Is Too Old to Laugh At (bestlifeonline.com) - 145 Best Dad Jokes That Will Have the Whole Family Laughing (countryliving.com) - Sarcasm, Self-Deprecation, and Inside Jokes: A User's Guide (hbr.org) - Poe's law - Wikipedia (wikipedia.org) - Managing Conflict with Humor - HelpGuide.org (helpguide.org) - 175 Bad Jokes That Are So Cringeworthy, You Can't ... - Parade (parade.com) - Encouraging Your Child's Sense of Humor (for Parents) - Kids ... (kidshealth.org) And here are the first 8 results from Brave Search: - phrase requests - Is there a word for "pretending to joke when ... (english.stackexchange.com) - Joke - Wikipedia (wikipedia.org) - “Are you joking or serious?” – The Caffeinated Autistic (thecaffeinatedautistic.wordpress.com) - How do I tell when people are joking or being serious? (reddit.com/r/socialskills) - be a joke | meaning of be a joke in Longman Dictionary of (ldoceonline.com) - Quote by Ricky Gervais: “If you can't joke about the most (goodreads.com) - How can you tell if someone is joking with you or not? (quora.com) - Poe's law - Wikipedia (wikipedia.org) ----- edit: I did not count to 8 correctly the first time. Fixed that.
  - 5y ago
axegon_ 5y ago
Two major reasons: costs to build and maintain and manpower needed. Both are practically impossible to come by.
tigerlily 5y ago
Surely there must be some way to have distributed search compute a la folding/seti@home or those mersenne prime guys. I'd gladly pool in some of my CPU time if it helps build a better search.
- teddyh 5y ago
  https://yacy.net/ https://yacy.net/
  - tigerlily 5y ago
    Thanks!
kumarsw 5y ago
I feel like we are at the low point or even losing the battle between search engines and SEO spam. Maybe it is time for the Yahoo-style curated directory to return? We seem to be getting a microcosm of this with the awsome-* GitHub lists and Gemini with its near-nonexistent search.
willcipriano 5y ago
I'd like to see a "just search" engine, all it does it search for a specific string, case insensitively, across the entire web. No curation or anything, just sorted in lexicological order closest match first maybe falling back to page age if it has more then one exact match. Perhaps give me some regular expressions as well.
- jeffbee 5y ago
  That would be easily the worst search engine ever deployed. Imagine just returning all docs containing the word “bicycle” in chronological order. Useless.
  - willcipriano 5y ago
    For "Bicycle" it would suck but I don't often use search engines that way, for "High Timber ALX 29" you'd probably get something like this: https://www.schwinnbikes.com/products/high-timber-alx-29?variant=41490393628841 https://www.schwinnbikes.com/products/high-timber-alx-29?var... I wouldn't use it for everything but sometimes that is the exact behavior that I want. I'd use duck duck go for more general searches.
    - jeffbee 5y ago
      That is the top hit on google for that search, so what’s your complaint?
      - willcipriano 5y ago
        Take a random part number off your car, or a portion of a error message and try finding that. It's annoying to have to scroll down over a page or two of autogenerated SEO answers to get to something useful. The first result to appear on the internet is less likely to be SEO and more likely to be the manufacturers documentation or the git commit that spawned your error. It isn't always, but that's why you have more then one search engine. Secondarily I think a search engine that is very simple in it's model and operation is useful for more general free speech purposes. If the major search engines decide they don't like a site like the pirate bay a search for '"Pirate Bay" And "Torrents"' on a search engine that does not curate could still get you there. I guess the point is without curation you have to work harder to find what you want, but nobody is actively preventing you from finding anything. It would help keep everybody honest.
motohagiography 5y ago
I do like the idea that instead of crawling and indexing, the next generation search will likely be more like a federated community search app that indexes the stuff members actually read. Google search isn't so much a repository as a consensus about what's important, hence why it's so politicized to the point of becoming unreliable, but also why it too is vulnerable to disruption. Imo, 2005 google got initial traction because of its tech forum post indexing, as I remember my switch to it was because it became an extension and then replacement for manpages. In that sense, what made it good was it reflected the consensus of what its incredibly influential userbase thought was important and just managed that really well. The demographic impact of the U.S. Gen X all using it at once didn't hurt either. The equivalent today, as a lot of us say, is that blockchains are in the 1997 internet phase, and the service that makes the content of those as navigable as the 90's internet, will likely grow in a similar way. Search that provides young people with privacy and freedom to pursue their true interests will be the dominant strategy. Its success will be because it's a product that rides growth, and not because it "solved a problem." Imo, we all index too much on the privacy pattern because the freedom pattern is too risky. What's changed since that time are the maturity of things like Bloom and other probabilistic filters, Apple's private set intersection, differential privacy, zksnarks, and everybody you'd ask an opinion from now gets their content through mobile devices. Apple's ecosystem is equipped to do this kind of search, but they're too exposed politically to get into it. Meta will likely go there, but nobody's going to trust them willingly. A protocol that generated a cryptograpically strong anonymous index from your browsing - and instead of putting it on google's servers, it was on a chain, or the content index information and its evolving consensus score was included in something like a DNS record - may still unseat these ensconced interests. IPFS and other P2P or torrents might do something like that as well. Blockchains maybe good for that consensus/desire score. It's not something you architect and design top down that has to solve all cases, it will be just another useful product that grows while riding a demographic change. It would be on the level of inventing HTML/HTTP again, which, when you think about it, was just another dude making a thing he needed.
pkamb 5y ago
I would use a search engine that only indexed Reddit, Stack Exchange, Wikipedia, and a small number of other sites. And that specifically blocked Pinterest, Quora, most non-personal “blogs”, etc. People suggest DDG ! operators, but I don’t want to use a site’s (bad, single-site) search box. I want a multi-site SERP that only displays results from known good sites, which are customizable.
- all2 5y ago
  If I could add sites I liked to the index that'd be great. Find a blogger/hacker I like? Add to the index. Can I share my index with others? Can I include their indices in my searches? Search engine as a social media platform? If I follow you, now I can search in your indices?
  - betwixthewires 5y ago
    Yacy might serve your needs well. It is a sort of distributed engine where users run their own index and "neighbors" share their indices with one another.
- pkamb 5y ago
  Even rules such as “if there is a Wikipedia result in the top 10, display it first”.
- monkeybutton 5y ago
  Too bad they whitelist which bots can access their sitemaps!
- guynamedloren 5y ago
  I've been thinking about this as well. As Google search results get increasingly worse, I find myself subconsciously filtering out all the garbage and gravitating towards a small number of known sites; and, as many other HNers do, I frequently mitigate this filtering step altogether by adding add "reddit" to any search in which I'm seeking out real human sentiment. I've done similar optimizations elsewhere to counter Google's trash results, e.g., I've been beefing up my personal recipe database, with the goal being that I can avoid a google search altogether whenever possible, only hitting google as a last resort. More and more I wonder, with the modern internet, is it even a feature that the whole web is indexed? Might be a bug.
drcongo 5y ago
I've been using kagi.com for a month or so now, and it consistently beats DDG and Ecosia for result quality. I'd guess it beats Google too, since last time I used Google it was nothing but ads and spam which is why I stopped.
- freediver 5y ago
  Thank you for the vote of confidence! Better than Google is our goal, glad you perceive it that way.
  - drcongo 5y ago
    You're welcome. I'm really impressed with it most of the time. Still not made it on to the Orion beta though ;)
baggachipz 5y ago
https://kagi.com/ https://kagi.com/ is a new engine (and Orion Browser) which seems like what you're talking about. I've been using it some and like it so far. The browser is fantastic.
moralestapia 5y ago
Please do it! Google is now complete trash. Also gmail, used to have the best spam filters out there, now it's utter crap. Emails from my google analytics account, for whatever reason and disregarding how many times I have clicked on "Not Spam", go to spam, and it's their own service; while messages who are textbook spam ("Hi, I just got some inheritance ...") go to my inbox. AI (in its current state) is crap, when is the industry going to accept these are the emperor's new clothes.
beefman 5y ago
Can you also create a web comparable to the 2005 web? Well, it's wikipedia. So just create a search engine for that, since their search sucks rocks.
freediver 5y ago
We are building one [1] as well as a few other people that I am aware of with different approaches and business models. We also need to be aware that when we remember past times it usually carries a romantic, nostalgic note. Web is very different than it was 15 years ago and the problem of search has evolved. What you are looking for is basically 'grep for the web' but it is just one facet of search that we use today. 15 years ago you would not get an instant answer to a question like you do today and many users would not be able to live without that today. There are also maps and location based answers, all sorts of widgets like translation etc. Also world became more polarized so an objective best search result became more difficult to produce, specially for events covered in news, which means bias inevitably starts to creep in. This is not to say that Google is good or bad today, it is what it is and they are doing best they can. Startups like ours see an opportunity on the market, in large part to help savvy users find what they want. [1] https://kagi.com https://kagi.com
ChrisArchitect 5y ago
related 2 days ago: Ask HN: Has Google search become quantitatively worse? https://news.ycombinator.com/item?id=29392702 https://news.ycombinator.com/item?id=29392702 Inviting all the paranoid/speculative/hearsay/personal experience responses. Lame Ask HNs!!!!!
ColinHayhurst 5y ago
You might call this a search engine based on the principle of Information Neutrality. “Information Neutrality is the principle to treat all information provided (by a service) equally. The information provided, after being processed by an information-neutral service, is the same for every user requesting it, independent of the user’s attributes, including, e.g., origin, history or personal preferences and independent of the financial or influential interest of the service provider, as well as independent of the timeliness of information." I wrote about this in relation to search [0]. We need to be allowed more freedom to choose search engines and services. One (default or selected) choice for search is unhealthy. We shouldn't have to choose between Google or Bing; DuckDuckGo or Startpage; Brave or Ecosia; Mojeek or Gigablast ..... Personally I use all 8 of these and more, as also explained [0]. [0] https://blog.mojeek.com/2021/09/multiple-choice-in-search.html https://blog.mojeek.com/2021/09/multiple-choice-in-search.ht...
- betwixthewires 5y ago
  I'm with you (you run a great engine BTW) and I've considered the UX of some attempts to help users do this. I like Firefox's UI when searching, where you can select the search engine of choice while typing a query. I like customizable metasearch engines like searx, I think it is a phenomenal idea. I wish more niche engines would implement OpenSearch so that they could easily be added. I have considered just making a simple web page with search boxes for multiple engines for personal use as a default home page, but there's friction and again, lots of engines I'd like to use don't implement OpenSearch. I wonder if there's some novel UX approaches to this out there. Meta search engines seem to be the best way so far to do it but there's the problem of customizing ranking, relevance of results and the like that just compounds the problems users experience.
mkbkn 5y ago
I am a non-dev and Ecosia and DuckDuckGo are perfect for me. Not used Google since more than 3 years now.
simonebrunozzi 5y ago
These guys [0] have built something really close to 2005-Google, and possibly slightly better. The parent company, Tiscali, was a huge hit in the 1990s, as it provided internet access to millions of Italians. It went through some struggle for several years, but lately the original founder, Renato Soru, came back to run the company. The company is based in Cagliari, the capital of Sardinia, Italy. [0]: https://www.istella.it/en/ https://www.istella.it/en/
chrisgoman 5y ago
too many crappy websites, probably needs a "committee" to whitelist domains (only good quality ones) but probably too much work for not enough money or needs some monetization strategy
ravenstine 5y ago
I think what [some] people actually want isn't the Google of 2005 but to have a search engine where they don't feel like they're being manipulated.
vangelis 5y ago
They have, sort of: https://search.marginalia.nu/ https://search.marginalia.nu/
fnord77 5y ago
information-dense pages of yore have been replaced by really wordy, probably generated SEO optimized blog junk.
s1k3s 5y ago
I don’t know how Google was in 2005, but in ~2010 I was able to pull a website on #1 with 0 cash spent, just by manipulating PR. That doesn’t seem great to me.
flipdot 5y ago
Not sure if this is any close to what you’re trying to find, but there’s https://github.com/benbusby/whoogle-search https://github.com/benbusby/whoogle-search
gorgoiler 5y ago
Random thought, based especially on using DuckDuckGo for two years: Search engine isn’t singular, it’s plural. (1) Search engine for something I know exists. (2) Search engine for finding something new. There’s a market for both, but you don’t have to solve both problems with the same product. Sometimes I switch to Google for the former, but the latter works well enough for me that I don’t care what else Google would’ve shown me. More often than not, my feeling is Google would only have shown me more ads in addition to whatever I could already find elsewhere.
lgrialn 5y ago
What I miss most of all from the Good Old Days was getting as many hits back as I could read. Rather than being told "No, there are only eight pages of results on anything in the goddamned world. Really. Would I lie to you?"
lolinder 5y ago
The consistent theme every time this comes up is that dealing with the sheer weight of the internet is almost impossible today. SEO spam is hard to fight and the index gets too heavy. However, I wonder if this is a sign that we're looking at the problem wrong. What if instead of even trying to index the entire web, we moved one step back towards the curated directories of the early web? Give users a search engine and indexer that they control and host. Allow them to "follow" domains (or any partial URLs, like subreddits) that they trust. Make it so that you can configure how many hops it is allowed to take from those trusted sources, similar to LinkedIn's levels of connections. If I'm hosting on my laptop, I might set it at 1 step removed, but if I've got an S3 bucket for my index I might go as far as 3 or 4 steps removed. There are further optimizations that you could do, such as having your instance not index Wikipedia or Stack Overflow or whatever (instead using the built-in search and aggregating results). I'm sure there are technical challenges I'm not thinking of, and this would absolutely be a tool that would best serve power users and programmers rather than average internet users. Such an engine wouldn't ever replace Google, but I'd think it would go a long way to making a better search engine for a single user's (or a certain subset of users') everyday web experience.
- throwaddzuzxd 5y ago
  I wonder if we could use some kind of federation (ActivityPub?) to build an aggregate of the search indexes of a curated community. Something like a giant federated whitelist of domains to index.
- skyde 5y ago
  what you are suggesting would make the problem of echo chamber (bubble) worse than it is today!
  - Nasrudith 5y ago
    Awkwardly complaints about echo chamber as a problem tends to not refer to feedback dynamics (crudely but disambiguating refered to as circle jerk) so much "People disagree with me, the nerve of them!". It is not viable to have parties A through Z sharing the same world and all having absolute control over all others. We see this same complaint every time modernation comes up, let alone the fundamentals of democracy.
not2b 5y ago
I think you're being nostalgic for something you don't remember very well. In that era, Google would return a match based on words that appear in the links to a URL but not in the article itself, meaning that it was easy to produce "Googlebombs". For example, from 2005-2007 the top hit for "miserable failure" was the Wikipedia article for George W. Bush. See https://www.screamingfrog.co.uk/google-bombs/ https://www.screamingfrog.co.uk/google-bombs/ for some of the "better" ones.
mrfusion 5y ago
I’ve always wondered why you can’t use SEO optimizations for GOOGLE as a negative weight and penalize those pages. For example if my search term appears in the URL I can almost guarantee I don’t want that page.
ab_testing 5y ago
I think a lot of people are ignoring the issue that the web has changed considerably since 2005. It is approximately 10 times larger in terms of number of websites and web pages. And a lot of it is SEO junk that is just designed for search engines to be easier to parse and show ads in your face. Also user preferences have changed in the last decade or so. I know millenaials and users in their late 30's or early 40's still yearn for the old web where they would type a search term and correct results would astonish them. However, younger users tend to gravitate to videos and that is why a large portion of the google results are now video results.
WalterBright 5y ago
I'd like to see categories like travel, science, history, art, etc. The web pages could pick which categories their page falls into using meta tags. The user has the option of selecting which category they are interested in searching within.
anotheraccount9 5y ago
Check out the dead internet theory. If most people browse 1% of the web, what's up with popular search engines?
erpellan 5y ago
Even if Google dusted off their 2005 codebase and ran it on today's web it wouldn't come close to the results quality of Google in 2005. The SEO industry has been in an arms race with the search engines for 16 years. 2005 Google would be like a goldfish in a piranha tank.
marksbrown 5y ago
I'd like a way of automatically filtering for websites that : * Don't use JS * Don't use Google analytics * Don't weigh more than a few kB per page * Don't show any sites with ads That would be a place to begin.
- betwixthewires 5y ago
  Wiby.me might work for you.
- XCSme 5y ago
  Almost all websites use JavaScript, including this one.
hereforphone 5y ago
Because the money lies in modulating your product according to the whims of the highest bidders.
peanut_worm 5y ago
Isn’t that what DuckduckGo is? DDG is pretty useless though unfortunately.
phendrenad2 5y ago
The 2005 Google model only made sense in the 2005 internet. Google had the luck to become a search monopoly, and they quickly created Chrome to ensure that no one would ever switch away from Google search, so they could maintain the monopoly. Now that Google exists, you can't create another one. There's only room for one. Another thing is the rise of "content sites", like this one (Hacker News). I'm sure YCombinator doesn't like getting hit by dozens of crawlers. The impulse to ban everything that crawls except (Google|Bing|Baidu|VK) is too great. A lot of alternative suggestions are being thrown into this discussion. Let me throw in mine: Reverse the concept of the "crawler". Instead of following links around the internet randomly, require sites to register with you and request to be crawled and/or submit a sitemap. It would be hard to get started, but once something like this gained momentum, I believe that there's room for several of these reverse-search-engines to compete.
gompertz 5y ago
I've been having a lot of good luck with Lycos (yeah, that Lycos, from 1995!)... It never returns pay wall or "opinion" based results (I.e. Medium).
grouphugs 5y ago
[dead]
aaron695 5y ago
> I seem to recall that Google consistently produced relevant results and strictly respected search operators in 2005 (?), unlike the modern Google. You recall wrong. Probably because you were a child and searching for "reddit". Now as an adult, Google can't just hand you adult results by magic. Search operators have changed, but that's because the internet is 1000's of times bigger since 2005. Where as the number of people went from ~1 Billion to ~5 Billion. > I think search results were the same for everyone, rather than being customized for each user. You are not a baby, turn off the customisation. The same issue existed ~2005, Google customised and we had to work to turn it off. Also my idea we were become one world was totally wrong. Google customising for my location was more correct than my idealism. It also helped local businesses get online. That Google is 'evil' by default is a shitty assumption.
boyter 5y ago
I had a brief stab at this with https://bonzamate.com.au https://bonzamate.com.au although its Australia specific to reduce the crawling and indexing requirements. It's main twist is that it runs entirely in AWS Lambda's meaning it costs nothing when it's not being used.
karmasimida 5y ago
Google does its job. I heard HN constantly crying over its deteriorating quality, but I am not noticing it that much, not better not worse, it just does its job. To create 05 Google, it is easily billions of dollars and years of investment, before people will treat you seriously. The reason we didn't get 05 Google could only because it is not profitable. Some nation state attempt to demonopolize the search engine business might work, but I didn't expect any for profit organization to easily attempt doing this, let alone individual hobbyists
jerhewet 5y ago
Do not force me into autocomplete mode when I'm typing in my search terms. I don't care what anyone's "reasons" are for forcing me to put up with flashing, irrelevant bullshit when I'm searching for something. I don't care how "fast" it is. Just let me type stuff into the search box -- including typo corrections and modifications to what I'm searching for -- and hit ENTER to start the actual search. When I'm ready to start my search I'll hit the fucking ENTER key. Stop annoying me with your stupid assumptions about what I'm looking for. This ONE THING is why I switched to Webcrawler.com two years ago. I type in five or ten words with ZERO craptastic guesses flashing around on my screen, hit ENTER, and THEN it returns what I'm looking for.
ineedasername 5y ago
SEO wars are at least part of it. Google's algorithm has evolved over time not just top optimize advertising views/clicks and take over more screen space, but also to battle the constant gamification of their algorithm by SEO that, once you eventually get to the real results, will surface less relevant/spammy/scammy etc results if Google doesn't constantly push back against the worst SEO abusers.
emodendroket 5y ago
Well, Cuil had a lot of money and couldn't do it. I don't know how you quantify your assertions but I suspect that if you brought back 2005 Google it would be easily gamed and struggle to deal with social media sites where a lot of content people are looking for is now found.
dang 5y ago
Ongoing related thread: Gigablast Search Engine - https://news.ycombinator.com/item?id=29421898 https://news.ycombinator.com/item?id=29421898 - Dec 2021 (10 comments)
8bitsrule 5y ago
Looks like millionshort.com (which I learned-of on HN) died recently. For me, its results were more useful than most others (even without the 'leave out the top nnn sources' feature). Hoping it was an experiment that will bear fruit.
jacquesm 5y ago
I'd love a much simpler version of search engines: an engine that I can give a long list of websites to crawl, and to completely ignore the rest.
hvasilev 5y ago
All big tech businesses at their core are monopolies. Once a significant field has been figured out, it is very difficult to compete with the market standard, unless they screw up so hard that that THE AVERAGE user starts searching for an alternative.
causi 5y ago
Lately I've noticed Google has just started ignoring search operators. Search results are missing terms in quotes and include terms with a leading - sign on them. It's like they've decided we're too stupid to know what we're looking for.
richardsocher 5y ago
you.com supports many of the standard operators and has specific reddit, stackoverflow, MDN apps for developers.
hermann123 5y ago
I use https://swisscows.com/ https://swisscows.com/

kenny11 5y ago

I get that for general-purpose searches this is a good idea, but it would be nice if there was an easy way to disable this when you know you don't want it - for example, for most programming searches, if I type SomeAPINameHere the most relevant results will always be those that include my search term verbatim. I don't need Google to helpfully suggest "Did you mean Some API Name Here?", which will virtually always return lower-quality search results. Early Google was a breath of fresh air compared to the stemming that its competitors at the time did, but nowadays even putting search terms in quotes doesn't seem to return the same quality of results for these types of queries that Google used to have.

thisisnotatest 5y ago
I feel your pain. Two workarounds when Google gets it wrong are to put the term in quotation marks, or to enable Verbatim mode in the toolbelt. (I know various people have come up with ways to add "Google Verbatim" as a search engine option in their browser, or use a browser extension to make Verbatim enabled by default.) Disclaimer: I work on Google search.
- Y_Y 5y ago
  Both of these options are disappointing, in my experience. Verbatim mode seems weirdly broken sometimes (maybe it's overly strict), and quoting things is rarely enough to convince Google that you really want to search for exactly that thing and not some totally different thing that it considers to be a synonym. One porridge is too hot and the other is too cold. I know Google could find a happy compromise here if it wanted to. In fact, I bet there's some internal-only hacked-together version that works this way and actually gives an acceptable user experience for the kind of people who have shown up to this thread to show their dissatisfaction.
- vdqtp3 5y ago
  Try this, go to Google and type in "eggzackly this". Two results not containing "eggz" at all. Two results containing "eggzackly<punctuation>this" Two results containing "eggzackly" but missing "this". Google Search is broken. It no longer does what it's directed, it just takes a guess. I suspect part of this is because someone decided that "no results found" was the worst possible result a search engine could give.
  - BbzzbB 5y ago
    Googling that with the brackets I get results containing "eggzackly this" ranked 3, 4, 6 (your comment) and 7 whereas the others contain just eggzackly (or with the 'this' preceded by punctuation as you mention). Therefor I don't see how your last sentence is the explanation (there are results), I've also happened to land on no results found sometimes with overly precise quoted queries (for coding errors mostly IIRC). But it is annoying that it doesn't seem stricktly enforced even when you want it to.

JPKab 5y ago

Regarding the Gatekeeper companies like Cloudflare, it sounds like anti-competitive behavior that could potentially be targeted with anti-trust legislation, correct?

shashashasha___ 5y ago
i would assume its mostly anti scraping protection which is mostly for privacy. you don't want to allow everyone scrap your website, pull and use your info. for example from fb, ig, LinkedIn, github, .... you can build a really big profiling db on people that way. so websites need to know you are a legit search engine first
- karmanyaahm 5y ago
  people can still be targeted if that information is public. anti scraping sounds like security by obscurity
gbmatt 5y ago
it should be. there should be some sort of 'bots rights' to level the playing field. perhaps this is something the FTC can look into. but, as it is right now big tech continues to keep their iron grip on the web and i don't see that changing any time soon. big tech has all the money and controls access to all the data and supply chains to prevent anyone else from being a competitive threat. look at linkedin (owned by microsoft unspiderable by all but google/bing). github (now microsoft using this to fuel its AI coding buddy, but if you try to spider this at capacity your IP is banned) facebook (unspiderable) .. the list goes on and on .. and as you can see, data is required to train advanced AI systems, too. So big tech has the advantage there as well. especially when they can swoop in and corrupt once non-profit companies like openai, and make them [partially] for-profit. and to rant on (yes, this is what i do :)) it very difficult to buy a computer now. have you tried to buy a raspberry pi or even a jetson nano lately? Who is getting preferred access to the chip factories? Does anyone know? Is big tech getting dibs on all the microchips now too?
technobabbler 5y ago
Cloudflare functions kinda like a private security company. They don't go around blocking sites willy-nilly, site owners have to specifically choose to use their service (and maybe pay for it), configuring the bot blocking rules themselves. That's not really Cloudflare's fault. Someone has to do it, whether it's them or a competitor or sys admins manually making firewall rules. Cloudflare just happens to be good enough and darned affordable, so many choose to use them. Hosting costs for small site owners would be much more expensive without Cloudflare shielding and caching.
- gbmatt 5y ago
  I've had extensively dealing with Cloudflare. They have a complex whitelisting system that is difficult to get on, and they also have an 'AI' system that determines if you should be kicked off that whitelist for whatever reason. Furthermore, they give Google preferred treatment in their UIs and backend algos because it is the incumbent and nobody cares about other smaller search engines. So there's a lot of detail to how they work in this domain. It's 100% Cloudflare's fault, and it's up to them to give everyone a fair shot. They just don't care. Also, you are overlooking the fact that Google is a major investor (and so is Bing and Baidu). So really this exacerbates the issue. Should Google be allowed (either directly or indirectly) to block competing crawlers from dowloading web pages?
  - technobabbler 5y ago
    These are all great points.
  - danielmarkbruce 5y ago
    It isn't up to them to give everyone a fair shot. That isn't what their customers actually want. Cloudflare aren't in the "fair shots for all search engines" business. They are in the "stop requests you don't want hitting your servers" business.
    - gbmatt 5y ago
      I'd argue that a level playing field and more competition in the search space is a good thing.
- foxfluff 5y ago
  No, I think it is partially Cloudflare's fault because they offer this service and make it easy to deploy. This shit has exploded with Cloudflare's popularity. Nobody has to do it, but a lot of people will do it when they notice there's an easy way to do it. Cloudflare is very much an enabler of bad behavior here. Now a lot of sites just toggle that on without even thinking about collateral.
adolph 5y ago
At a theoretical level it looks like Cloudflare won't block search engine crawlers. The docs are very Google and Bing oriented and also oriented towards supporting their customers, not random new search engine crawler. Cloudflare allows search engine crawlers and bots. If you observe crawl issues or Cloudflare challenges presented to the search engine crawler or bot, contact Cloudflare support with the information you gather when troubleshooting the crawl errors via the methods outlined in this guide. https://support.cloudflare.com/hc/en-us/articles/200169806 https://support.cloudflare.com/hc/en-us/articles/200169806
AtlasBarfed 5y ago
"targeted with anti-trust legislation" Um, this is America. Every market is basically a trust, cartel, or monopoly. And I don't know if you can hear that, but there is literally laughter in the halls of power. All the show hearings by congress on social media and tech companies only has to do with two things: 1) one political party thinking the other is getting an advantage by them 2) shaking them down for more lobbying and campaign donations No one in the halls of power give two shits about competition. Larger companies mean larger campaign donations, and more powerful people to hobnob with if/when you leave or lose your political office. Of course I think that breaking up the cartels in every major sector would lead to massive improvements: more companies is more employment, more domestic employment, more people trying to innovate in management and product development, more product choice, lower prices, more competition, more redundancy/tolerance to supply chain disruption, less corruption in government and possibly better regulation. Every large company brazenly does market abuse up and to the point of one and only one limiter: the "bad PR" line. So I guess we have that.
- astrange 5y ago
  Companies don't make campaign donations. The people "exposing" them are showing their employees making donations, and employees don't have the same interests as their employer.
danielmarkbruce 5y ago
No, it is not. Cloudflare is giving it's customers what they want. They don't want all kinds of bots claiming to be search engines crawling their sites. Cloudflare isn't hurting cloudflare competitors by doing this. Cloudflare isn't hurting their customers by doing this. To repeat - most websites don't want lots and lots of crawlers. They want the 2 or 3 which matter and no more, because at some point it's difficult to tell what the crawler is doing... (is it a search engine???). They aren't obliged to help search engines. Even if Cloudflare wasn't offering this, bigger customers would roll their own and do.. more or less the same thing.

mrkramer 5y ago

I'm sorry to say but your project is 20 years old and it had no impact at all. You are doing something wrong. Innovation and initiative is needed ala Bitcoin and DeFi not hobby projects which are not picking up in popularity and utility.

ErrrNoMate 5y ago
Bitcoin and DeFi don't have utility outside of gambling and pump and dumps. Not everything (tbh not really anything) needs crypto.
- jquery 5y ago
  Crypto’s biggest achievement is being the financial equivalent of the gulf war oil fires. Just massive pollution. Think of all the good things that computing could be used for… we used to have all kinds of interesting collaboration projects. Instead we are setting those CPU cycles on fire for short term profit.
  - Sohcahtoa82 5y ago
    Imagine if all that processing power was used for Folding@Home. The problem is that cryptocurrencies do not inherently need tons of processing power to operate. You could theoretically run the entire Bitcoin network on a Raspberry Pi. But the PoW algorithm was designed to always produce a block every 10 minutes, no matter how much hashing power was dedicated to the network. Everyone wanted a piece of the block reward pie, so the arms race was created. Proof-of-stake algorithms would eliminate this problem entirely, but PoS is a shitty "rich get richer" method. Granted, with how expensive mining power is, even PoW results in the rich getting richer, but at least it doesn't result in the wasting of gigawatts of electricity.
    - ilammy 5y ago
      > Everyone wanted a piece of the block reward pie, so the arms race was created. And that's intentional – getting people pursue the goal for their own egoistic reasons, because that's bound to succeed. As a result, they all increase the security and stability of the network whether they want it or not, the only way to not do this is to not participate. If the network were running on a single Raspberry, someone bringing two Raspberries could effectively outcompete the other person on block rewards. I'm not sure how this can be avoided without fundamental changes in society with regards to competition and adversity.
- mrkramer 5y ago
  Read Bitcoin whitepaper. Bitcoin was meant to decentralize trust and to eradicate fraud through transparent decentralized database called Blockchain. It is certainly more impactful than hobby search engines taking in consideration Bitcoin was also hobby project but really revolutionary one. Go search what Larry Page said 20 years ago: If innovation is commercially successful it can have more widespread impact.
  - spiderice 5y ago
    So your response to the author saying "I'm trying to be commercially successful, but it's really hard for these reasons" is "You should try being commercially successful"? Ok...
    - mrkramer 5y ago
      I respect his effort but the project is 20 years old and yet not commercially successful? There must be a reason behind it. The project is not good enough. Like I said only innovation can displace Google. Innovation is not something new and different innovation is something better.
  - _jal 5y ago
    The bitcoin brainworms do bad things to people. I suggest you update your patter some, though. A good coin scam needs to sound a lot less dated.

sockaddr 5y ago

Nice. I'd pay 5-10$/mo for a search engine that didn't just funnel me into the revenue-extracting regions of the web like Google does.

RhysU 5y ago
A subscriber-supported search engine sounds cool to me. Any precedent?
- xtracto 5y ago
  Copernic ( https://copernic.com/ https://copernic.com/ ) had Copernic Agent Professional, a for-pay desktop application that had really good search features, a while ago . Not sure if they discontinued it.
  - gompertz 5y ago
    Wow blast from the past. I think I was using Copernic all the way back in 2003... Forgot all about them. Thanks!
- gianthockey495 5y ago
  You'll like https://neeva.com/ https://neeva.com/
  - fxtentacle 5y ago
    How do they pay for it?
    - yellow_postit 5y ago
      From the FAQ: > …Eventually, we plan to charge our members $4.95/month.
- samcrawford 5y ago
  Kagi.com does this. In closed beta at the moment, but you can email and request access.
  - eitland 5y ago
    I've tested Kagi a bit. It nicely gave me exactly what I wanted even in cases where names could have different meanings in different contexts (I tested with Kotlin) The basic results are good with some nice touches here and there like including a "blast from the past" section with older results which is actually what I sometimes want and another section where it widens up a bit (i.e. what Google does by default?). Furthermore you can apply pre defined search "lenses" that focuses your search, or even make your own, and you can boost or de-rank sites. I had not expected this to happen so quickly but I'm going to move from DDG to kagi as my default search engine for at least a couple of days because I am fed up with both Googles and DDGs inability to actually respect my queries. If ir continues to work as well as it does today I'll happily pay $10 a month and I might also buy 6 months gift cards for close friends and family for next Christmas. Think about it, unlike an ad financed engine incentives are extremely closely aligned here: the smartest thing Kagi can do is to get my results as fast as possible to conserve server resources (and delight their customer). For an ad financed engine abd especially one that also serves ads on search results pages as well the obvious thing to do is to keep me bouncing between tweaking my search query and various that almost has my question answered but not quite. (That said, if one us going to stay mainstream I recommend DDG over Google since 1. for me at least Googles results are just as bad and 2. with DDG it is at least extremely easy to check with Google as well to see if they have a better result 3. competition is good)
- KarlKemp 5y ago
  As a general rule, nobody is willing to pay what they are worth to advertisers. Facebook makes 70$ / y / user in the US. You would pay $70 for an ad-free Facebook? Congratulation, you must be an above-average earner. Also: your value to advertisers just tripled. If you are willing to pay $210, it will immediately triple again.
  - derekjdanserl 5y ago
    Great point! So simple, but as someone who has never worked on this side of things I never thought about it. How would legal limitations on data collection, like GDPR, influence the ratio? None? Only an insignificant degree? Or enough to actually influence business decisions?

Archelaos 5y ago

I tried out four search words with your search engine, and I am not convinced that it is mainly the index size and not the algorithm that is to blame for bad search results. There are way too much high ranking false positives. Here is what I tried: a) "Berlin": 1. The movie festival "Berlinale" 2. The Wikipedia entry about Berlin 3. Something about a venue "Little Berlin", but the link resolves to an online gaming site from Singapure 4. "Visit Berlin", the official tourism site of Berlin 5. The hash tag "#Berlin" on Twitter 6. "1011 Now" a local news site for Lincoln, Nebraska 7. "Freie Universität Berlin" 8. Some random "Berlin" videos on Youtube 9. The Berlin Declaration of the Open Access Initiative 10. Some random "Berlin" entries on IMDb 11. A "Berlin" Nightclub from Chicago 12. Some random "Berlin" books on Amazon 13. The town of Berlin, Maryland 14. Some random "Berlin" entries on Facebook 15. The BMW Berlin Marathon b) "philosophy" 1. The Wikipedia entry about philosophy 2. "Skin Care, Fragrances, and Bath & Body Gifts" from philosophy.com 3. "Unconditional Love Shampoo, Bath & Shower Gel" from philosophy.com 4. Definition of Philosophy at Dictionary.com 5. The Stanford Encyclopedia of Philosophy 6. PhilPapers, an index and bibliography of philosophy 7. The University of Science and Philosophy, a rather insignificant institution that happens to use the domain philosophy.org 8. "What Can I Do With This Major?" section about philosophy 9. Pages on "philosophy" from "Psychology Today". I looked at the first and found it to be too short and eclectic to be useful. 10. The Department of philosophy of Tufts University c) "history" 1. Some random pages from history.com 2. "Watch Full Episodes of Your Favorite Shows" from history.com 3. Some random pages from history.org 4. "Battle of Bunker Hill begins" from history.com 5. Some random "History" pages from bbc.co.uk 6. Some random pages from historyplace.com 7. The hash tag "#history" on Twitter 8. The Missouri Historical Society (mohistory.com) 9. Some random pages from History Channel 10. Some random pages from the U.S. Census Bureau (www.census.gov/history/) d) "Caesar" 1. The Wikipedia entry about Caesar 2. Little Caesars Pizza 3. "CAESAR", a source for body measurement data. But the link is dead and resolves to SAE International, a professional association for engineering 4. The Caesar Stiftung, a neuroethology institute 5. Some random "Caesar" books on Amazon 6. Hotels and Casinos of a Caesars group 7. A very short bio of Julius Ceasar on livius.org 8. Texts on and from Caesar provided by a University of Chicago scholar 9. (Extremely short) articles related to Caesar from britannica.com 10. "Syria: Stories Behind Photos of Killed Detainees | Human Rights Watch". The photos were by an organization called the Caesar Files Group So what I can see are some high ranked false positives that are somehow using the search term, but not in its basic meaning (a3, a11, b2, b3, d2, d3, d4, d6) or not even that (a6). Some results are ranking prominently although they are of minor importance for the (general) search term (a9, a13, b7, b8 -- perhaps a15 and d10). Then there are the links to the usual suspects such as Wikipedia, Twitter, Amazon, etc. (a2, a5, a8, a10, a12, a14, b7, c5, d1, d5); I understand that Wikipedia articles are featuring prominently, but for the others I would rather go directly to eg. Amazon when I am interested in finding a book (or use a search term like "Caesar amazon" or "Caesar books"). Well, and then there are the search results that are not completely off, but either contain almost no information, at least compared to the corresponding Wikipedia article and its summary (b4, b9, d7, d9), or that are too specific for the general search term (c1, c2, c3, c4, c6, c9, c10). That leaves me with the following more or less high quality results (outside of the Wikipedia pages): a1, a4, a7, b5, b6, b10, and d8. The a15 and d10 results I could tolerate if there had been more high quality results in front of them; but as a fourth and second, respectively, good result they seem to me to be too prominent. Also in the case of "Berlin" a4 should have been more prominent than a1, and a7 is somewhat arbitrary, because Humbolt University and the Technical University of Berlin are likewise important; what is completely missing is the official Website of the city of Berlin (English version at www.berlin.de/en/). All in all, I would say that your ranking algorithm lacks semantic context. It seems the prominence of an entry is mainly determined by either just being from the big players like Twitter, Youtube, Amazon, Facebook, etc. or by the search term appearing in the domain name or the path of the resource, regardless of the quality of the content.

dgivney 5y ago
I think you have some great feedback here but for me it also highlights how subjective search results can be for individuals - for example, these false positives that you mention (b2, b3) appear as the top result on Google for me for that query. It makes me think there must be some fairly large segment of the population that want that domain returned as a result for their query, no?
- Archelaos 5y ago
  I would not deny that a large part of subjectivity is involved. This is why I used several markers of subjectivity in my evaluation ("what I can see", "that leaves me", "they seem to me", "I would say", etc.). And related to that: I also agree with other responses that a search often needs to be refined. So my four examples where in no way an exhaustive evaluation, but an explorative experiment, where I just used two proper names, one for a city and one for a historical person, and two general disciplines as search words, in order to see what happens and what is noteworthy (to me). So much to the subjective side. But what can be said about ideal search results for these terms beyond subjectivity? I do not think that we can arrive at an objective search result, but are nevertheless allowed to criticise search results with respect to their (hidden or obvious) agenda. Let me give an example of the good old days: When I was searching for my surename on Google in the early 2000s the search results contained a lot of university papers or personal Web-sites (then called "homepages") from other people of that name. But suddenly, I can't remember when exactly this was, the search results contained almost exlusively companies that contained that surename in its company name. The shift was not gradually, as if it were representing a slow shift in the contents of the Internet itself, but abrupt. It was apparently due to an intentional modification of the ranking algorithm that put business far above anything else on the Internet. My explanation for this is the following: The objective metrics for Google search results is the stream of revenue they generate for Google. But not only for Google. The fundamental monetary incentive to Bing (and its derivative Ecosia) is more or less the same. And how different the impact of the somewhat different business model of Duck Duck Go is, is open for debate. If maximum revenue is the goal, the aim is to provide the best search results according to the business model (advertisment, market research and whatever else) without driving the users away. But the best search results according to the business model are not necessarily the optimal search results for the typical user. And as long as all relevant competitors are following the same economic pressure of maximizing revenue the basic situation an thus the qualitiy of the search results for the user will not improve above a certain level. If we want this situation to change, we need competitors with a different, non-commerical agenda. Either from the public sector (an analogon to the excellent information services about physical books provided by libraries) or from non-profit organizations (an analogon to Wikipedia or Open Street Map). To answer your question about b2 and b3: I checked with other search engines; besides Google they appear for me also on Bing (as #8, same product but on a different Web-site) and Duck Duck Go (as #10); Bing also has a reference to them in the right margin as a suggestion for a refined search (this time exactly b2 and b3). Although I do not think that the results from those search engines should be considered as a general benchmark for good search results for the reasons given above, we may speculate why they appear on the first page of search results. I would guess that it is a combination of gaming the search engines by using a generic term as a product and domain name to get free advertising, and search engine algorithms making this possible by generally ranking products and companies high in their search results.

ma2rten 5y ago

It's much more expensive now to build a large index (50B+ pages) Do you have a cost estimate? Also could you be more selective in indexing, e.g. by having users requests sites to be crawled.

ampersandy 5y ago
Requiring users to know what sites they want in advance somewhat defeats the purpose of a search engine, no?
- robbomacrae 5y ago
  Not at all. You only have to fail the first request. It is an approach I took with my own attempt at a search engine way back! In fact I know personally that there is at least one patent out there that suggests initial 1st time request users being asked to provide the appropriate response as an efficient way to teach systems for future users. Obviously failing first requests isn't ideal but for popular requests it quickly becomes insignificant. Wikipedia might (if they don't already) want to make a similar suggestion for users to contribute when finding a low content/missing page.
  - lowwave 5y ago
    > Obviously failing first requests isn't ideal but for popular requests it quickly becomes insignificant. The first request can also be called asynchronously, and display a message to the user that it is 'processing....'.
- convolvatron 5y ago
  since sites are so desperate to be indexed, doesn't it seem better to put the onus on them to announce themselves? it would be great if dns registries publshed public keys .. maybe they do in newer schemes?
  - ma2rten 5y ago
    That works once your search engine is more widely used, but not a lot of sites are going to register with a niche search engines. Many users on the other hand really want a search engine like this and would be willing to invest some time.
  - fragmede 5y ago
    Certificate Transparency (CT) Logs are this.
- ma2rten 5y ago
  More often than not I have an idea which site a result might be on when I issue a query: If I search for a news event it's a news site. If I search an error message, I know the result is going to likely be stackoverflow, github issues or the forum of the library. etc. I don't think this strategy will get you all the way there, but it could be combined with other ways of curating sites to crawl.

SamBam 5y ago

> You do a search on Gigablast and say, well, why didn't it get this result that Google got. And that's because the index isn't big enough I wionder how much this is true, and how much (despite all our rhetoric to the contrary) it's because we have actually come to expect Google's modern proprietary page ranking, which counts more than just inbound links but all sorts of other signals (freshness, relevance to our previous queries, etc.). We dislike the additional signals when it feels like Google is trying to second-guess our intentions, but we probably don't notice how well they work when they give us the result we expect in the first three links.

1vuio0pswjnm7 5y ago

I really like GigaBlast. I wrote a "meta" search utility for myself that can query multiple search engines from the command line.^1 It mixes the results into a simplified SERP ("metaSERP"), optimised for a text-only browser, with indicators to show which search engine each result came from. The key feature is that it allows for what I might call "continuation searches". Each metaSERPs contains timestamps in its page source to indicate when searches were executed, as well as preformatted HTTP paths. The next search can thus pick up where the previous one left off. Thus I can, if desired, build a maximum-sized metaSERP for each query. The reason I wrote this is because search engines (not GigaBlast) funded by ads are increasingly trying to keep users on page one, where the "top ads" are, and they want to keep the number of results small. That's one change from 2005 and earlier. With AltaVista I used to dig deep into SERPs and there was a feeling of comprehensiveness; leave no stone unturned. Google has gradually ruined the ability to perform this type of searching with their now secretive and obviously biased behind-the-scenes ranking procedures. Why is there no way to re-order results according to objective criteria, e.g., alphabetical; the user must accept the search engines' ordering, giving them the ability to "hide" results on pages the user will never view or simply not return them. That design is more favorable to advertising and less favorable to intellectual curiosity. Each metaSERP, OTOH, is a file and is saved in a search directory for future reference; I will often go back to previous queries. I can later add more results to a metaSERP if desired. I actually like that GigaBlast's results are different than other search engines. The variety of results I get from different sources arguably improves the quality of the metaSERP. And, of course, metSERPs can be sorted according to objective criteria. This is, AFAIK, a different way of searching. The "meta-search engines" of yesteryear did not do "continuations", probably because it was not necessary. Nor was there en expectation that user would want to save meta-searches to local files. Users were not trying to minimise their usage of a website, they were not trying to "un-google". Today's world of web search is different, IMO. There seems to be a belief that the operator of a search engine can guess what a user is searching for, that a user who sends a query is only searching for one specific thing, and that the website has an ad to match with that query. At least, those are the only searches that really matter for advertising purposes. Serendipitous discovery while perusing results is not contemplated in the design. By serendipitous discovery I do not mean sending a random query, e.g., adding an "I'm feeling lucky" button, which to me always seemed like a bad joke. The only downside so far is I ocassionally have to prune "one-off" searches that I do not want to save from the search directory. I am going to add an indicator at search time that a search is to be considered "ephemeral" and not meant to be saved. Periodically these ephemeral searches can then be pruned from the search directory automatically. 1. Of course this is not limited to web search engines. I also include various individual site search engines, e.g., Github.

betwixthewires 5y ago
Wow, do you happen to have published your utility so that other people can play with it?
- 1vuio0pswjnm7 5y ago
  The problem is that (1) I am a minimalist and dislike lots of "features" and (2) I prefer extremely simple HTML that targets the links browser. Most users are probably using graphical, Javascript- and CSS-enabled browsers so while this may work great for me, it may be of little interest to others who have higher aesthetic expectations. Another problem is I prefer to write tiny shell scripts and small programs in C that can be used in such scripts. To be interesting to a wider audience, I would likely have to be re-write this in some popular language I do not care for. If I see people on HN complain about how few results they get from search engines, then that could provide some motivation to publish. I am just not sure this is a problem for others besides me. Many results I get from search engines are garbage. By creating a metaSERP with a much higher number of results overall, from a variety of sources, I believe I get a higher number of quality ones.
  - betwixthewires 5y ago
    Well something like that would be interesting to a particular demographic. I prefer minimal aesthetic cruft as well, and like terminal stuff like links. If you ever do decide to publish, be sure to post it here!

gbmatt 5y ago

both ddg and brave are bing (microsoft) in disguise.

pythux 5y ago
This is not correct. Brave Search owns its own (growing) index and relies on third-parties like Bing for some fraction of the requests. Which is not the same thing as relying fully on Bing or third-parties for results like so many meta-search engines. More detailed answer here: https://search.brave.com/help/independence https://search.brave.com/help/independence Edit: Forgot to say that I work on Brave Search.
- gbmatt 5y ago
  brave 'falls back' to bing. which in my experience is most of the time. in fact, out of all the queries i did a while back, they all seemed to come directly from bing. is there a way to disable the reliance on bing and get pure 'brave only' results? and can you be more specific as to what this fraction is? do you blend at all?
  - BrendanEich 5y ago
    What independence percentage do you see when you click on the gear in upper right of the Brave Search results page? I get 84% personal (browser-based), 87% global (which means we hit Bing only 13% of the time from our server side).
  - pythux 5y ago
    You can check exactly which fraction of the results were fetched from Brave's index vs. third-parties using the "independence score" found in the setting drawers (opening can be done with the cog icon at the top right of any page on search.brave.com). There is there a global and personalized score of independence (respectively aggregated on all user's and for your queries only). Explanation is also found here with screenshots: https://search.brave.com/help/independence https://search.brave.com/help/independence
    - gbmatt 5y ago
      So Brave is still dependent on Google and Bing it seems. Also is this Brave's CEO: https://www.bbc.com/news/technology-26868536 https://www.bbc.com/news/technology-26868536 https://www.nytimes.com/2020/12/22/business/brave-brendan-eich-covid-19.html https://www.nytimes.com/2020/12/22/business/brave-brendan-ei... ? "Brendan Eich's opposition to same-sex marriage cost him his job at Mozilla." "Covid comments get a tech C.E.O. in hot water, again."

lessname 5y ago

This might work well in some situations (e.g. research, development), however it would also increase the effect of echo chambers I think.

lolinder 5y ago
Possibly, but I'm not convinced. Google's not exactly working against the echo chamber problem, and I think that's because to do so would be to work against its own reason for existing. There are two goals here that are fundamentally at odds with each other: 1) Finding what you're looking for. 2) Finding a new perspective on something. A search engine's job is to address the first challenge: finding something that the user is looking for. The search engine might end up serving both needs if they're looking for a new perspective on something, but if these two goals ever come into conflict with each other the search engine does (and I would argue it should) choose the first goal. Failing to do so will just lead to people ignoring the results.
theduder99 5y ago
echo chambers are what most people want :)
- dougSF70 5y ago
  echo chambers are what most people want :)
jonathankoren 5y ago
Part of the thing with echo chambers is that the search terms themselves can be indicative of a particular bubble. For example, there's a difference in the people that refer to the Bureau of Alcohol, Tobacco, and Firearms by the official initialism, "ATF", and those that use "BATF". There's a strong antigun control bent to the `"BATF" guns` query, compared to the `"ATF" guns` query. If you're indexing forums or social media, the same site is going to give back the bubbled responses, possibly without the person even being aware they're in a bubble. https://www.google.com/search?q=%22BATF%22+guns&client=safari&rls=en&sxsrf=AOaemvJS_Q3Nc1e40-VcQYu2xvyRTabBBQ:1638485361910&ei=cU2pYd3vNo65qtsPnZyYkA4&start=10&sa=N&ved=2ahUKEwid7NzPmcb0AhWOnGoFHR0OBuIQ8tMDegQIARA8&biw=967&bih=756&dpr=2&safari_group=9 https://www.google.com/search?q=%22BATF%22+guns&client=safar... https://www.google.com/search?q=%22ATF%22+guns&client=safari&rls=en&ei=z02pYZb0GY_PtQaTvrnADA&start=10&sa=N&ved=2ahUKEwjWl6n8mcb0AhWPZ80KHRNfDsgQ8NMDegQIARBJ&biw=1280&bih=970&dpr=2 https://www.google.com/search?q=%22ATF%22+guns&client=safari...
- GuB-42 5y ago
  Kind of like when searching for "jew" on Google led to antisemitic websites, that's because jews usually prefer the term "jewish". Interestingly, back then, Google was big on neutrality and refused to do anything, stating that it reflected the way people used the word. It was finally addressed using "Google bombing" techniques. Something that Google didn't care much about back them because of its low impact.

Terretta 5y ago
You inspired me to try an even less specific search: thing Subjectively felt the gigablast results were a relative delight.
- Archelaos 5y ago
  No bad idea. At the risk of being sidelined: "philosophy" was not so a bad term either. Start with an arbitrary Wikipedia link and click on the first keyword of the summary after the linguistic annotations (or other annotations in brackets) and repeat the process until you reach a loop. You will almost always end with "philosophy" -> "metaphysics" -> "philosophy" -> ... This works for "Berlin", "history" and "Caesar" as well as for "thing". For the latter very fast: "thing" -> "object" -> "philosophy".
  - gbmatt 5y ago
    that's tripped out. where did you hear about that?
    - Archelaos 5y ago
      I can't remember. Probably on Hacker News.
dgivney 5y ago
Oh of course you can criticise the result, I more found it interesting that a billion dollar, optimized search experience thought your false positive was actually a top result. A huge variance in the subjectivity between your experience and their invested reasoning. But while we're speculating on how the domain the appears at the top of the list, let me hazard a guess... Philosophy.com was registered in 1999 and according to waybackmachine, has been selling cosmetics on the site since 2000 (20+ years). The company sold in 2010 for ~$1B to a holding company with revenues of $10B+ today (Unfortunately I couldn't find how much it contributes to that revenue). According to Wikipedia, the Philosophy brand has been endorsed by celebrities, including "long-time endorser" Oprah Winfrey, possibly the biggest endorsement you could get for their industry/demographic. I think it is a long established business, with strong revenues and there's more people online searching for cosmetic brands than for philosophers. In the same way (admittedly in the extreme) when I'm researching deforestation and I query to see how things are going for the 'amazon', the top result is another successful company registered pre 2000, with strong revenues that most likely attracts more visitors..
- Archelaos 5y ago
  Okay, you convinced me that it should (inter-subjectivly) count not as a real false positive as I first thought. Nevertheless, when I try to analyze what is going on here, I would rather use the word "context" instead of "subjectivity", since I think (or at least hope) that my surprise to find this brand on place #2 in my Google results for "philosophy" is shared by quite a lot of people who lack the context to give it meaning, because this brand is unknown to them. I have the excuse that it is a North American brand irrelevant in my German context. Interestingly, when I search for "philosophy" on amazon.com (without refining the search), I get almost exclusively beauty products and related items as a result, but when I search for "philosophy" on amazon.de it is only books. Google nevertheless has the beauty brand as #2 in Germany. Can we agree that Amazon is better at considering the context of the search for "philosophy" than Google? As an aside: Your "amazon" example reminds me when I was searching for "Davidson" expecting to find information about Donald Davidson, but received a lot of results about Harley-Davidson. (But since I was aware of the importance of this brand, it was understandable to me.)
  - dgivney 5y ago
    We can agree on that, yes =) I was thinking about this and when you look at the top keyword searches on Google, it's dominated by people searching brands each year, so I think Google is just naturally optimised for this. I think any Search Engine designed for the masses would probably have to behave like this too. https://www.siegemedia.com/seo/most-popular-keywords https://www.siegemedia.com/seo/most-popular-keywords I agree, I think the early web was used more for general information rather than specific brand information (and was more useful for people like myself). I'm not sure what is needed to get more results such as university papers or personal web-sites as I think that people use the internet differently now and that the link structure reflects that. It's interesting that Google isn't used to search for people anymore (I couldn't see any people in the recent top 100 keyword search data).
    - Archelaos 5y ago
      Some observations: Most of the "brands" in the top 100, especially at the beginning, are rather Internet services. These search terms seem to have been entered not with the intention to "search" in the sense to find some new information, but as a substitute for a bookmark to the respectice service. Who searches for #1 "youtube" does not want information about youtube, but wants to use the youtube Web-site as a portal to find videos there. I would also guess that most of these searches haven't been initiated through the Google Web-site, but directly from the browser's adress/search bar or a smartphone app. They exhibit a specific usage pattern, but do not show what the people, that entered them, were really searching for, if they were searching at all. What are those people who search for "youtube" doing next: either search again on youtube or log into their youtube account and browse their youtube bookmarks. The early Internet did not have so many different service people used at a daily basis, and those that existed were more diverse (think of the many differen online email providers in those days) so that the search terms spread out more. Also browsers had no direct integration with a search engine. The incentive was higher to use bookmarks for your favourite service, since otherwise you had to use a boomark to a search engine anyway. Perhaps it would be more approbriate to compare the use of the early Google not with the current Google, but the current Google Scholar?

kilburn 5y ago

I don't know about others, but when I think of the "good old google days" I'm _not_ expecting the results for your example queries to be any good. In those days querying took some effort but the effort paid off. The results for "history" just couldn't matter less in this mindset. You search for "USA history" or "house commons history" or "lake whatever history" instead. If the results come up with unexpected things mixed in, you refine the query. It was almost like a dialog. As a user, you brought in some context. The engine showed you its results, with a healthy mix of examples of everything it thought was in scope. Then you narrowed the scope by adding keywords (or forcing keywords out). Rinse and repeat. As a user, you were in command and the results reflected that. The idea that the engine should "understand what you mean" is what took us to the current state. Now it feels like queries don't matter anymore. Google thinks it knows the semantics better than you, and steering it off its chosen path is sometimes obnoxiously hard.

1024core 5y ago
> The idea that the engine should "understand what you mean" is what took us to the current state. Now it feels like queries don't matter anymore. Google thinks it knows the semantics better than you, and steering it off its chosen path is sometimes obnoxiously hard. Bingo! If you cede control to Google, it _will_ do what it's optimized to do, and not what _you_ are looking for.
- melenaboija 5y ago
  What is optimized to do says nothing. Optimizing for open text queries means dealing with a massive search space, the thing is choosing a subspace where to search and that is the part engines have to refine, how that is done is a different story. Some people may agree to use their location, search history and visits to online stores to do so but some may not.
yellowstuff 5y ago
I get what you mean, but part of the whole initial appeal of Google was that it gave much more relevant results initially than Altavista or the other options. That was why Google put in the audacious "I'm feeling lucky" button.
- 1_player 5y ago
  Yeah but it's from that same philosophy that Google Search is useless as it optimises for the first result. There is no search engine that searches literally for what you asked and nothing else. Search is shit in 2021 because it tries to be too clever. I'm more clever than it, let me do the refining.
- DarkmSparks 5y ago
  This was the result of two things. mapreduce. using links rank the pages. Using links to rank the pages is not really possible any longer because of seo spam links.
- BbzzbB 5y ago
  >"I'm feeling lucky" button My brain got so used to ignoring it I completely forgot it's a thing. I'm also unclear what it does? On an empty request, it gets me to their doodles page and with text in the box, gets me to my account history landing page.
  - Too 5y ago
    It automatically redirects to the first search result.
    - BbzzbB 5y ago
      Right, not sure why it wasn't working yesterday as opposed to now, I swear I wasn't doing it wrong (or how I could've).
Archelaos 5y ago
This is why, in the good old days, my favourite search engine was Alta Vista. In its left margin it had arranged key words like a directory tree that could be used to further refine the search. So my ideal search should do something like this if I type in a generic term: provide me with relevant information about the general topic and than help me to refine my search. The way of Wikipedia to provide a principal article and a structured disambiguation page is the way I would prefer. I admit, my evaluation of the search engine was just a simple test how much I can get out of the results for some generic key words in the first place. A more detailed evaluation should, of course, look deeper. It was more of a test balloon to see if this search engine raises any hope that it could be better than Google with regard to my own (subjective) expectations of a decent result set.