7 ms·

Google was still useful for things that are non-marketable. But in the last year or so, I have started to realize how awfully bad has gotten even at that. Case

Google was still useful for things that are non-marketable. But in the last year or so, I have started to realize how awfully bad has gotten even at that. Case in point: Open source mailing list archives. For various reasons, some times I have to search for entire email threads that are available in the web interfaces of some mailing list archives. I have part of a phrase of some email, I put that into google with quotes, and it returns some results, usually with the main archive being in the first results. In the last year or so, Google has started returning no results for some phrases of emails that do exist and are available in one or more public archives. When that started to happen, I tried Bing. And Bing returns results with links to archive. So does duckduckgo most times. But for Google, it's like if that particular email I'm searching at that moment didn't exist (for other emails in the same mailing list it works fine). It only happens occasionally, but it's getting worse. So I have started to rely on DDG and Bing more and more, because they always find what I want. So Google is starting to fail at some of the most basic aspects of a search engine, it's not just the ordering of the search results - there are some public web pages that it won't see for whatever reason.

betwixthewires 5y ago
One thing I learned from the unreliability of Google search is that we should not rely on an external service to organize our information. No matter the service, it will change in some way, and you will lose access to some information.
- ReactiveJelly 5y ago
  Agreed. "Indexing the world's information" sounds like something that can't go wrong. It's only in the last year of trying to index and organize information within my own company that whoever controls the index holds a _lot_ of power to index it poorly.
- SpelingBeeChamp 5y ago
  What do you suggest as an alternative? Index the internet yourself? Web search as a public utility?
  - betwixthewires 5y ago
    Well there are semantic web concepts, and there are concepts like a searchable information graph. I read a really good article on this site a few days ago laying out a pretty good architecture for searching a distributed knowledge graph. The core problem is a disconnect between the knowledge available on the internet and search tools. Indexing is a stop gap to enable the web to be usable without solving this problem, but it is becoming untenable and was always known to be non viable, just treated as a problem for another day. The sheer amount of information online and its disorganization, as well as the misaligned incentives of search tools are beginning to make this once far away manageable problem a day to day reality for all of us, the solution requires an architectural change to the way information is stored and accessed and updated online.
    - SpelingBeeChamp 5y ago
      Do you have any idea of what that solution might look like? The existence of a problem doesn't mean that there necessarily exists a better alternative.
- visarga 5y ago
  Why are browsers so forgetful? Local text search is an easy problem at single-user scale. I am working on a browser extension to automatically grab text (visible phrases) from the pages I am reading in the browser, save them to my index and implement search. I want to always be able to recall anything I see. In order to match texts I will use neural embeddings computed with a pretrained Huggingface model, fast enough even on CPU. An alternative way to implement this would be to take screenshots every few seconds (only after recent user interaction). Then I can OCR the images and search anything that passed on the screen, in any app. It's as if I become the Google Bot responsible with exploration and filtering of potentially useful data. In the future I might be able to use my index for neural question answering, add a voice interface to my data. Of course the local index could work for images too. You can grab images and embed them with CLIP to get Google Photos like semantic search. I think web browsers are going to morph into personal internet agents, responsible with logging, searching, interfacing with external APIs and services, filtering ads, news and mail, making sure we don't accidentally leak personal information, etc. In other words AI that works for the user, locally.
  - Jonovono 5y ago
    Hey, can you email me (url in bio), would love to chat!
  - hutzlibu 5y ago
    "I think web browsers are going to morph into personal internet agents, responsible with logging, searching, interfacing with external APIs and services, filtering ads, news and mail, making sure we don't accidentally leak personal information" And you think that will happen, while the browsers are still mostly add funded? But I am with you, that this is a change, I want to see, too.
    - skinkestek 5y ago
      There is an extension for Firefox (Memex Go?) that attempts something very similar and succeeds to a large degree. I think I sponsored it for a year or something by paying for it even though I didn't use it much but I must have used it to some degree because on one or more of my machines it injects interesting results from my earlier browsing when I search in DDG (or was it Google?) at least. I'm really impressed how far they've taken this idea even as Mozilla has simultaneously been busy dismantling the extension API that made Firefox great.
noizejoy 5y ago
I for one am grateful that some of my worst mailing list accidents (like accidentally sending a privately intended message to a mailing list, rather then an individual email address) are starting to disappear from Google’s relentless claws.
hubraumhugo 5y ago
Curious to hear: what are some rising Google competitors that try to do a better job?
- XoS-490 5y ago
  https://you.com https://you.com
  - hubraumhugo 5y ago
    https://neeva.com https://neeva.com looks similar
    - everybodyknows 5y ago
      > 6. Fees. The Services are currently offered by Neeva free of charge. Neeva reserves the right to charge fees for the Services in the future. https://neeva.com/terms https://neeva.com/terms Sustainability TBD, but they do have a plan.
  - everybodyknows 5y ago
    > We’re in beta and feel like our product is not yet fully realized. Sustainability remains TBD.
- JSONderulo 5y ago
  I've been playing around with you.com after someone mentioned on HN. I dig the layout/quality of results. It's kind of interesting to have it feel like you're netflixing your search results by source (reddit, stack overflow, etc.) I'm still toggling back and forth on you to google because i feel like i might miss out something. Surprisingly some the results have exceeded my expectations. We'll see.
  - ffhhj 5y ago
    The side scrolling is nice, but when searching some code related question it doesn't show useful results. Even the Code Complete snippets aren't displayed in a readable way, lines are too short, and the language highlight is not recognized (every script is python by default?). Thanks for mentioning it since I'm creating a snippet search engine and wanted some competitors to compare.
srcreigh 5y ago
Blacklisting non advertisable media from the index. That's news to me. Hmm. Why? Cost savings? Missing hard dependency on marketing ops knowhow? Utter domination of non-marketing material? Forcing experienced privacy-conscious users to a different platform to reduce bad media coverage?
- dannyw 5y ago
  Monkey run experiment, revenue goes up, monkey get promotion.
  - eitland 5y ago
    At this point I cannot be alone in thinking that what we are seeing is a runaway AI system? Let me just speculate wildly here: Seems no-ones Google is the same and how do one manage approximately 4 billion [1] a/b/c/etc experiments if not using artificial intelligence and reinforcement? Now imagine 4 billion agents ("monkeys") who flip bits everywhere and every time one of them is unusually successful that pattern is reinforced across a part of the "population" of agents. Every time one of them does something particularly stupid Google relies on users to report an error and swaps out that agent. (This explains why if you report an error in search it appears to magically be fixed 15 minutes later.) Let's say this has been running since way before 2008 and 2008 just happened to be the time it got completely out of hand: Mostly it works very well economically speaking but search quality suffers year by year and no one can do anything about it since by now so many optimizations have been applied that for a decade no single human has been able to really understand it. The AI - using end to end optimization techniques - learns to prioritize ad-heavy sites (since thise are more likely to result in positive reinforcement in the form of juicy ad impressions or clicks) and sites that doesn't really answer the question (since that brings the user back to Google for a new chance to serve ads both in search results and on other high ranking pages.) In the same manner the system learns to avoid easily understood clean pages without ads as that means game over for this query. By now everyone who knows anything about it sees that it broken but nobody dares to fix it since that would result in a massive revenue loss. [1]: assuming half the world uses Google
nullc 5y ago
Likewise, I constantly find that google won't turn up mailing lists posts and forum posts even when I have a fragment to search for exactly. Some investigation seemed to suggest they were hard excluding any page which hadn't changed in ~5 years. I have to wonder if the poor behavior from google doesn't come from automatically a/b testing behavior changes against advertising revenue. Up to the point that you make users give up completely giving worse results will result in more search traffic...
- r721 5y ago
  >Some investigation seemed to suggest they were hard excluding any page which hadn't changed in ~5 years. I had another hypothesis: that they exclude pages which have no backlinks from other websites, and at the same time are hard to reach from front page of the website they are on. It seems old unpopular mailing lists posts and forum posts are exactly like that.

betwixthewires 5y ago

One thing I learned from the unreliability of Google search is that we should not rely on an external service to organize our information. No matter the service, it will change in some way, and you will lose access to some information.

ReactiveJelly 5y ago
Agreed. "Indexing the world's information" sounds like something that can't go wrong. It's only in the last year of trying to index and organize information within my own company that whoever controls the index holds a _lot_ of power to index it poorly.
SpelingBeeChamp 5y ago
What do you suggest as an alternative? Index the internet yourself? Web search as a public utility?
- betwixthewires 5y ago
  Well there are semantic web concepts, and there are concepts like a searchable information graph. I read a really good article on this site a few days ago laying out a pretty good architecture for searching a distributed knowledge graph. The core problem is a disconnect between the knowledge available on the internet and search tools. Indexing is a stop gap to enable the web to be usable without solving this problem, but it is becoming untenable and was always known to be non viable, just treated as a problem for another day. The sheer amount of information online and its disorganization, as well as the misaligned incentives of search tools are beginning to make this once far away manageable problem a day to day reality for all of us, the solution requires an architectural change to the way information is stored and accessed and updated online.
  - SpelingBeeChamp 5y ago
    Do you have any idea of what that solution might look like? The existence of a problem doesn't mean that there necessarily exists a better alternative.
visarga 5y ago
Why are browsers so forgetful? Local text search is an easy problem at single-user scale. I am working on a browser extension to automatically grab text (visible phrases) from the pages I am reading in the browser, save them to my index and implement search. I want to always be able to recall anything I see. In order to match texts I will use neural embeddings computed with a pretrained Huggingface model, fast enough even on CPU. An alternative way to implement this would be to take screenshots every few seconds (only after recent user interaction). Then I can OCR the images and search anything that passed on the screen, in any app. It's as if I become the Google Bot responsible with exploration and filtering of potentially useful data. In the future I might be able to use my index for neural question answering, add a voice interface to my data. Of course the local index could work for images too. You can grab images and embed them with CLIP to get Google Photos like semantic search. I think web browsers are going to morph into personal internet agents, responsible with logging, searching, interfacing with external APIs and services, filtering ads, news and mail, making sure we don't accidentally leak personal information, etc. In other words AI that works for the user, locally.
- Jonovono 5y ago
  Hey, can you email me (url in bio), would love to chat!
- hutzlibu 5y ago
  "I think web browsers are going to morph into personal internet agents, responsible with logging, searching, interfacing with external APIs and services, filtering ads, news and mail, making sure we don't accidentally leak personal information" And you think that will happen, while the browsers are still mostly add funded? But I am with you, that this is a change, I want to see, too.
  - skinkestek 5y ago
    There is an extension for Firefox (Memex Go?) that attempts something very similar and succeeds to a large degree. I think I sponsored it for a year or something by paying for it even though I didn't use it much but I must have used it to some degree because on one or more of my machines it injects interesting results from my earlier browsing when I search in DDG (or was it Google?) at least. I'm really impressed how far they've taken this idea even as Mozilla has simultaneously been busy dismantling the extension API that made Firefox great.

noizejoy 5y ago

I for one am grateful that some of my worst mailing list accidents (like accidentally sending a privately intended message to a mailing list, rather then an individual email address) are starting to disappear from Google’s relentless claws.

hubraumhugo 5y ago

Curious to hear: what are some rising Google competitors that try to do a better job?

XoS-490 5y ago
https://you.com https://you.com
- hubraumhugo 5y ago
  https://neeva.com https://neeva.com looks similar
  - everybodyknows 5y ago
    > 6. Fees. The Services are currently offered by Neeva free of charge. Neeva reserves the right to charge fees for the Services in the future. https://neeva.com/terms https://neeva.com/terms Sustainability TBD, but they do have a plan.
- everybodyknows 5y ago
  > We’re in beta and feel like our product is not yet fully realized. Sustainability remains TBD.
JSONderulo 5y ago
I've been playing around with you.com after someone mentioned on HN. I dig the layout/quality of results. It's kind of interesting to have it feel like you're netflixing your search results by source (reddit, stack overflow, etc.) I'm still toggling back and forth on you to google because i feel like i might miss out something. Surprisingly some the results have exceeded my expectations. We'll see.
- ffhhj 5y ago
  The side scrolling is nice, but when searching some code related question it doesn't show useful results. Even the Code Complete snippets aren't displayed in a readable way, lines are too short, and the language highlight is not recognized (every script is python by default?). Thanks for mentioning it since I'm creating a snippet search engine and wanted some competitors to compare.

srcreigh 5y ago

Blacklisting non advertisable media from the index. That's news to me. Hmm. Why? Cost savings? Missing hard dependency on marketing ops knowhow? Utter domination of non-marketing material? Forcing experienced privacy-conscious users to a different platform to reduce bad media coverage?

dannyw 5y ago
Monkey run experiment, revenue goes up, monkey get promotion.
- eitland 5y ago
  At this point I cannot be alone in thinking that what we are seeing is a runaway AI system? Let me just speculate wildly here: Seems no-ones Google is the same and how do one manage approximately 4 billion [1] a/b/c/etc experiments if not using artificial intelligence and reinforcement? Now imagine 4 billion agents ("monkeys") who flip bits everywhere and every time one of them is unusually successful that pattern is reinforced across a part of the "population" of agents. Every time one of them does something particularly stupid Google relies on users to report an error and swaps out that agent. (This explains why if you report an error in search it appears to magically be fixed 15 minutes later.) Let's say this has been running since way before 2008 and 2008 just happened to be the time it got completely out of hand: Mostly it works very well economically speaking but search quality suffers year by year and no one can do anything about it since by now so many optimizations have been applied that for a decade no single human has been able to really understand it. The AI - using end to end optimization techniques - learns to prioritize ad-heavy sites (since thise are more likely to result in positive reinforcement in the form of juicy ad impressions or clicks) and sites that doesn't really answer the question (since that brings the user back to Google for a new chance to serve ads both in search results and on other high ranking pages.) In the same manner the system learns to avoid easily understood clean pages without ads as that means game over for this query. By now everyone who knows anything about it sees that it broken but nobody dares to fix it since that would result in a massive revenue loss. [1]: assuming half the world uses Google

nullc 5y ago

Likewise, I constantly find that google won't turn up mailing lists posts and forum posts even when I have a fragment to search for exactly. Some investigation seemed to suggest they were hard excluding any page which hadn't changed in ~5 years. I have to wonder if the poor behavior from google doesn't come from automatically a/b testing behavior changes against advertising revenue. Up to the point that you make users give up completely giving worse results will result in more search traffic...

r721 5y ago
>Some investigation seemed to suggest they were hard excluding any page which hadn't changed in ~5 years. I had another hypothesis: that they exclude pages which have no backlinks from other websites, and at the same time are hard to reach from front page of the website they are on. It seems old unpopular mailing lists posts and forum posts are exactly like that.