4 ms·

Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the te

Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. There was a period of time where the ratio between indexing techniques and content worked, but we’re leaving that era now. All those things you hate about modern Google are necessary survival strategies for this model. AI search can’t solve that problem. What it can do is give up on the idea of surfacing specific sources in favor of synthesizing answers from slurry of all the content, finding (or hallucinating) signal in the noise. If you want strict keywords and boolean operators, you can’t expect to find that from the universal search engines that you’re used to. You want to look for and support curated engines that aggressively restrict what they index so that it contains high signal and low noise. Internal walled garden searches do this. Marginalia does this. Others probably do too. That’s what you’re looking for.

JohnFen 4y ago
> All those things you hate about modern Google are necessary survival strategies for this model. The reason I doubt this is because right now, I can find what I'm looking for through keyword searches on search engines that allow that, where I very often can't through the likes of Google.
- lll-o-lll 4y ago
  What search engines allow this?
  - JohnFen 4y ago
    Very few, but the closest approximation that I can find right now is DDG. Kagi might fulfill this role better. I haven't investigated it yet, though.
ravenstine 4y ago
Supposedly. The trouble is there's no way for me to know how true this noise theory is. I'm given no option by these search engines to "just show me all the crap." What search engines do today is so filtered and heuristical that there are apparently phrases and ideas that have never been expressed on the internet ever, or at least that's the impression I'd get if I was more naive. Besides, it's not like there isn't noise anyway. Ever click on the top results only to find that nothing matches your query and probably never did?
- allturtles 4y ago
  Do you really expect them to maintain an entire other version of the search product just to prove to skeptics that that version is not good?
  - ravenstine 4y ago
    No.
- Clent 4y ago
  Google is not a search engine. It is a curation service. It only searches this collection of data that it decided was worthy of curation. Google tells us this is for the best. We are forced to take their word for it since their algorithms are not public. Google tells us this is also for the best.
- pixl97 4y ago
  There is a way, just start spending millions/billions of dollars to index the internet... Just figure out how not to index the endless content generation scripts, and places like Reddit that are 100s of millions of pages themselves. Or the content farms that are just copies of other sites. Dead internet prophecy is coming true.
eitland 4y ago
> Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. Go check search.marginalia.nu. Do some sample searches about history or linux or git or something and see how it often outclasses Google. Then ponder the fact that it is run by one Swede and it runs on one (admittedly beefy but nothing unusual) workstation in his living room. Personally at least I have seen enough to have made up my mind that it isn't impossible to do better than Google, they just don't want to.
- swatcoder 4y ago
  Did you read my whole comment? I even mention Marginalia!
  - eitland 4y ago
    I clearly didn't read your whole post. Have my upvote. That said I think my point still stands: If 1 Swede and 1 server can do a much better job than Google in significant niches such as history, Linux and Git, then maybe a hundred Swedes and 10 000 servers can outcompete Google across the board for traditional search, leaving Google to deal with plane tickets and local search? ; - ) And no, I'm not a Swede.

JohnFen 4y ago

> All those things you hate about modern Google are necessary survival strategies for this model. The reason I doubt this is because right now, I can find what I'm looking for through keyword searches on search engines that allow that, where I very often can't through the likes of Google.

lll-o-lll 4y ago
What search engines allow this?
- JohnFen 4y ago
  Very few, but the closest approximation that I can find right now is DDG. Kagi might fulfill this role better. I haven't investigated it yet, though.

ravenstine 4y ago

Supposedly. The trouble is there's no way for me to know how true this noise theory is. I'm given no option by these search engines to "just show me all the crap." What search engines do today is so filtered and heuristical that there are apparently phrases and ideas that have never been expressed on the internet ever, or at least that's the impression I'd get if I was more naive. Besides, it's not like there isn't noise anyway. Ever click on the top results only to find that nothing matches your query and probably never did?

allturtles 4y ago
Do you really expect them to maintain an entire other version of the search product just to prove to skeptics that that version is not good?
- ravenstine 4y ago
  No.
Clent 4y ago
Google is not a search engine. It is a curation service. It only searches this collection of data that it decided was worthy of curation. Google tells us this is for the best. We are forced to take their word for it since their algorithms are not public. Google tells us this is also for the best.
pixl97 4y ago
There is a way, just start spending millions/billions of dollars to index the internet... Just figure out how not to index the endless content generation scripts, and places like Reddit that are 100s of millions of pages themselves. Or the content farms that are just copies of other sites. Dead internet prophecy is coming true.

eitland 4y ago

> Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. Go check search.marginalia.nu. Do some sample searches about history or linux or git or something and see how it often outclasses Google. Then ponder the fact that it is run by one Swede and it runs on one (admittedly beefy but nothing unusual) workstation in his living room. Personally at least I have seen enough to have made up my mind that it isn't impossible to do better than Google, they just don't want to.

swatcoder 4y ago
Did you read my whole comment? I even mention Marginalia!
- eitland 4y ago
  I clearly didn't read your whole post. Have my upvote. That said I think my point still stands: If 1 Swede and 1 server can do a much better job than Google in significant niches such as history, Linux and Git, then maybe a hundred Swedes and 10 000 servers can outcompete Google across the board for traditional search, leaving Google to deal with plane tickets and local search? ; - ) And no, I'm not a Swede.