7 ms·

I just want an effective boolean search. I don't want it automatically fixing type-os or including synonyms in results. If I remember a specific phrase, or mi

I just want an effective boolean search. I don't want it automatically fixing type-os or including synonyms in results. If I remember a specific phrase, or misspelling from some random blog, I want to type that in and find it. Google search has gotten significantly worse since they started trying to be a generalized question/answer tool. Of course it doesn't help that most of the internet is walled off now. I suspect that AI search will be very useful for looking up information, but awful for searching webpages.

swatcoder 4y ago
Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. There was a period of time where the ratio between indexing techniques and content worked, but we’re leaving that era now. All those things you hate about modern Google are necessary survival strategies for this model. AI search can’t solve that problem. What it can do is give up on the idea of surfacing specific sources in favor of synthesizing answers from slurry of all the content, finding (or hallucinating) signal in the noise. If you want strict keywords and boolean operators, you can’t expect to find that from the universal search engines that you’re used to. You want to look for and support curated engines that aggressively restrict what they index so that it contains high signal and low noise. Internal walled garden searches do this. Marginalia does this. Others probably do too. That’s what you’re looking for.
- JohnFen 4y ago
  > All those things you hate about modern Google are necessary survival strategies for this model. The reason I doubt this is because right now, I can find what I'm looking for through keyword searches on search engines that allow that, where I very often can't through the likes of Google.
  - lll-o-lll 4y ago
    What search engines allow this?
    - JohnFen 4y ago
      Very few, but the closest approximation that I can find right now is DDG. Kagi might fulfill this role better. I haven't investigated it yet, though.
- ravenstine 4y ago
  Supposedly. The trouble is there's no way for me to know how true this noise theory is. I'm given no option by these search engines to "just show me all the crap." What search engines do today is so filtered and heuristical that there are apparently phrases and ideas that have never been expressed on the internet ever, or at least that's the impression I'd get if I was more naive. Besides, it's not like there isn't noise anyway. Ever click on the top results only to find that nothing matches your query and probably never did?
  - allturtles 4y ago
    Do you really expect them to maintain an entire other version of the search product just to prove to skeptics that that version is not good?
    - ravenstine 4y ago
      No.
  - Clent 4y ago
    Google is not a search engine. It is a curation service. It only searches this collection of data that it decided was worthy of curation. Google tells us this is for the best. We are forced to take their word for it since their algorithms are not public. Google tells us this is also for the best.
  - pixl97 4y ago
    There is a way, just start spending millions/billions of dollars to index the internet... Just figure out how not to index the endless content generation scripts, and places like Reddit that are 100s of millions of pages themselves. Or the content farms that are just copies of other sites. Dead internet prophecy is coming true.
- eitland 4y ago
  > Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. Go check search.marginalia.nu. Do some sample searches about history or linux or git or something and see how it often outclasses Google. Then ponder the fact that it is run by one Swede and it runs on one (admittedly beefy but nothing unusual) workstation in his living room. Personally at least I have seen enough to have made up my mind that it isn't impossible to do better than Google, they just don't want to.
  - swatcoder 4y ago
    Did you read my whole comment? I even mention Marginalia!
    - eitland 4y ago
      I clearly didn't read your whole post. Have my upvote. That said I think my point still stands: If 1 Swede and 1 server can do a much better job than Google in significant niches such as history, Linux and Git, then maybe a hundred Swedes and 10 000 servers can outcompete Google across the board for traditional search, leaving Google to deal with plane tickets and local search? ; - ) And no, I'm not a Swede.
substation13 4y ago
Sometimes I want a simpler system so that I can understand how it works in order to make it do exactly what I want. This is like driving your own car. Other times, I want an intelligent system that figures out what I really meant and take care of the details. This is like having a chauffeur. Which is better? It depends. But Google seems to think that only chauffeur style is important.
- OJFord 4y ago
  Much as I'm a bit of a sceptic, this is actually quite a compelling aspect of ChatGPT isn't it? It's not even the LLM necessarily, it's the interaction and the NLP: 'those results are not what I'm looking for [remove all those], I remember something I saw before [if you have access to my history..] which included the phrase 'hunter2' [refine for exactly that, not hunter without number, not different numbers, not synonyms for hunter]'
  - JohnFen 4y ago
    When I'm looking for web pages, though, I actively don't want to use natural language. Natural language is too vague and open to interpretation. To make it work, you have to get very precise in your wording, and it's just easier to get the right precision using keywords and operators.
    - _puk 4y ago
      Why does it have to be so binary? Natural Language is great for loosely defining what you want, whilst literal searches are great for finding the exact occurrence of something you know you are looking for. I can imagine something like: Search for the error code: "XY123", triggered by product "ABC" when I try and start it bound to port 123. I'm using version 1.0.0, and want results from the official documentation or sites of good repute. Only include solved problems, and give me references in your results. Granted, a lot of that could be covered by special incantations in a more literal search, but that just feels far more flexible. Note I'm trying to delineate between literal (in quotes) and Natural Language in this example if it's not clear.
KMag 4y ago
Early Google was somewhat intentionally optimized for long-tail searches, the thought being that common searches are served fine by most search engines and it's the difficult long-tail searches where people actually try other search engines and cause people to switch their defaults/habits. I suspect it's very difficult to properly define a machine learning objective to specifically optimize for the long-tail. Without very careful crafting, I suspect learn-to-rank optimizes more evenly, resulting in poorer ranking performance in the difficult long-tail. Back 20 years ago when I worked on web search infra at Google, a bunch of the "do what I mean, not what I say" features were disabled if you used any of the search operators in your query. If I recall correctly, it even used its "Kansas" database of per-user data to check if you'd recently used any search operators. Most of that would be very difficult and/or costly to implement with learn-to-rank, though, I guess you could pass a "power_user" boolean to learn-to-rank to try and regain some of the lost functionality.
- RC_ITR 4y ago
  Are you sure? From the original Google paper: In fact, as of November 1997, only one of the top four commercial search engines finds itself (returns its own search page in response to its name in the top ten results). [0] [0]http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html
  - KMag 4y ago
    Circa 2003, Yahoo wasn't terrible. By our internal metrics, Yahoo search had better results than Bing before Yahoo started using Bing as their back-end. By that point, I suspect all of the major search engines were using some form of link-graph based ranking signals, likely modified both for scalability and avoiding the PageRank patent. Maybe I was a bit loose with my definition of "Early Google", but in the 2003-2010 time frame, the major search engines had pretty well figured out the most common 95% to 99% of queries.
jimhefferon 4y ago
Most of the time when I have a tool that tries to guess what I want, I hate it. I hate it with a deep and burning hatred. I call it the "Clippy" effect.
- cubefox 4y ago
  We a right at the point in history were this is about to change. I tried "new Bing" and it understands better than ChatGPT while also being able to use online search. And Google's Bard is just a few months, or weeks, away. Just a few years ago (pre GPT-3) this would have been unthinkable.
- pphysch 4y ago
  Suggestions/autocomplete are force multipliers that can also go <1. When they are accurate, they save you a lot of typing/work. When they are inaccurate, you waste time processing the noise and have to do the work anyways.
  - munk-a 4y ago
    Here's the problem (and why I personally hate overly smart IDEs) - my job is a creative one where most of the time I'm thinking and want my hands to be on autopilot. Writing out a skeleton class definition is something I can do blindfolded it takes almost no effort - an autocompleter getting that skeleton wrong (or writing a skeleton for a class with no interfaces and requiring me to backtrack through its garbled crap) takes effort. Reacting to things (observing the world and acting in response) will always take more mental effort than mindlessly slamming out some rote BS. IDEs optimize a part of my job that I honestly could care less about (writing massive amounts of green field code) at the expense of making occasional mistakes when I'm doing my actual job (examining and tweaking existing code). This trade off is not worth it. I'll uh - I'll keep function autocompletion though. Trying to remember whether my coworker called it `getWidget()` or `WidgetRegistry::get()` is something I don't need to waste brain cells on.
    - JoeAltmaier 4y ago
      The history of useless code-automation tools is long. Back in the day, it was generating boilerplate API code for Windows etc. Write an app in a day! But the lie was, in an app development schedule of 2000 hours, the tool made the 6-hour window definition effort into a 1-hour effort. So what? It was good for demo-ware showing what an app might look like. But pointless for real work. Popular among the vaporware crowd, marketing and pitching etc. But not for software engineers.
xnx 4y ago
Is quoting terms not working for you? I don't think I've had problems with any specific searches when I explicitly quote my terms. e.g. "this is exactly the phrase I want to search" or "search" "these" "words"
- JohnFen 4y ago
  Google's quoting has never worked well for me since they made the shift away from keyword searching. Oddly, before they made that shift, it did work.
- Izkata 4y ago
  You also have to go into the "Tools" menu at the top of the results and change "All results" to "Verbatim", otherwise it'll probably include conjugations even with the quotes.
  - deleted 4y ago
    [deleted]
1vuio0pswjnm7 4y ago
Fixing typos in "search" illustrates the difference between "search" as a means to find out whether something exists in a database versus a means for an advertising company to direct the searcher to the most optimum page for supporting advertising services.1 What is the difference between showing someone the winning advertisement versus showing them the winning "search result".1 The advertising company makes a decision according to secret criteria which ad/search result to show. A long, successful tradition of using alphabetical or chronological sorting for information, objective criteria, is jettisoned in favour of manipulation for commercial benefit of a middleman. The advertising company gets to decide what comes first and last, according to subjective criteria. Then make it infeasible to see what is last. Google has admitted that in the majority of cases, an appropriate ad does not exist to pair with a given search query. That is what I call the noncommercial web. It has little direct value to "tech" companies and advertisers. Yet "tech" company intermediaries want to surveil and control our access to it nonetheless. As the parent states, "most of the internet is walled off." In sum, the advertising company, and "AI", allow people to search for web content that exists, cf. allowing people to search for whether certain web content exists. The advertising company will make "suggestions". However not all content that exists is made searchable by the advertising company. If the content has no commercial value, e.g., it is "unpopular", it is considered garbage.3 1. When we search Wikipedia, typos do not result in "guessing" by an advertising company. If something does not exist, Wikipedia does not try to guess "what we want". 2. The term "winning" applies because the advertising company makes the process an actual auction or an effective contest. 3. If the web is mostly garbage, then its users should be entitled to know that through discovery. It seems as if the discoverable web, the one optimised for the advertising company, is mostly commercial garbage anyway. It is difficult to be serious about "AI" when it is trained on mostly garbage.
- deleted 4y ago
  [deleted]
- klooney 4y ago
  I'm really on the fence about Kagi. Results are good, but I'm just so accustomed to search being free.
drewcoo 4y ago
> fixing type-os I see what you did there ("typos"). Your reasons for search usefulness declining somehow ignore the elephant in the room: SEO. Because search as a business tends to be about advertising monetization, that's the system that gets gamed. Want better search? Or at least a differently-skewed search? Find a way to make it not be about ad money.
cpeterso 4y ago
Google PageRank was a breakthrough, but IIRC, another huge search improvement was Google AND'ing search terms instead of OR'ing them: if you searched for apple pie, AltaVista would return a list of pages that include apple OR pie (perhaps a lot of useless hits about Apple Computer and cherry pies), while would Google return only pages with apple AND pie.
deleted 4y ago
[deleted]
dangus 4y ago
Most likely, this is what you think you want. If Google actually gave you this you'd probably be wondering why your results were so irrelevant. It's similar to how music players don't use true random algorithms for shuffling music. The truth is that what people want from "random" is "I am not so likely to hear the same song too quickly" not "true random," which would have the possibility of playing the same song twice in a row. By the way, in Google you can put exact queries in quotes, and there's also advanced search: https://www.google.com/advanced_search https://www.google.com/advanced_search

swatcoder 4y ago

Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. There was a period of time where the ratio between indexing techniques and content worked, but we’re leaving that era now. All those things you hate about modern Google are necessary survival strategies for this model. AI search can’t solve that problem. What it can do is give up on the idea of surfacing specific sources in favor of synthesizing answers from slurry of all the content, finding (or hallucinating) signal in the noise. If you want strict keywords and boolean operators, you can’t expect to find that from the universal search engines that you’re used to. You want to look for and support curated engines that aggressively restrict what they index so that it contains high signal and low noise. Internal walled garden searches do this. Marginalia does this. Others probably do too. That’s what you’re looking for.

JohnFen 4y ago
> All those things you hate about modern Google are necessary survival strategies for this model. The reason I doubt this is because right now, I can find what I'm looking for through keyword searches on search engines that allow that, where I very often can't through the likes of Google.
- lll-o-lll 4y ago
  What search engines allow this?
  - JohnFen 4y ago
    Very few, but the closest approximation that I can find right now is DDG. Kagi might fulfill this role better. I haven't investigated it yet, though.
ravenstine 4y ago
Supposedly. The trouble is there's no way for me to know how true this noise theory is. I'm given no option by these search engines to "just show me all the crap." What search engines do today is so filtered and heuristical that there are apparently phrases and ideas that have never been expressed on the internet ever, or at least that's the impression I'd get if I was more naive. Besides, it's not like there isn't noise anyway. Ever click on the top results only to find that nothing matches your query and probably never did?
- allturtles 4y ago
  Do you really expect them to maintain an entire other version of the search product just to prove to skeptics that that version is not good?
  - ravenstine 4y ago
    No.
- Clent 4y ago
  Google is not a search engine. It is a curation service. It only searches this collection of data that it decided was worthy of curation. Google tells us this is for the best. We are forced to take their word for it since their algorithms are not public. Google tells us this is also for the best.
- pixl97 4y ago
  There is a way, just start spending millions/billions of dollars to index the internet... Just figure out how not to index the endless content generation scripts, and places like Reddit that are 100s of millions of pages themselves. Or the content farms that are just copies of other sites. Dead internet prophecy is coming true.
eitland 4y ago
> Google search can’t do what you want because there’s just too much noise in the “crawl everything” internet and the noise grows exponentially faster than the techniques that can automatically organize and rank it. Go check search.marginalia.nu. Do some sample searches about history or linux or git or something and see how it often outclasses Google. Then ponder the fact that it is run by one Swede and it runs on one (admittedly beefy but nothing unusual) workstation in his living room. Personally at least I have seen enough to have made up my mind that it isn't impossible to do better than Google, they just don't want to.
- swatcoder 4y ago
  Did you read my whole comment? I even mention Marginalia!
  - eitland 4y ago
    I clearly didn't read your whole post. Have my upvote. That said I think my point still stands: If 1 Swede and 1 server can do a much better job than Google in significant niches such as history, Linux and Git, then maybe a hundred Swedes and 10 000 servers can outcompete Google across the board for traditional search, leaving Google to deal with plane tickets and local search? ; - ) And no, I'm not a Swede.

substation13 4y ago

Sometimes I want a simpler system so that I can understand how it works in order to make it do exactly what I want. This is like driving your own car. Other times, I want an intelligent system that figures out what I really meant and take care of the details. This is like having a chauffeur. Which is better? It depends. But Google seems to think that only chauffeur style is important.

OJFord 4y ago
Much as I'm a bit of a sceptic, this is actually quite a compelling aspect of ChatGPT isn't it? It's not even the LLM necessarily, it's the interaction and the NLP: 'those results are not what I'm looking for [remove all those], I remember something I saw before [if you have access to my history..] which included the phrase 'hunter2' [refine for exactly that, not hunter without number, not different numbers, not synonyms for hunter]'
- JohnFen 4y ago
  When I'm looking for web pages, though, I actively don't want to use natural language. Natural language is too vague and open to interpretation. To make it work, you have to get very precise in your wording, and it's just easier to get the right precision using keywords and operators.
  - _puk 4y ago
    Why does it have to be so binary? Natural Language is great for loosely defining what you want, whilst literal searches are great for finding the exact occurrence of something you know you are looking for. I can imagine something like: Search for the error code: "XY123", triggered by product "ABC" when I try and start it bound to port 123. I'm using version 1.0.0, and want results from the official documentation or sites of good repute. Only include solved problems, and give me references in your results. Granted, a lot of that could be covered by special incantations in a more literal search, but that just feels far more flexible. Note I'm trying to delineate between literal (in quotes) and Natural Language in this example if it's not clear.

KMag 4y ago

Early Google was somewhat intentionally optimized for long-tail searches, the thought being that common searches are served fine by most search engines and it's the difficult long-tail searches where people actually try other search engines and cause people to switch their defaults/habits. I suspect it's very difficult to properly define a machine learning objective to specifically optimize for the long-tail. Without very careful crafting, I suspect learn-to-rank optimizes more evenly, resulting in poorer ranking performance in the difficult long-tail. Back 20 years ago when I worked on web search infra at Google, a bunch of the "do what I mean, not what I say" features were disabled if you used any of the search operators in your query. If I recall correctly, it even used its "Kansas" database of per-user data to check if you'd recently used any search operators. Most of that would be very difficult and/or costly to implement with learn-to-rank, though, I guess you could pass a "power_user" boolean to learn-to-rank to try and regain some of the lost functionality.

RC_ITR 4y ago
Are you sure? From the original Google paper: In fact, as of November 1997, only one of the top four commercial search engines finds itself (returns its own search page in response to its name in the top ten results). [0] [0]http://infolab.stanford.edu/~backrub/google.html http://infolab.stanford.edu/~backrub/google.html
- KMag 4y ago
  Circa 2003, Yahoo wasn't terrible. By our internal metrics, Yahoo search had better results than Bing before Yahoo started using Bing as their back-end. By that point, I suspect all of the major search engines were using some form of link-graph based ranking signals, likely modified both for scalability and avoiding the PageRank patent. Maybe I was a bit loose with my definition of "Early Google", but in the 2003-2010 time frame, the major search engines had pretty well figured out the most common 95% to 99% of queries.

jimhefferon 4y ago

Most of the time when I have a tool that tries to guess what I want, I hate it. I hate it with a deep and burning hatred. I call it the "Clippy" effect.

cubefox 4y ago
We a right at the point in history were this is about to change. I tried "new Bing" and it understands better than ChatGPT while also being able to use online search. And Google's Bard is just a few months, or weeks, away. Just a few years ago (pre GPT-3) this would have been unthinkable.
pphysch 4y ago
Suggestions/autocomplete are force multipliers that can also go <1. When they are accurate, they save you a lot of typing/work. When they are inaccurate, you waste time processing the noise and have to do the work anyways.
- munk-a 4y ago
  Here's the problem (and why I personally hate overly smart IDEs) - my job is a creative one where most of the time I'm thinking and want my hands to be on autopilot. Writing out a skeleton class definition is something I can do blindfolded it takes almost no effort - an autocompleter getting that skeleton wrong (or writing a skeleton for a class with no interfaces and requiring me to backtrack through its garbled crap) takes effort. Reacting to things (observing the world and acting in response) will always take more mental effort than mindlessly slamming out some rote BS. IDEs optimize a part of my job that I honestly could care less about (writing massive amounts of green field code) at the expense of making occasional mistakes when I'm doing my actual job (examining and tweaking existing code). This trade off is not worth it. I'll uh - I'll keep function autocompletion though. Trying to remember whether my coworker called it `getWidget()` or `WidgetRegistry::get()` is something I don't need to waste brain cells on.
  - JoeAltmaier 4y ago
    The history of useless code-automation tools is long. Back in the day, it was generating boilerplate API code for Windows etc. Write an app in a day! But the lie was, in an app development schedule of 2000 hours, the tool made the 6-hour window definition effort into a 1-hour effort. So what? It was good for demo-ware showing what an app might look like. But pointless for real work. Popular among the vaporware crowd, marketing and pitching etc. But not for software engineers.

xnx 4y ago

Is quoting terms not working for you? I don't think I've had problems with any specific searches when I explicitly quote my terms. e.g. "this is exactly the phrase I want to search" or "search" "these" "words"

JohnFen 4y ago
Google's quoting has never worked well for me since they made the shift away from keyword searching. Oddly, before they made that shift, it did work.
Izkata 4y ago
You also have to go into the "Tools" menu at the top of the results and change "All results" to "Verbatim", otherwise it'll probably include conjugations even with the quotes.
- deleted 4y ago
  [deleted]

1vuio0pswjnm7 4y ago

Fixing typos in "search" illustrates the difference between "search" as a means to find out whether something exists in a database versus a means for an advertising company to direct the searcher to the most optimum page for supporting advertising services.1 What is the difference between showing someone the winning advertisement versus showing them the winning "search result".1 The advertising company makes a decision according to secret criteria which ad/search result to show. A long, successful tradition of using alphabetical or chronological sorting for information, objective criteria, is jettisoned in favour of manipulation for commercial benefit of a middleman. The advertising company gets to decide what comes first and last, according to subjective criteria. Then make it infeasible to see what is last. Google has admitted that in the majority of cases, an appropriate ad does not exist to pair with a given search query. That is what I call the noncommercial web. It has little direct value to "tech" companies and advertisers. Yet "tech" company intermediaries want to surveil and control our access to it nonetheless. As the parent states, "most of the internet is walled off." In sum, the advertising company, and "AI", allow people to search for web content that exists, cf. allowing people to search for whether certain web content exists. The advertising company will make "suggestions". However not all content that exists is made searchable by the advertising company. If the content has no commercial value, e.g., it is "unpopular", it is considered garbage.3 1. When we search Wikipedia, typos do not result in "guessing" by an advertising company. If something does not exist, Wikipedia does not try to guess "what we want". 2. The term "winning" applies because the advertising company makes the process an actual auction or an effective contest. 3. If the web is mostly garbage, then its users should be entitled to know that through discovery. It seems as if the discoverable web, the one optimised for the advertising company, is mostly commercial garbage anyway. It is difficult to be serious about "AI" when it is trained on mostly garbage.

deleted 4y ago
[deleted]
klooney 4y ago
I'm really on the fence about Kagi. Results are good, but I'm just so accustomed to search being free.

drewcoo 4y ago

> fixing type-os I see what you did there ("typos"). Your reasons for search usefulness declining somehow ignore the elephant in the room: SEO. Because search as a business tends to be about advertising monetization, that's the system that gets gamed. Want better search? Or at least a differently-skewed search? Find a way to make it not be about ad money.

cpeterso 4y ago

Google PageRank was a breakthrough, but IIRC, another huge search improvement was Google AND'ing search terms instead of OR'ing them: if you searched for apple pie, AltaVista would return a list of pages that include apple OR pie (perhaps a lot of useless hits about Apple Computer and cherry pies), while would Google return only pages with apple AND pie.

deleted 4y ago

[deleted]

dangus 4y ago

Most likely, this is what you think you want. If Google actually gave you this you'd probably be wondering why your results were so irrelevant. It's similar to how music players don't use true random algorithms for shuffling music. The truth is that what people want from "random" is "I am not so likely to hear the same song too quickly" not "true random," which would have the possibility of playing the same song twice in a row. By the way, in Google you can put exact queries in quotes, and there's also advanced search: https://www.google.com/advanced_search https://www.google.com/advanced_search