4 ms·

How not to break a search engine

57 points by akpa1 5y ago

amelius 5y ago
I want to know how to benchmark a search engine.
- chii 5y ago
  I'm sure there's tonnes of research and prior art on this subject, but it's an interesting inquiry. off the top of my head, there's two meaning - performance benchmarking (how fast the search results comes back), and accuracy/fit-for-purposeness benchmarking (how good it is at finding something the user intends). Performance is easy. It's the accuracy/fit-for-purposeness that would be an interesting benchmark. I wonder if you have to use an empirical measurement for accuracy - that is, give a random sample of people a target piece of code (or file) to find, and see how long or how many queries it takes to find it.
  - toast0 5y ago
    For quality, you really do need to do human qualititative measures to get a full measure, with all of the fun that involves. However, you can do things like generate search terms from your top N documents through some method, and then do the queries and confirm the document you generated the term from shows up in the top M results. This can be circular though if you're not careful; the top N documents may not include important documents that nobody could find.
- stevenbedrick 5y ago
  A good place to start might be the “Evaluation” chapter of Manning et al.‘a “Introduction to Information Retrieval”: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-in-information-retrieval-1.html https://nlp.stanford.edu/IR-book/html/htmledition/evaluation... The whole book is superb, and while some of the content is a bit long in the tooth at this point, the evaluation chapter has aged extremely well. Oh, and if you’re interested in how to evaluate search engine user interfaces, the equivalent chapter in Hearst’s book on Search UI has you covered: https://searchuserinterfaces.com/book/sui_ch2_evaluation.html https://searchuserinterfaces.com/book/sui_ch2_evaluation.htm...
sqs 5y ago
@rijnard (the blog post author) is awesome, and all of the code changes he talks about in the blog post are public. You can see all of his recent changes to search code in https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+type:diff+author:rijnard+f:search&patternType=literal https://sourcegraph.com/search?q=context:global+repo:%5Egith... in case you want to follow along by just reading the code (that query shows all of his diffs that touch paths containing `search`).
walrus01 5y ago
If you want to see a very bad example of a broken search engine, use the Google voice Assistant to search for "how many raccoons can fit"
- oogali 5y ago
  Wow, I was not expecting _that_ result.
  - walrus01 5y ago
    Neither were the raccoons
- fsckboy 5y ago
  i just did a regular text search for it and the top results where "how many raccoons can fit up your butt"... no voice assistant involved, and the destination urls were not google's.
- Jap2-0 5y ago
  For something a little more innocuous, type "Cicero" in the address bar in Chrome or Firefox and see what it autocompletes it to.
  - microtherion 5y ago
    You mean the 4.5 millimeters? I believe that's a legitimate unit in use in the publishing business.
    - Jap2-0 5y ago
      Yes, and it is; however, my (intended) point was that it is what almost no one is looking for when they type "Cicero", and if you simply search for Cicero by itself, you get nothing telling you that it's a typographic unit.
- Dah00n 5y ago
  I'm not sure that is "broken" but more like inappropriate. I read an article with a list of bad default results (as in with no filter bubble) that are way worse. Like searching for "tween" on bing shows auto suggestions like "tween swimsuits inappropriate" and "tween budding images". Same for all search engines that uses bing like Brave and DDG (at least at the time I read about it). It seems pretty obvious what those searches are looking for.
1970-01-01 5y ago
I was playing with breaking them a long time ago. Here are some interesting searches that seem to do almost nothing: Quoted Single ASCII Chars: https://search.yahoo.com/search?p="~" https://search.yahoo.com/search?p="~" https://www.google.com/search?&q="~" https://www.google.com/search?&q="~" https://www.bing.com/search?q="~" https://www.bing.com/search?q="~" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="`" https://www.google.com/search?&q= https://www.google.com/search?&q="`" https://www.bing.com/search?q= https://www.bing.com/search?q="`" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="!" https://www.google.com/search?&q= https://www.google.com/search?&q="!" https://www.bing.com/search?q= https://www.bing.com/search?q="!" https://search.yahoo.com/search?p="@" https://search.yahoo.com/search?p="@" https://www.google.com/search?&q="@" https://www.google.com/search?&q="@" https://www.bing.com/search?q="@" https://www.bing.com/search?q="@" https://search.yahoo.com/search?p="#" https://search.yahoo.com/search?p="#" https://www.google.com/search?&q="#" https://www.google.com/search?&q="#" https://www.bing.com/search?q="#" https://www.bing.com/search?q="#" https://search.yahoo.com/search?p="$" https://search.yahoo.com/search?p="$" https://www.google.com/search?&q="$" https://www.google.com/search?&q="$" https://www.bing.com/search?q="$" https://www.bing.com/search?q="$" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="%" https://www.google.com/search?&q="%" https://www.google.com/search?&q="%" https://www.bing.com/search?q="%" https://www.bing.com/search?q="%" https://search.yahoo.com/search?p="^" https://search.yahoo.com/search?p="^" https://www.google.com/search?&q="^" https://www.google.com/search?&q="^" https://www.bing.com/search?q="^" https://www.bing.com/search?q="^" https://search.yahoo.com/search?p="&" https://search.yahoo.com/search?p="&" https://www.google.com/search?&q="&" https://www.google.com/search?&q="&" https://www.bing.com/search?q="&" https://www.bing.com/search?q="&" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="" https://www.google.com/search?&q= https://www.google.com/search?&q="" https://www.bing.com/search?q= https://www.bing.com/search?q="" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="(" https://www.google.com/search?&q= https://www.google.com/search?&q="(" https://www.bing.com/search?q= https://www.bing.com/search?q="(" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p=")" https://www.google.com/search?&q= https://www.google.com/search?&q=")" https://www.bing.com/search?q= https://www.bing.com/search?q=")" https://search.yahoo.com/search?p="-" https://search.yahoo.com/search?p="-" https://www.google.com/search?&q="-" https://www.google.com/search?&q="-" https://www.bing.com/search?q="-" https://www.bing.com/search?q="-" https://search.yahoo.com/search?p="_" https://search.yahoo.com/search?p="_" https://www.google.com/search?&q="_" https://www.google.com/search?&q="_" https://www.bing.com/search?q="_" https://www.bing.com/search?q="_" https://search.yahoo.com/search?p="+" https://search.yahoo.com/search?p="+" https://www.google.com/search?&q="+" https://www.google.com/search?&q="+" https://www.bing.com/search?q="+" https://www.bing.com/search?q="+" https://search.yahoo.com/search?p="=" https://search.yahoo.com/search?p="=" https://www.google.com/search?&q="=" https://www.google.com/search?&q="=" https://www.bing.com/search?q="=" https://www.bing.com/search?q="=" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="\" https://search.yahoo.com/search?p="\" https://search.yahoo.com/search?p=%22\%22 https://search.yahoo.com/search?p=%22\%22 https://search.yahoo.com/search?p="#" https://search.yahoo.com/search?p="#" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p=":" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="%" https://www.bing.com/search?q="%28%29" https://www.bing.com/search?q="%28%29" https://www.bing.com/search?q= https://www.bing.com/search?q="." https://www.bing.com/search?q="==" https://www.bing.com/search?q="==" https://www.bing.com/search?q="$" https://www.bing.com/search?q="$" https://www.bing.com/search?q="+" https://www.bing.com/search?q="+" https://www.google.com/search?&q="~" https://www.google.com/search?&q="~" https://www.google.com/search?&q= https://www.google.com/search?&q="`" https://www.google.com/search?&q= https://www.google.com/search?&q="!" https://www.google.com/search?&q= https://www.google.com/search?&q="!!" https://www.google.com/search?&q="@" https://www.google.com/search?&q="@" https://www.google.com/search?&q="#" https://www.google.com/search?&q="#" https://www.google.com/search?&q="$" https://www.google.com/search?&q="$" https://www.google.com/search?&q="%" https://www.google.com/search?&q="%" https://www.google.com/search?&q="^" https://www.google.com/search?&q="^" https://www.google.com/search?&q="&" https://www.google.com/search?&q="&" https://www.google.com/search?&q= https://www.google.com/search?&q= #quotes do nothing here https://www.google.com/search?&q="\ https://www.google.com/search?&q="\*" #same as above https://www.google.com/search?&q= https://www.google.com/search?&q=":)" https://www.google.com/search?&q="~" https://www.google.com/search?&q="~" https://www.google.com/search?&q="\" https://www.google.com/search?&q="\" https://www.google.com/search?&q= https://www.google.com/search?&q=")" https://www.google.com/search?&q="%" https://www.google.com/search?&q="%"
tantalor 5y ago
No live traffic experiment? Seems risky to launch without

57 points by akpa1 5y ago

amelius 5y ago

I want to know how to benchmark a search engine.

chii 5y ago
I'm sure there's tonnes of research and prior art on this subject, but it's an interesting inquiry. off the top of my head, there's two meaning - performance benchmarking (how fast the search results comes back), and accuracy/fit-for-purposeness benchmarking (how good it is at finding something the user intends). Performance is easy. It's the accuracy/fit-for-purposeness that would be an interesting benchmark. I wonder if you have to use an empirical measurement for accuracy - that is, give a random sample of people a target piece of code (or file) to find, and see how long or how many queries it takes to find it.
- toast0 5y ago
  For quality, you really do need to do human qualititative measures to get a full measure, with all of the fun that involves. However, you can do things like generate search terms from your top N documents through some method, and then do the queries and confirm the document you generated the term from shows up in the top M results. This can be circular though if you're not careful; the top N documents may not include important documents that nobody could find.
stevenbedrick 5y ago
A good place to start might be the “Evaluation” chapter of Manning et al.‘a “Introduction to Information Retrieval”: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-in-information-retrieval-1.html https://nlp.stanford.edu/IR-book/html/htmledition/evaluation... The whole book is superb, and while some of the content is a bit long in the tooth at this point, the evaluation chapter has aged extremely well. Oh, and if you’re interested in how to evaluate search engine user interfaces, the equivalent chapter in Hearst’s book on Search UI has you covered: https://searchuserinterfaces.com/book/sui_ch2_evaluation.html https://searchuserinterfaces.com/book/sui_ch2_evaluation.htm...

sqs 5y ago

@rijnard (the blog post author) is awesome, and all of the code changes he talks about in the blog post are public. You can see all of his recent changes to search code in https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+type:diff+author:rijnard+f:search&patternType=literal https://sourcegraph.com/search?q=context:global+repo:%5Egith... in case you want to follow along by just reading the code (that query shows all of his diffs that touch paths containing `search`).

walrus01 5y ago

If you want to see a very bad example of a broken search engine, use the Google voice Assistant to search for "how many raccoons can fit"

oogali 5y ago
Wow, I was not expecting _that_ result.
- walrus01 5y ago
  Neither were the raccoons
fsckboy 5y ago
i just did a regular text search for it and the top results where "how many raccoons can fit up your butt"... no voice assistant involved, and the destination urls were not google's.
Jap2-0 5y ago
For something a little more innocuous, type "Cicero" in the address bar in Chrome or Firefox and see what it autocompletes it to.
- microtherion 5y ago
  You mean the 4.5 millimeters? I believe that's a legitimate unit in use in the publishing business.
  - Jap2-0 5y ago
    Yes, and it is; however, my (intended) point was that it is what almost no one is looking for when they type "Cicero", and if you simply search for Cicero by itself, you get nothing telling you that it's a typographic unit.
Dah00n 5y ago
I'm not sure that is "broken" but more like inappropriate. I read an article with a list of bad default results (as in with no filter bubble) that are way worse. Like searching for "tween" on bing shows auto suggestions like "tween swimsuits inappropriate" and "tween budding images". Same for all search engines that uses bing like Brave and DDG (at least at the time I read about it). It seems pretty obvious what those searches are looking for.

1970-01-01 5y ago

I was playing with breaking them a long time ago. Here are some interesting searches that seem to do almost nothing: Quoted Single ASCII Chars: https://search.yahoo.com/search?p="~" https://search.yahoo.com/search?p="~" https://www.google.com/search?&q="~" https://www.google.com/search?&q="~" https://www.bing.com/search?q="~" https://www.bing.com/search?q="~" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="`" https://www.google.com/search?&q= https://www.google.com/search?&q="`" https://www.bing.com/search?q= https://www.bing.com/search?q="`" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="!" https://www.google.com/search?&q= https://www.google.com/search?&q="!" https://www.bing.com/search?q= https://www.bing.com/search?q="!" https://search.yahoo.com/search?p="@" https://search.yahoo.com/search?p="@" https://www.google.com/search?&q="@" https://www.google.com/search?&q="@" https://www.bing.com/search?q="@" https://www.bing.com/search?q="@" https://search.yahoo.com/search?p="#" https://search.yahoo.com/search?p="#" https://www.google.com/search?&q="#" https://www.google.com/search?&q="#" https://www.bing.com/search?q="#" https://www.bing.com/search?q="#" https://search.yahoo.com/search?p="$" https://search.yahoo.com/search?p="$" https://www.google.com/search?&q="$" https://www.google.com/search?&q="$" https://www.bing.com/search?q="$" https://www.bing.com/search?q="$" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="%" https://www.google.com/search?&q="%" https://www.google.com/search?&q="%" https://www.bing.com/search?q="%" https://www.bing.com/search?q="%" https://search.yahoo.com/search?p="^" https://search.yahoo.com/search?p="^" https://www.google.com/search?&q="^" https://www.google.com/search?&q="^" https://www.bing.com/search?q="^" https://www.bing.com/search?q="^" https://search.yahoo.com/search?p="&" https://search.yahoo.com/search?p="&" https://www.google.com/search?&q="&" https://www.google.com/search?&q="&" https://www.bing.com/search?q="&" https://www.bing.com/search?q="&" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="" https://www.google.com/search?&q= https://www.google.com/search?&q="" https://www.bing.com/search?q= https://www.bing.com/search?q="" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p="(" https://www.google.com/search?&q= https://www.google.com/search?&q="(" https://www.bing.com/search?q= https://www.bing.com/search?q="(" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p=")" https://www.google.com/search?&q= https://www.google.com/search?&q=")" https://www.bing.com/search?q= https://www.bing.com/search?q=")" https://search.yahoo.com/search?p="-" https://search.yahoo.com/search?p="-" https://www.google.com/search?&q="-" https://www.google.com/search?&q="-" https://www.bing.com/search?q="-" https://www.bing.com/search?q="-" https://search.yahoo.com/search?p="_" https://search.yahoo.com/search?p="_" https://www.google.com/search?&q="_" https://www.google.com/search?&q="_" https://www.bing.com/search?q="_" https://www.bing.com/search?q="_" https://search.yahoo.com/search?p="+" https://search.yahoo.com/search?p="+" https://www.google.com/search?&q="+" https://www.google.com/search?&q="+" https://www.bing.com/search?q="+" https://www.bing.com/search?q="+" https://search.yahoo.com/search?p="=" https://search.yahoo.com/search?p="=" https://www.google.com/search?&q="=" https://www.google.com/search?&q="=" https://www.bing.com/search?q="=" https://www.bing.com/search?q="=" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="\" https://search.yahoo.com/search?p="\" https://search.yahoo.com/search?p=%22\%22 https://search.yahoo.com/search?p=%22\%22 https://search.yahoo.com/search?p="#" https://search.yahoo.com/search?p="#" https://search.yahoo.com/search?p= https://search.yahoo.com/search?p=":" https://search.yahoo.com/search?p="%" https://search.yahoo.com/search?p="%" https://www.bing.com/search?q="%28%29" https://www.bing.com/search?q="%28%29" https://www.bing.com/search?q= https://www.bing.com/search?q="." https://www.bing.com/search?q="==" https://www.bing.com/search?q="==" https://www.bing.com/search?q="$" https://www.bing.com/search?q="$" https://www.bing.com/search?q="+" https://www.bing.com/search?q="+" https://www.google.com/search?&q="~" https://www.google.com/search?&q="~" https://www.google.com/search?&q= https://www.google.com/search?&q="`" https://www.google.com/search?&q= https://www.google.com/search?&q="!" https://www.google.com/search?&q= https://www.google.com/search?&q="!!" https://www.google.com/search?&q="@" https://www.google.com/search?&q="@" https://www.google.com/search?&q="#" https://www.google.com/search?&q="#" https://www.google.com/search?&q="$" https://www.google.com/search?&q="$" https://www.google.com/search?&q="%" https://www.google.com/search?&q="%" https://www.google.com/search?&q="^" https://www.google.com/search?&q="^" https://www.google.com/search?&q="&" https://www.google.com/search?&q="&" https://www.google.com/search?&q= https://www.google.com/search?&q= #quotes do nothing here https://www.google.com/search?&q="\ https://www.google.com/search?&q="\*" #same as above https://www.google.com/search?&q= https://www.google.com/search?&q=":)" https://www.google.com/search?&q="~" https://www.google.com/search?&q="~" https://www.google.com/search?&q="\" https://www.google.com/search?&q="\" https://www.google.com/search?&q= https://www.google.com/search?&q=")" https://www.google.com/search?&q="%" https://www.google.com/search?&q="%"

tantalor 5y ago

No live traffic experiment? Seems risky to launch without