5 ms·

Stop Me If You've Seen This Word Before

47 points by Oompa 18y ago

tumult 18y ago
http://www.google.com/search?hl=en&safe=off&q=%22the+the%22+band&btnG=Search http://www.google.com/search?hl=en&safe=off&q=%22the... as soon as the article asserted this wouldn't work, i tried googling, and it worked fine. i stopped reading after that. edit: for whatever reason, if you follow the link directly, the search results are wrong. you might have to submit the query again after the page loads to get the right results. weird! maybe he was onto something (nope)
- jwilliams 18y ago
  He says in the article: Google doesn't seem to use stop words any more. (in fact he makes a point of that fact).
  - tumult 18y ago
    then why did he start the article with it? :( i guess i should have read it more then
  - fhars 18y ago
    And in this case, yahoo seems to be better than google: http://twingine.no/search.php?q=the+the http://twingine.no/search.php?q=the+the Now all we need is a MySQL stopword story contest...
- unfoldedorigami 18y ago
  Maybe you should have read all the way through.
jgrahamc 18y ago
POPFile has stopwords because people in the community insisted on it. My commercial email filtering software does not because it turned out that in my tests that the accuracy difference they made was so small as to be in the noise. And they were costly in terms of time to check, and to maintain across different languages.
dwwatk01 18y ago
I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"
- a-priori 18y ago
  Glad to hear I'm not the only one who's had trouble with this. I remember reading some C code once and I saw some strange operators I had never seen before... <? and >?, if I remember correctly. I eventually figured out they were GCC extensions for "min" and "max", but only after a painful experience trying to get more info out of Google than "here's how you do addition and multiplication in C!"
  - technoguyrob 18y ago
    http://www.google.com/codesearch?hl=en&lr=&q=lang%3Ac%2B%2B+%22+%3E%3F+%22&sbtn=Search http://www.google.com/codesearch?hl=en&lr=&q=lang%3A...
  - deleted 18y ago
    [deleted]
- technoguyrob 18y ago
  http://www.google.com/codesearch?hl=en&lr=&q=lang%3Aperl+%22%24.%22&sbtn=Search http://www.google.com/codesearch?hl=en&lr=&q=lang%3A...
- jrockway 18y ago
  $. -> "perldoc perlvar" -> $INPUT_LINE_NUMBER -> http://www.google.com/search?q=%24INPUT_LINE_NUMBER+perl http://www.google.com/search?q=%24INPUT_LINE_NUMBER+perl
whacked_new 18y ago
> "The" is one of the most common words in the English language "the" is THE most common word in English.
robg 18y ago
Way back in 2004, I ran a little experiment with Google -- over a period of a week, I searched for an entire dictionary of ~110k individual English words and recorded how many hits Google returned for each. Of course, a word can appear on a page multiple times. That's why, I think, folks used to ignore the stopwords. They introduced noise when trying to access the content words. Now, with span constraints, you can incorporate them into the analysis. So "a matrix" and "the matrix" returns very different results, even without quotes.
gills 18y ago
It makes sense that low-information terms would have a lower preference when searching without any context. If your index models the context around terms, you can get better results from a low-information search. I think...I'm kind of shooting from the hip here relating it to context modeling in lossless compression schemes like CABAC and PPM. Could you overcome stop words with some sort of Bayesian phrase matching over some learned hidden states?
liuliu 18y ago
it is about how to sort with stop word. Tranditional tf-idf method didn't work well as it didn't contain any information about each word relative location in its context. a simple method is to index "the the", the word group instead of single "the". I guess it is what Google does now with "to be or not to be". However, the word grouping tech is a common method in CJK full text search.
jcromartie 18y ago
I like how the top Google words are all generic web marketing words, with the two exceptions of "hotels" and "women."
randomuser7 18y ago
I guess the idea was to help allow English search queries (i.e. exclude words people were using to describe their query but shouldn't be searched for).

47 points by Oompa 18y ago

http://www.google.com/search?hl=en&safe=off&q=%22the+the%22+band&btnG=Search http://www.google.com/search?hl=en&safe=off&q=%22the... as soon as the article asserted this wouldn't work, i tried googling, and it worked fine. i stopped reading after that. edit: for whatever reason, if you follow the link directly, the search results are wrong. you might have to submit the query again after the page loads to get the right results. weird! maybe he was onto something (nope)

jwilliams 18y ago
He says in the article: Google doesn't seem to use stop words any more. (in fact he makes a point of that fact).
- tumult 18y ago
  then why did he start the article with it? :( i guess i should have read it more then
- fhars 18y ago
  And in this case, yahoo seems to be better than google: http://twingine.no/search.php?q=the+the http://twingine.no/search.php?q=the+the Now all we need is a MySQL stopword story contest...
unfoldedorigami 18y ago
Maybe you should have read all the way through.

jgrahamc 18y ago

POPFile has stopwords because people in the community insisted on it. My commercial email filtering software does not because it turned out that in my tests that the accuracy difference they made was so small as to be in the noise. And they were costly in terms of time to check, and to maintain across different languages.

dwwatk01 18y ago

I ran into a similar problem teaching myself perl a couple years ago by doing a short tutorial then foolishly jumping into a co-worker's code. "What the hell is '$.'? Hmm, well I'm sure Google can help me. What? No matching documents?!? What is this crazy s.o.b. doing here??"

a-priori 18y ago
Glad to hear I'm not the only one who's had trouble with this. I remember reading some C code once and I saw some strange operators I had never seen before... <? and >?, if I remember correctly. I eventually figured out they were GCC extensions for "min" and "max", but only after a painful experience trying to get more info out of Google than "here's how you do addition and multiplication in C!"
- technoguyrob 18y ago
  http://www.google.com/codesearch?hl=en&lr=&q=lang%3Ac%2B%2B+%22+%3E%3F+%22&sbtn=Search http://www.google.com/codesearch?hl=en&lr=&q=lang%3A...
- deleted 18y ago
  [deleted]
technoguyrob 18y ago
http://www.google.com/codesearch?hl=en&lr=&q=lang%3Aperl+%22%24.%22&sbtn=Search http://www.google.com/codesearch?hl=en&lr=&q=lang%3A...
jrockway 18y ago
$. -> "perldoc perlvar" -> $INPUT_LINE_NUMBER -> http://www.google.com/search?q=%24INPUT_LINE_NUMBER+perl http://www.google.com/search?q=%24INPUT_LINE_NUMBER+perl

whacked_new 18y ago

> "The" is one of the most common words in the English language "the" is THE most common word in English.

robg 18y ago

Way back in 2004, I ran a little experiment with Google -- over a period of a week, I searched for an entire dictionary of ~110k individual English words and recorded how many hits Google returned for each. Of course, a word can appear on a page multiple times. That's why, I think, folks used to ignore the stopwords. They introduced noise when trying to access the content words. Now, with span constraints, you can incorporate them into the analysis. So "a matrix" and "the matrix" returns very different results, even without quotes.

gills 18y ago

It makes sense that low-information terms would have a lower preference when searching without any context. If your index models the context around terms, you can get better results from a low-information search. I think...I'm kind of shooting from the hip here relating it to context modeling in lossless compression schemes like CABAC and PPM. Could you overcome stop words with some sort of Bayesian phrase matching over some learned hidden states?

liuliu 18y ago

it is about how to sort with stop word. Tranditional tf-idf method didn't work well as it didn't contain any information about each word relative location in its context. a simple method is to index "the the", the word group instead of single "the". I guess it is what Google does now with "to be or not to be". However, the word grouping tech is a common method in CJK full text search.

jcromartie 18y ago

I like how the top Google words are all generic web marketing words, with the two exceptions of "hotels" and "women."

randomuser7 18y ago

I guess the idea was to help allow English search queries (i.e. exclude words people were using to describe their query but shouldn't be searched for).