3 ms·

Thanks for the feedback. Punctuation is tricky, because it only adds value for power searchers, but indexing it would swell the index size quite a lot. Otherwis

Thanks for the feedback. Punctuation is tricky, because it only adds value for power searchers, but indexing it would swell the index size quite a lot. Otherwise, using double-quotes should do an exact search. On content farms, we've definitely heard that feedback. One point up for debate is whether to respond with algorithms-only, or whether we should update our quality guidelines to call out low-quality content farms as a type of webspam. Both DuckDuckGo and Blekko seem delighted to remove sites that most people don't like from the search results. Here's a link for DDG for example: http://www.technologyreview.com/blog/mimssbits/25532/ http://www.technologyreview.com/blog/mimssbits/25532/ . The question is: would you feel comfortable if Google removed results that a lot of people don't like from our index? And how do you balance the goal of reducing clutter and junk with the goal of being fair and comprehensive?

MikeCapone 16y ago
One potential problem of removing links that people report is that it's possible to game that system. You'll soon have spammers sending a massive amount of requests to have their competitors removed form the index.
imajes 16y ago
Yes! and if people wanted a fuller result, you can say, "more lower quality links found. Click here to view all" similar to the way that searching in gmail lets you search in trash & spam.
anigbrowl 16y ago
I know there's no simple answers for these things. On punctuation, consider the (admittedly obscure) situation of searching for some command line string: obscure_utility "-unknown" "-switches" - either the - sign excludes stuff you want, or gets stripped inside the quotes. Farming-wise, I think you should probably keep all those results in your index, even the ones that are composed of nothing more than your top searches separated by random phrases! even if it's not there now, in future it'll be possible to score page content on whether or not it has semantic value and draw inferences about sites or entire domains that are filled with junk. That will be interesting and useful from security, economic, and scientific points of view. In the meantime people will find useful analyses to run against that 'bad' data in your results which would not be practical if you purged too aggressively. What I had envisioned (which might be a tad ambitious, but bear in mind that you already have 5% of my local CPU for the asking with the desktop tool installed) is per-user search filtering. I may like sculpture but hate politics, so I would always search for 'statue -liberty', you are the other way around so your searches tend more towards 'liberty -statue'; I would very much like to be able to have complex filters on the client side, either locally or on the client-facing parts of your servers - and not just for spam sites. DDG takes the approach of allowing regex, which is a neat thing for the people who know enough to want it, and it would be interesting if search patterns and/or selective exclusions (as described above) could be stored locally, as either weighting tables or some sort of white/blacklist - always include wikipedia, never include about.com. I'm already running chrome and using a Nexus one, so perhaps some hashing could take place on my computers rather than increasing the load on your servers. The other reason besides spam is that lately I find myself wanting to do specialized searches, but I don't know how to specify the bounds of the search space. For example, I'm interested in law. But a lot of legal terms are in popular currency, so even if I search for "theft +legal" I may get tons of results for cheap car alarms or something. It would be fantastic if I could get a large set of results by specifying a large number of domain-specific terms - say, "tortfeasor privity precedent appellate" - and then hash and save that result set as a 'search space'. So then I could do more specific searches and know that my results would mostly come from websites devoted to the subject, with few that mention it only incidentally. In actual fact, the legal resources searchable via Scholar and Books are fantastic. I just picked law as an example of where you might want to temporarily limit your search set because most people can appreciate the difference between writing specifically about legal topics vs things that just mention the subject in passing. If you want to learn how to write a good disclaimer, "legal disclaimer" is not a good start because every 3rd landing page on the net includes that phrase as boilerplate. If users could save and reuse result sets, we'd get more actionable results, you'd (maybe) get lower server loads but more importantly, every successful hit (where the user doesn't search again or try another result for several minutes) is an implicit vote for the relevance of the result to the set, and thus a valid input to a semantic classifier.

MikeCapone 16y ago

One potential problem of removing links that people report is that it's possible to game that system. You'll soon have spammers sending a massive amount of requests to have their competitors removed form the index.

imajes 16y ago

Yes! and if people wanted a fuller result, you can say, "more lower quality links found. Click here to view all" similar to the way that searching in gmail lets you search in trash & spam.

anigbrowl 16y ago

I know there's no simple answers for these things. On punctuation, consider the (admittedly obscure) situation of searching for some command line string: obscure_utility "-unknown" "-switches" - either the - sign excludes stuff you want, or gets stripped inside the quotes. Farming-wise, I think you should probably keep all those results in your index, even the ones that are composed of nothing more than your top searches separated by random phrases! even if it's not there now, in future it'll be possible to score page content on whether or not it has semantic value and draw inferences about sites or entire domains that are filled with junk. That will be interesting and useful from security, economic, and scientific points of view. In the meantime people will find useful analyses to run against that 'bad' data in your results which would not be practical if you purged too aggressively. What I had envisioned (which might be a tad ambitious, but bear in mind that you already have 5% of my local CPU for the asking with the desktop tool installed) is per-user search filtering. I may like sculpture but hate politics, so I would always search for 'statue -liberty', you are the other way around so your searches tend more towards 'liberty -statue'; I would very much like to be able to have complex filters on the client side, either locally or on the client-facing parts of your servers - and not just for spam sites. DDG takes the approach of allowing regex, which is a neat thing for the people who know enough to want it, and it would be interesting if search patterns and/or selective exclusions (as described above) could be stored locally, as either weighting tables or some sort of white/blacklist - always include wikipedia, never include about.com. I'm already running chrome and using a Nexus one, so perhaps some hashing could take place on my computers rather than increasing the load on your servers. The other reason besides spam is that lately I find myself wanting to do specialized searches, but I don't know how to specify the bounds of the search space. For example, I'm interested in law. But a lot of legal terms are in popular currency, so even if I search for "theft +legal" I may get tons of results for cheap car alarms or something. It would be fantastic if I could get a large set of results by specifying a large number of domain-specific terms - say, "tortfeasor privity precedent appellate" - and then hash and save that result set as a 'search space'. So then I could do more specific searches and know that my results would mostly come from websites devoted to the subject, with few that mention it only incidentally. In actual fact, the legal resources searchable via Scholar and Books are fantastic. I just picked law as an example of where you might want to temporarily limit your search set because most people can appreciate the difference between writing specifically about legal topics vs things that just mention the subject in passing. If you want to learn how to write a good disclaimer, "legal disclaimer" is not a good start because every 3rd landing page on the net includes that phrase as boilerplate. If users could save and reuse result sets, we'd get more actionable results, you'd (maybe) get lower server loads but more importantly, every successful hit (where the user doesn't search again or try another result for several minutes) is an implicit vote for the relevance of the result to the set, and thus a valid input to a semantic classifier.