5 ms·

There are way more than two things that Google does wrong. Remapping my search terms into oblivion so it can pretend it's fast is the worst one. Especially when

There are way more than two things that Google does wrong. Remapping my search terms into oblivion so it can pretend it's fast is the worst one. Especially when this happens to a query I've modified to quote "every" "single" "flipping" "term." I think Google is cheating, and that their usable index is much shallower than they'd have you believe. What's needed is a search engine with functional queries (as opposed to Google, which now only operates in "the user is drunk" mode), that doesn't give a damn about your robots.txt, and that can capture content in a way that is more akin to archive.org than Google's shoddy and increasingly absent cache. Another issue is spam/false matches. Why does Google return illegitimate results? Because, let me tell you, any search for "some nifty computer book pdf" returns pages upon pages of bogus links leading to ad link mazes. A crawler should be able to trivially crawl such a page, determine that no PDF is linked, and blacklist the result, but this doesn't happen. Google is slow and preoccupied. Their business is ripe for disruption.

kjhughes 10y ago
To disable remapping of terms in Google, use verbatim search: On the search results page, choose "Search tools -> All results -> Verbatim"
- dredmorbius 10y ago
  There's no syntax of which I'm aware which tells Google to not "correct" or expand the results. When I'm seeking specific matches, I'm seeking really fucking specific matches. Google annoy me to no end.
  - user5994461 10y ago
    +word It used to force to search the word exactly as written.
    - donretag 10y ago
      The plus/+ notation has been removed. You must know surround the term in quotes "word".
    - dredmorbius 10y ago
      "Used to" is the operative phrase. That was among the negative consequences of Google+, as +<term> notation apparently was going to be reserved or repurposed for that somehow, but never was. I've done some large-scale searching where the most relevant detail is how many results are returned, most particularly for a specific domain. (For which, incidentally, there's no handy mechanism to accomplish, so it's <array of terms> * <array of domains>, and multiplicative explosion of searches, plus about a 45s timeout per query to avoid triggering bot defenses by Google. Such as this: https://www.reddit.com/r/dredmorbius/comments/3hp41w/tracking_the_conversation_fp_global_100_thinkers/ https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...
  - mtreis86 10y ago
    Boolean search would be wonderful
  - gamzer 10y ago
    I believe it was possible in the past using URL parameters (e.g. complete=0 to disable auto-suggest) but it does not seem to work anymore. Here it was documented in 2012: https://airstraining.wordpress.com/2012/06/13/google-search-parameters-in-2012-the-ultimate-url-manipulation-its-url/ https://airstraining.wordpress.com/2012/06/13/google-search-... There are (or at least were) others with related effects.
- wildpeaks 10y ago
  I wish there was a way to use both Verbatim and Date Range at the same time: you end up having to choose between recent but irrelevant results, or relevant but obsolete results.
Analemma_ 10y ago
Any search crawler that ignores robots.txt is going to be blocked by site operators in a hurry.
- thesmallestcat 10y ago
  If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots.txt is security by obscurity at best.
  - Analemma_ 10y ago
    People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
    - thesmallestcat 10y ago
      Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.
      - desdiv 10y ago
        Google doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
        deleted 10y ago
        [deleted]
cocktailpeanuts 10y ago
Those are all things Google can already do but decide not to. Which means if any startup attempts and gets a moderate success, google will of course decide to do the same. Your startup will be nothing more than an R&D department Google uses for free. Disruption happens by doing something completely different that happens to cannibalize the incumbent's business in a way that they can't change. All these little features mentioned are something Google can easily do so these are not really disruptive.
- thesmallestcat 10y ago
  Keep in mind that it's been 20 years since Google's differantiator (PageRank) was invented. It's been about 10 years since Google's last meaningful changes to its search UI (autocomplete and "instant" search). So, while I respect Google's omniscience and omnipotence, I disagree strongly that Google/Alphabet are in a position to respond to a serious upstart competitor. Their primary response, as you hinted would be to acquire/acquihire, which is unimaginative and not exactly how you remain on the cutting edge. That strategy falls on its face as soon as it meets that one damn company that doesn't want to be purchased.
  - cocktailpeanuts 10y ago
    There are already many "serious upstarts" that are doing just fine but will never reach the status of Google, like duckduckgo, bing, etc. Some even say bing has better search quality than google. But that's not what matters. No matter how good the product is, if you can't get distribution it doesn't matter. Also, even if you do have distribution, it may not be the right context so it won't be effective. If anyone is trying to really build a search engine to compete directly against Google, I can only say good luck, and prepare to be the most patient person on earth, because it won't happen fast, and probably won't even happen. Instead, while you're slowly making progress day by day with small traffic increase, you'll see some random new thing come out that had no intention of becoming a "search engine" (just like Youtube) and "disrupt" Google. I have seen a lot of people talk about "disrupting something", and also have seen a lot of people who actually have disrupted their industry. The former are just wannabes, and the latter never said anything about "disrupting" something. They just saw an opportunity and went for it. And it ended up disrupting. Hope this makes sense. p.s. Google doesn't operate on the original pagerank anymore, there are tons of other things going on underneath. Also they now own many innovative AND popular "search UI" both through acquisitions and their R&D. Google Maps owns a lot of the map space, Youtube owns the video search space, and so on. Look deeper and you'll find even more that you've been taking for granted.
icebraining 10y ago
Another issue is spam/false matches. Why does Google return illegitimate results? Because, let me tell you, any search for "some nifty computer book pdf" returns pages upon pages of bogus links leading to ad link mazes. A crawler should be able to trivially crawl such a page, determine that no PDF is linked, and blacklist the result, but this doesn't happen. The problem is interesting, but I think you make it seem easier than it is. What if there's a CAPTCHA, as there often is? Should the engine still ignore it? That could lead to a whole lot of missed content.
- thesmallestcat 10y ago
  I'd argue that paywalled sites, Captcha-guarded pages, things like that are second-class content and should be treated as such. If somebody wants to dig deeper and see such results, that should be an option, but as you said, this is a tough problem to solve satisfactorily and the truth lies somewhere inbetween. An obstacle to this happening right now is that Google doesn't want to shake the advertising tree too hard. Their business model is spammy so they tolerate spamminess. That's the hardest problem of all, and one I don't think anybody's close to solving yet: How do you monetize web search without automatically wrapping results in ads? But some bright person will come up with an answer and hopefully defeat this huge conflict of interest.
bambax 10y ago
> a query I've modified to quote "every" "single" "flipping" "term." Use Tools > Verbatim, it's much faster than quoting every search term, and it works better.

kjhughes 10y ago

To disable remapping of terms in Google, use verbatim search: On the search results page, choose "Search tools -> All results -> Verbatim"

dredmorbius 10y ago
There's no syntax of which I'm aware which tells Google to not "correct" or expand the results. When I'm seeking specific matches, I'm seeking really fucking specific matches. Google annoy me to no end.
- user5994461 10y ago
  +word It used to force to search the word exactly as written.
  - donretag 10y ago
    The plus/+ notation has been removed. You must know surround the term in quotes "word".
  - dredmorbius 10y ago
    "Used to" is the operative phrase. That was among the negative consequences of Google+, as +<term> notation apparently was going to be reserved or repurposed for that somehow, but never was. I've done some large-scale searching where the most relevant detail is how many results are returned, most particularly for a specific domain. (For which, incidentally, there's no handy mechanism to accomplish, so it's <array of terms> * <array of domains>, and multiplicative explosion of searches, plus about a 45s timeout per query to avoid triggering bot defenses by Google. Such as this: https://www.reddit.com/r/dredmorbius/comments/3hp41w/tracking_the_conversation_fp_global_100_thinkers/ https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...
- mtreis86 10y ago
  Boolean search would be wonderful
- gamzer 10y ago
  I believe it was possible in the past using URL parameters (e.g. complete=0 to disable auto-suggest) but it does not seem to work anymore. Here it was documented in 2012: https://airstraining.wordpress.com/2012/06/13/google-search-parameters-in-2012-the-ultimate-url-manipulation-its-url/ https://airstraining.wordpress.com/2012/06/13/google-search-... There are (or at least were) others with related effects.
wildpeaks 10y ago
I wish there was a way to use both Verbatim and Date Range at the same time: you end up having to choose between recent but irrelevant results, or relevant but obsolete results.

Analemma_ 10y ago

Any search crawler that ignores robots.txt is going to be blocked by site operators in a hurry.

thesmallestcat 10y ago
If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots.txt is security by obscurity at best.
- Analemma_ 10y ago
  People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
  - thesmallestcat 10y ago
    Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.
    - desdiv 10y ago
      Google doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
      - deleted 10y ago
        [deleted]

cocktailpeanuts 10y ago

Those are all things Google can already do but decide not to. Which means if any startup attempts and gets a moderate success, google will of course decide to do the same. Your startup will be nothing more than an R&D department Google uses for free. Disruption happens by doing something completely different that happens to cannibalize the incumbent's business in a way that they can't change. All these little features mentioned are something Google can easily do so these are not really disruptive.

thesmallestcat 10y ago
Keep in mind that it's been 20 years since Google's differantiator (PageRank) was invented. It's been about 10 years since Google's last meaningful changes to its search UI (autocomplete and "instant" search). So, while I respect Google's omniscience and omnipotence, I disagree strongly that Google/Alphabet are in a position to respond to a serious upstart competitor. Their primary response, as you hinted would be to acquire/acquihire, which is unimaginative and not exactly how you remain on the cutting edge. That strategy falls on its face as soon as it meets that one damn company that doesn't want to be purchased.
- cocktailpeanuts 10y ago
  There are already many "serious upstarts" that are doing just fine but will never reach the status of Google, like duckduckgo, bing, etc. Some even say bing has better search quality than google. But that's not what matters. No matter how good the product is, if you can't get distribution it doesn't matter. Also, even if you do have distribution, it may not be the right context so it won't be effective. If anyone is trying to really build a search engine to compete directly against Google, I can only say good luck, and prepare to be the most patient person on earth, because it won't happen fast, and probably won't even happen. Instead, while you're slowly making progress day by day with small traffic increase, you'll see some random new thing come out that had no intention of becoming a "search engine" (just like Youtube) and "disrupt" Google. I have seen a lot of people talk about "disrupting something", and also have seen a lot of people who actually have disrupted their industry. The former are just wannabes, and the latter never said anything about "disrupting" something. They just saw an opportunity and went for it. And it ended up disrupting. Hope this makes sense. p.s. Google doesn't operate on the original pagerank anymore, there are tons of other things going on underneath. Also they now own many innovative AND popular "search UI" both through acquisitions and their R&D. Google Maps owns a lot of the map space, Youtube owns the video search space, and so on. Look deeper and you'll find even more that you've been taking for granted.

icebraining 10y ago

Another issue is spam/false matches. Why does Google return illegitimate results? Because, let me tell you, any search for "some nifty computer book pdf" returns pages upon pages of bogus links leading to ad link mazes. A crawler should be able to trivially crawl such a page, determine that no PDF is linked, and blacklist the result, but this doesn't happen. The problem is interesting, but I think you make it seem easier than it is. What if there's a CAPTCHA, as there often is? Should the engine still ignore it? That could lead to a whole lot of missed content.

thesmallestcat 10y ago
I'd argue that paywalled sites, Captcha-guarded pages, things like that are second-class content and should be treated as such. If somebody wants to dig deeper and see such results, that should be an option, but as you said, this is a tough problem to solve satisfactorily and the truth lies somewhere inbetween. An obstacle to this happening right now is that Google doesn't want to shake the advertising tree too hard. Their business model is spammy so they tolerate spamminess. That's the hardest problem of all, and one I don't think anybody's close to solving yet: How do you monetize web search without automatically wrapping results in ads? But some bright person will come up with an answer and hopefully defeat this huge conflict of interest.

bambax 10y ago

> a query I've modified to quote "every" "single" "flipping" "term." Use Tools > Verbatim, it's much faster than quoting every search term, and it works better.