6 ms·

There are structural issues that make it very difficult for new competitors to gain traction in search. For example, many website owners are now hostile to new

by ericd 6y ago

There are structural issues that make it very difficult for new competitors to gain traction in search. For example, many website owners are now hostile to new web crawlers, but they're happy to allow Google to hammer their servers because they want that sweet, sweet search traffic. Mandating a commonly accessible crawl, with cached versions of the pages, would help new entrants a lot. Also, there're large network effects with ad networks. It seems unlikely that many marketing managers are going to take the time to do targeted keyword queries on your search engine with 1/1,000,000 the traffic of Google.

fastball 6y ago
I don't know a single website owner that is hostile to any search engine web crawler, unless that web crawler is slamming them with so many requests they're effectively getting DDoS'd.
- Grimm1 6y ago
  Reddit, twitter, facebook are just three to start. There are plenty that disallow crawlers except google. We've crawled a significant amount now and just because you are unaware of them doesn't mean they don't exist. I can attest they're there. I'll also add plenty of sites don't block any engine but confer special privileges to google bot which depending on the site and their size are almost the same thing. Edit: And I'll add to limit confusion Reddit hides the sitemap and denies access there's is not an outright ban -- it just makes it a lot harder.
  - oh_sigh 6y ago
    Can't you just set your useragent to googlebot? Or something which isn't googlebot but which matches the most common regexes like "Not-Googlebot/2.1"
    - Grimm1 6y ago
      A lot of sites do some variation of these when you set googlebot as your UA, certainly the larger more sophisticated sites do. https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not. There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
      - ericd 6y ago
        wrt legal cases, are you referring to HiQ?
        Grimm1 6y ago
        I am
    - ricardo81 6y ago
      That's "poor man's cloaking". Most people that genuinely care if it's Googlebot or not will verify it appropriately by doing a DNS lookup and a reverse DNS of that IP to ensure it's one of Google's IPs. 'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.
  - threatofrain 6y ago
    Content kingdoms have their own reasons to be hostile to Everyone searching, including Google. Even when their content is "searchable" by Google, they'll tease you with something and gate almost all content. They are part of the story of why Google is degrading, not why they're doing well.
- temp667 6y ago
  I do, some third party search crawlers are just badly programmed, and after you get burned a bunch of times you just want to deny anyone who isn't one of the main players. I think they are basically startups with a lot of money to spend on crawl compute, but who haven't really figured out their crawl engine, and it can go wild on your site. You also have bots that seem to be credential stuffing, bots that seem to be content scrapping (stuff with same typos shows up elsewhere after their visits, really obvious on new / fresh articles, bots that seem to be exploring for copyright claims, rando bots (maybe comment sentiment analysis for stock trading) etc. Google is much more welcome by comparison.
  - fastball 6y ago
    Right, but that's not Google's fault, that's explicitly everyone who does a shit job of crawling's fault. Generally I wouldn't qualify a lot of those as search engine web crawlers but more web scrapers looking to re-use data, not just surface it.
- kobalsky 6y ago
  Try to crawl amazon.
  - fastball 6y ago
    Right, but that’s not really anti-search-engine, that’s anti-people-who-want-our-actual-data. Like I said in another comment, if you’re someone that just wants to surface Amazon results they love you. It’s the people that want to take advantage of amazon data in some other way they’re trying to stop.
ColinHayhurst 6y ago
We run a search engine crawler at mojeek and have no hostility problems
- Grimm1 6y ago
  Hey we do too at Whize, surely you've seen sites that give special privilege to Google and or lock the sitemap away. While I don't know if I'd say "hostility" I would say passive aggressive to other crawlers.
- ericd 6y ago
  Oh, good to know, thanks. Are you running relatively complete crawls of larger sites at similar rates to Googlebot? I've definitely seen my share of robots.txt that give special permission to Googlebot, but maybe my corner of the web was unusually aggressive toward crawlers.
  - ColinHayhurst 6y ago
    No, not complete crawls of any sites, we've aimed to get a wider coverage in place of deeper crawls. This means we have most sites indexed and are now gradually going deeper. We do encounter robots.txt blocks, but we don't see it as a major issue right now. We have an API, in case of interest.
gjs278 6y ago
made up nonsense

by ericd 6y ago

fastball 6y ago

I don't know a single website owner that is hostile to any search engine web crawler, unless that web crawler is slamming them with so many requests they're effectively getting DDoS'd.

Grimm1 6y ago
Reddit, twitter, facebook are just three to start. There are plenty that disallow crawlers except google. We've crawled a significant amount now and just because you are unaware of them doesn't mean they don't exist. I can attest they're there. I'll also add plenty of sites don't block any engine but confer special privileges to google bot which depending on the site and their size are almost the same thing. Edit: And I'll add to limit confusion Reddit hides the sitemap and denies access there's is not an outright ban -- it just makes it a lot harder.
- oh_sigh 6y ago
  Can't you just set your useragent to googlebot? Or something which isn't googlebot but which matches the most common regexes like "Not-Googlebot/2.1"
  - Grimm1 6y ago
    A lot of sites do some variation of these when you set googlebot as your UA, certainly the larger more sophisticated sites do. https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not. There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
    - ericd 6y ago
      wrt legal cases, are you referring to HiQ?
      - Grimm1 6y ago
        I am
  - ricardo81 6y ago
    That's "poor man's cloaking". Most people that genuinely care if it's Googlebot or not will verify it appropriately by doing a DNS lookup and a reverse DNS of that IP to ensure it's one of Google's IPs. 'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.
- threatofrain 6y ago
  Content kingdoms have their own reasons to be hostile to Everyone searching, including Google. Even when their content is "searchable" by Google, they'll tease you with something and gate almost all content. They are part of the story of why Google is degrading, not why they're doing well.
temp667 6y ago
I do, some third party search crawlers are just badly programmed, and after you get burned a bunch of times you just want to deny anyone who isn't one of the main players. I think they are basically startups with a lot of money to spend on crawl compute, but who haven't really figured out their crawl engine, and it can go wild on your site. You also have bots that seem to be credential stuffing, bots that seem to be content scrapping (stuff with same typos shows up elsewhere after their visits, really obvious on new / fresh articles, bots that seem to be exploring for copyright claims, rando bots (maybe comment sentiment analysis for stock trading) etc. Google is much more welcome by comparison.
- fastball 6y ago
  Right, but that's not Google's fault, that's explicitly everyone who does a shit job of crawling's fault. Generally I wouldn't qualify a lot of those as search engine web crawlers but more web scrapers looking to re-use data, not just surface it.
kobalsky 6y ago
Try to crawl amazon.
- fastball 6y ago
  Right, but that’s not really anti-search-engine, that’s anti-people-who-want-our-actual-data. Like I said in another comment, if you’re someone that just wants to surface Amazon results they love you. It’s the people that want to take advantage of amazon data in some other way they’re trying to stop.

ColinHayhurst 6y ago

We run a search engine crawler at mojeek and have no hostility problems

Grimm1 6y ago
Hey we do too at Whize, surely you've seen sites that give special privilege to Google and or lock the sitemap away. While I don't know if I'd say "hostility" I would say passive aggressive to other crawlers.
ericd 6y ago
Oh, good to know, thanks. Are you running relatively complete crawls of larger sites at similar rates to Googlebot? I've definitely seen my share of robots.txt that give special permission to Googlebot, but maybe my corner of the web was unusually aggressive toward crawlers.
- ColinHayhurst 6y ago
  No, not complete crawls of any sites, we've aimed to get a wider coverage in place of deeper crawls. This means we have most sites indexed and are now gradually going deeper. We do encounter robots.txt blocks, but we don't see it as a major issue right now. We have an API, in case of interest.

gjs278 6y ago

made up nonsense