3 ms·

Reddit, twitter, facebook are just three to start. There are plenty that disallow crawlers except google. We've crawled a significant amount now and just becaus

by Grimm1 6y ago

Reddit, twitter, facebook are just three to start. There are plenty that disallow crawlers except google. We've crawled a significant amount now and just because you are unaware of them doesn't mean they don't exist. I can attest they're there. I'll also add plenty of sites don't block any engine but confer special privileges to google bot which depending on the site and their size are almost the same thing. Edit: And I'll add to limit confusion Reddit hides the sitemap and denies access there's is not an outright ban -- it just makes it a lot harder.

oh_sigh 6y ago
Can't you just set your useragent to googlebot? Or something which isn't googlebot but which matches the most common regexes like "Not-Googlebot/2.1"
- Grimm1 6y ago
  A lot of sites do some variation of these when you set googlebot as your UA, certainly the larger more sophisticated sites do. https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not. There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
  - ericd 6y ago
    wrt legal cases, are you referring to HiQ?
    - Grimm1 6y ago
      I am
- ricardo81 6y ago
  That's "poor man's cloaking". Most people that genuinely care if it's Googlebot or not will verify it appropriately by doing a DNS lookup and a reverse DNS of that IP to ensure it's one of Google's IPs. 'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.
threatofrain 6y ago
Content kingdoms have their own reasons to be hostile to Everyone searching, including Google. Even when their content is "searchable" by Google, they'll tease you with something and gate almost all content. They are part of the story of why Google is degrading, not why they're doing well.

by Grimm1 6y ago

oh_sigh 6y ago

Can't you just set your useragent to googlebot? Or something which isn't googlebot but which matches the most common regexes like "Not-Googlebot/2.1"

Grimm1 6y ago
A lot of sites do some variation of these when you set googlebot as your UA, certainly the larger more sophisticated sites do. https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not. There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
- ericd 6y ago
  wrt legal cases, are you referring to HiQ?
  - Grimm1 6y ago
    I am
ricardo81 6y ago
That's "poor man's cloaking". Most people that genuinely care if it's Googlebot or not will verify it appropriately by doing a DNS lookup and a reverse DNS of that IP to ensure it's one of Google's IPs. 'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.

threatofrain 6y ago

Content kingdoms have their own reasons to be hostile to Everyone searching, including Google. Even when their content is "searchable" by Google, they'll tease you with something and gate almost all content. They are part of the story of why Google is degrading, not why they're doing well.