3 ms·

What's so odd about that? All bots get a long list of directories to not crawl, and only GBot-Image is blocked from the whole site? Seems quite normal for a la

What's so odd about that? All bots get a long list of directories to not crawl, and only GBot-Image is blocked from the whole site? Seems quite normal for a large website like Oracle's.

brazzy 16y ago
It's not even "only" GoogleBot-Image that's completely blocked, there are quite a few others.
- pierrefar 16y ago
  Yeah, sorry, bad phrasing. My "only" was for the "only Google bot", not "only bot". Most of the others that are completely blocked are just bad behaving or spammy bots.