3 ms·
Or, you can spoof the google bot or Bing bot user agent and try to scrape the site that way.
by janmo 5y ago
Or, you can spoof the google bot or Bing bot user agent and try to scrape the site that way.
- notriddle 5y agoImpersonating Googlebot is a great way to get blocked. Real Googlebot only comes from certain IP addresses.
- AlexanderTheGr8 5y agoalmost all sites uses "host" command to see if the IP belongs to the googlebot (this is recommended way by Google). And you can easily find many IPs which return googlebot when you "host" them. So you can use any of these IP addresses to spoof GoogleBot. To find more IPs, make your own website, and wait until GoogleBot eventually shows up :)