4 ms·

So GoogleBot can't scrape more than 10 NYT articles per month, right?

• leorocky 12y ago
  Googlebot doesn't maintain a nytimes.com session, so it can crawl an unlimited number of articles same as you could if you tell your browser to not accept cookies from nytimes.com