3 ms·
I wonder if you could add your POST URLs to robots.txt if you don't want the crawler to access them. If other crawlers start doing this, it should probably be
by nostromo 15y ago
I wonder if you could add your POST URLs to robots.txt if you don't want the crawler to access them.
If other crawlers start doing this, it should probably be added to robots.txt formally.
- martian 15y agoI'm an engineer at Thumbtack. And yes: we have noticed that Googlebot does seem to obey robots.txt even when issuing these AJAX requests.