7 ms·
I'm an engineer at Thumbtack. And yes: we have noticed that Googlebot does seem to obey robots.txt even when issuing these AJAX requests.
by martian 15y ago
I'm an engineer at Thumbtack. And yes: we have noticed that Googlebot does seem to obey robots.txt even when issuing these AJAX requests.