4 ms·
Googlebot does sometimes submit search forms in an attempt to discover new pages, though I'm surprised to hear "hundreds of thousands". Either that's an exagger
by dlubarov 8y ago
Googlebot does sometimes submit search forms in an attempt to discover new pages, though I'm surprised to hear "hundreds of thousands". Either that's an exaggeration or it was over a very long time period.
As other comments mentioned, he could have easily prevented this with robots.txt:
User-agent: *
Disallow: /search
- mynameisvlad 8y agoWould that work if the search box was on the main page, though? Presumably, GoogleBot wouldn't index the results, but is it smart enough to know that the form redirects to the disallowed page?
- greglindahl 8y agoYes, Googlebot (and everyone else, really) is smart enough to figure out if a form action is disallowed in robots.txt.