3 ms·
A small number of search engines respect it, no one else does. Just about every content scraping bot ignores it, including a number of Google's.
by micromacrofoot 2y ago
A small number of search engines respect it, no one else does. Just about every content scraping bot ignores it, including a number of Google's.