3 ms·Isn't this what robots.txt was created to convey?by jimktrains2 8y agoIsn't this what robots.txt was created to convey?mc32 8y agoYes. But not all scrapers and indexers respect that? So it’s still a problem.baroffoos 8y agoRequire a login/invite then