3 ms·

robots.txt is nothing more than a request. It's essentially a "no dog on lawn" sign in the yard of a vacation home.