4 ms·Couldn't compliance with a robots.txt file have prevented all of this?by omniglottal 4y agoCouldn't compliance with a robots.txt file have prevented all of this?