3 ms·
How does this relate to the robots.txt file? Even being deferential it doesn't make much sense to respect it.
by throwaway91111 10y ago
How does this relate to the robots.txt file? Even being deferential it doesn't make much sense to respect it.