3 ms·
It doesn't have to be government or foreign actors. Abiding by the contents of robots.txt isn't required of anybody at all. It's merely a social convention.
by JohnFen 3y ago
It doesn't have to be government or foreign actors. Abiding by the contents of robots.txt isn't required of anybody at all. It's merely a social convention.