3 ms·
Ignoring robots.txt is the only way to be a "system of record". Otherwise you don't record what actual humans see.
by account42 4y ago
Ignoring robots.txt is the only way to be a "system of record". Otherwise you don't record what actual humans see.