6 ms·
If it respected the standard then a lack of a robits.txt implies do not crawl, which they openly state they ignore
by rho138 5mo ago
If it respected the standard then a lack of a robits.txt implies do not crawl, which they openly state they ignore