4 ms·
They could just ignore it, right? Or do any new rules actually require honoring robots.txt?
by shirakawasuna 6y ago
They could just ignore it, right? Or do any new rules actually require honoring robots.txt?
- occamrazor 6y agoAll major search engines follow robots.txt