4 ms·
It's a shame that most companies that ignore robots.txt also operate horribly behaved crawlers. I feel like they forfeit the moral high ground to render judgeme
by josefritzishere 1y ago
It's a shame that most companies that ignore robots.txt also operate horribly behaved crawlers. I feel like they forfeit the moral high ground to render judgement.