7 ms·LLMs broadly speaking disregard robots.txt, so I don’t see why they would bother regarding this.by TimorousBestie 1y agoLLMs broadly speaking disregard robots.txt, so I don’t see why they would bother regarding this.