3 ms·
I mean, that's just not true.
by dabeeeenster 1y ago
I mean, that's just not true.
- esskay 1y agoWhich part? It's widely established and known that many AI crawlers are ignoring the robots.txt file, perplexity being one of them [1] [1]https://www.tomshardware.com/tech-industry/artificial-intelligence/several-ai-companies-said-to-be-ignoring-robots-dot-txt-exclusion-scraping-content-without-permission-report https://www.tomshardware.com/tech-industry/artificial-intell...