4 ms·
Always find it amusing when people write about „blocking“ requests using robots.txt as if they are deploying a firewall
by itsbjoern 2y ago
Always find it amusing when people write about „blocking“ requests using robots.txt as if they are deploying a firewall
- gitgud 2y agoAgreed, all it takes is another site to copy the content, then an LLM could just scrape that… An open web that block scraping… is likely “not an open web”