3 ms·
Crawlers and Agents and Bots, Oh My: Time to Clarify Robots.txt
- bediger4000 2y agoIt would help if big sites like Bytedance's Bytespider actually observed robots.txt
- deleted 2y ago[deleted]
- ChrisArchitect 2y agoSnippet from "today's Wired article"? Article from Jun 27th. Get out of here Techdirt. Wired article: https://www.wired.com/story/aws-perplexity-bot-scraping-investigation/ https://www.wired.com/story/aws-perplexity-bot-scraping-inve... Related Perplexity discussions: Perplexity is a bullshit machine https://news.ycombinator.com/item?id=40728732 https://news.ycombinator.com/item?id=40728732 Perplexity's grand theft AI https://news.ycombinator.com/item?id=40819628 https://news.ycombinator.com/item?id=40819628 Perplexity AI is lying about their user agent https://news.ycombinator.com/item?id=40690898 https://news.ycombinator.com/item?id=40690898