3 ms·

> They also respect robots.txt All the reports I've heard from organizations dealing with AI crawler bots say they are not honest about their user agent and do

> They also respect robots.txt All the reports I've heard from organizations dealing with AI crawler bots say they are not honest about their user agent and do not respect robots.txt "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/ https://xeiaso.net/notes/2025/amazon-crawler/

eesmith 1y ago
Further info along the same lines at https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali... > If you think these [AI] crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic. Sourcehut (the site described) used Anubis before swithing "to go-away, which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better)." https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/ https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/
immibis 1y ago
However, there's no evidence those bots are really OpenAI et al.
grishka 1y ago
Speaking from my own experience, which is admittedly limited, but still — I had AI bots crawling my fediverse server, I added them to my robots.txt as "Disallow: *", they stopped. As I said, it might be someone else entirely using OpenAI/Amazon/Meta/etc user agents to hide their real identity while ignoring robots.txt. What's to stop them? People blame those companies anyway.
mkfs 1y ago
> "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/ https://xeiaso.net/notes/2025/amazon-crawler/ If this isn't the case, then the bot detection systems big sites are using must be pretty bad, because I do almost all of my browsing on a desktop originating from residential ASN IP address, and I routinely run up against CAPTCHAs. E.g., any Stack Exchange site on first visit, and even Amazon. What reason would there be for this, unless these crawlers are laundering their traffic through residential IPs?

eesmith 1y ago

Further info along the same lines at https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali... > If you think these [AI] crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic. Sourcehut (the site described) used Anubis before swithing "to go-away, which is more configurable and allows us to reduce the user impact of Anubis (e.g. by offering challenges that don’t require JavaScript, or support text-mode browsers better)." https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/ https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/

immibis 1y ago

However, there's no evidence those bots are really OpenAI et al.

grishka 1y ago

Speaking from my own experience, which is admittedly limited, but still — I had AI bots crawling my fediverse server, I added them to my robots.txt as "Disallow: *", they stopped. As I said, it might be someone else entirely using OpenAI/Amazon/Meta/etc user agents to hide their real identity while ignoring robots.txt. What's to stop them? People blame those companies anyway.

mkfs 1y ago

> "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/ https://xeiaso.net/notes/2025/amazon-crawler/ If this isn't the case, then the bot detection systems big sites are using must be pretty bad, because I do almost all of my browsing on a desktop originating from residential ASN IP address, and I routinely run up against CAPTCHAs. E.g., any Stack Exchange site on first visit, and even Amazon. What reason would there be for this, unless these crawlers are laundering their traffic through residential IPs?