> "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/

> "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more." https://xeiaso.net/notes/2025/amazon-crawler/ https://xeiaso.net/notes/2025/amazon-crawler/ If this isn't the case, then the bot detection systems big sites are using must be pretty bad, because I do almost all of my browsing on a desktop originating from residential ASN IP address, and I routinely run up against CAPTCHAs. E.g., any Stack Exchange site on first visit, and even Amazon. What reason would there be for this, unless these crawlers are laundering their traffic through residential IPs?