11 ms·
I work for Amazon, but not directly on web crawling. Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazo
by cmeacham98 2y ago
I work for Amazon, but not directly on web crawling.
Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazon. Amazonbot is supposed to respect robots.txt and should always come from an Amazon-owned IP address (You can see verification steps here: https://developer.amazon.com/en/amazonbot https://developer.amazon.com/en/amazonbot).
I've forwarded this internally just in case there is some crazy internal team I'm not aware of pulling this stunt, but I would strongly suggest the author treats this traffic as malicious and lying about its user agent.
- xena 2y agoRandomly selected IPs from my logs show that 80% of them have the matching that forward confirming reverse DNS domain. The most aggressive ones were from the amazonbot domain. Believe what you want though. Search for `xeiaso.net` in ticketing if you want proof.
- petee 2y agoReverse DNS doesn't mean much, they can set it to anything; can you forward match them to any amazon domain?
- xena 2y agoIt's forward confirming reverse DNS. I assumed that everyone does that by default.
- petee 2y agoWhat everyone does by default doesn't matter really here, it's that an IP owner/user can literally set the reverse to any arbitrary domain regardless if the actual domain has a record for that IP. What matters is both match, thats all I meant
- guardiangod 2y agoSo you said the IPs are residential IP, but their reverse DNS points to a amazonbot domain? Does that even make sense?