10 ms·

Amazonbot is finally respecting robots.txt

171 points by xena 5mo ago

bstsb 5mo ago
> Get Outlook for Mac this bit made me laugh. was the email drafted in Outlook? was it sent to some sort of forwarding mailbox, or did they just BCC every customer in?
- jdiff 5mo ago
  > Looking at the email headers it has a bunch of Exchange-specific headers so it's probably actually from Outlook for Mac. My guess would be some sort of internal forwarding mailing list, yeah.
- hkchad 5mo ago
  They can't get out of SES sandbox /s
jacobn 5mo ago
I just complained to them the other day! They were scraping our weather website to no end, very much including the disallowed path prefixes. Did end up just adding them to our WAF blocklist, which is weirdly ironic - hosting on their infra & using their services to block their AI scraper...
- BLKNSLVR 5mo ago
  I hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still incredibly amateur software with 'Amazon-priorities' rather than 'responsible internet traffic' priorities.
  - tardedmeme 5mo ago
    The responsible internet is dead. Every big actor on the internet is selfish now that there's money involved. And has been for 20 years. Google only respected it because blocking Google from crawling your site used to hurt you more than it hurt Google.
    - BLKNSLVR 5mo ago
      Time to switch to allow lists instead of block lists...
- adrianvi 5mo ago
  step 1: create the problem, step 2: sell the solution, step 3: profit
deleted 5mo ago
[deleted]
namegulf 5mo ago
Robots.txt is lame BTW, there is no way to enforce it. It is up to the bot to decide to crawl or not and most cases they don't care. Cloudflare had a nice technic to address the bot problem (if you use their name servers). It'll respect and use the robots.txt while sending the remaining bots to a deep black hole.
- input_sh 5mo ago
  Yes, we know, its purpose is to guide the bots, not forcibly block them. That said, one of the biggest websites in the world not respecting it is definitely a noteworthy story. Hopefully another one of the biggest websites in the world (formerly known as Twitter) eventually respects it as well instead of not even disclosing itself via a user agent and pretending to be Safari running on iOS.
  - namegulf 5mo ago
    Why down vote a comment? You're talking about one (yes, biggest) but millions of other bots don't follow must be a bigger story.
- llbbdd 5mo ago
  Yeah, robots.txt is a great herald example of the type of solution invented by people who don't understand incentives whatsoever.
  - Ferret7446 5mo ago
    robots.txt is a great herald example of people misunderstanding and misusing a tool. The file was designed to help crawlers, by pointing them to the most valuable to index content and help them avoid wasting resources on useless pages. The people trying to use it to block or limit bots are uninformed and/or misinformed.
- marginalia_nu 5mo ago
  Robots.txt is great if you're trying to run an above board operation. Much easier than trying to guess how a webmaster wishes the crawler to behave, and then getting angry emails when you guess wrong.
arjie 5mo ago
Huh, I get a lot of traffic from Amazonbot (relative to humans) and try as I might, it would get stuck in a tarpit of no creation because it would sit there and keep blasting every variation of my recent pages because Mediawiki lists many links. I have them appropriately nofollow and warning the bot not to waste its time with robots.txt but it just goes and sticks itself on nonsense internal pages. The traffic isn't a problem. I've got Cloudflare in front and the machine itself is relatively overpowered, and downtime isn't critical. But I'd just like the thing to be able to spider me properly. Someone did point out to me that maybe I wasn't receiving actual Amazonbot but some other spider: https://news.ycombinator.com/item?id=46352723 https://news.ycombinator.com/item?id=46352723
TurdF3rguson 5mo ago
Why does Amazonbot even exist, can someone explain? I don't understand why an ecommerce play would be crawling other websites.
- embedding-shape 5mo ago
  Amazonbot is specifically the user agent they use for crawling for "provide more accurate information to customers" (whatever that means, could be anything it sounds like) and also when they scrape for data used in AI training, according to https://developer.amazon.com/amazonbot https://developer.amazon.com/amazonbot
- reaperducer 5mo ago
  AI. Gotta slurp the world.
- input_sh 5mo ago
  To train AI. Not even a hyperbole, that is the only concrete example they list in their explanation: https://developer.amazon.com/amazonbot https://developer.amazon.com/amazonbot > Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models.
  - tardedmeme 5mo ago
    It would be more fun to respond with false data.
- tintor 5mo ago
  To ensure Amazon marketplace sellers aren't offering lower prices on other ecommerce websites. Also AI.
  - b112 5mo ago
    I was wondering about this. And it makes me think this is all mistruth, unless they plan to drop this pricing tactic. They've been getting some heat on it lately, but I find it hard to believe they're going to give up entirely? And if so, what's to stop someone from just flouting their rules on pricing, and then doing the robots.txt thing to prevent issues?
captn3m0 5mo ago
Good place to ask, saw a new AWS User agent in logs today: Amazon-Quick-on-Behalf-of-$HEXID I found a mention on some user agent trackers but no official documentation. Anyone knows if it’s documented? Asking because I am seeing decent traffic (30GB/week) from this.
- iLoveOncall 5mo ago
  Amazon Quick is the new name of Quicksight, which is the BI tool from AWS. It has AI agents included so I guess this can just come from it searching the web based on user requests.
- embedding-shape 5mo ago
  Came across this recently too, seems to be from "Amazon Quick" where crawling other's websites is basically a feature of the product: https://docs.aws.amazon.com/quick/latest/userguide/web-crawler-integration.html https://docs.aws.amazon.com/quick/latest/userguide/web-crawl... > Crawling behavior [...] Crawler identification: Identifies itself with user-agent string "aws-quick-on-behalf-of-<UUID>" in request headers. Maybe people found a way of using it as a loophole for something or Amazon Quick is just picking up in usage, and your website is popular amongst whoever uses that sort of stuff.
phdelightful 5mo ago
I just put Anubis in front of my self-hosted forge this morning because AmazonBot had helped itself to 750 GiB (!) of traffic to my public repos this month! At least, it claimed to be AmazonBot…
- nathanmills 5mo ago
  Do you have a robots.txt?
  - xena 5mo ago
    > We are writing to inform you that starting Monday, June 15, 2026, crawl preferences for Amazonbot will be managed solely through the industry-standard directives. They will in the future, but not today.
- Bender 5mo ago
  Are they in this space? [1] One could map the ranges into a web daemon and rate limit them or just 'ip route add blackhole ${cidr}' each cidr block. [1] - https://ip-ranges.amazonaws.com/ip-ranges.json https://ip-ranges.amazonaws.com/ip-ranges.json
  - rnhmjoj 5mo ago
    I just do this for the IP ranges of Amazon, OpenAI, Huawei and other companies that run these insane crawlers: it's 100% effective and it doesn't annoy real users with a captcha or some PoW thing. There's simply no reason for them to reach my homeserver other than to scrape the hell out of it.
  - lofaszvanitt 5mo ago
    That list is a tad bit too long. Why don't they enforce a rule on these big corps to publicly state which range does what.
    - Bender 5mo ago
      That would indeed by handy but I think the answer is that people would block specific ranges. By not segmenting into specific groups people are forced to either: - play the game of whack-a-mole - use difficult implementations of user validation checks that potentially cause pain for real humans - block all Amazon CIDR blocks which they know most corporations will not do. This forces the majority to just tolerate whatever comes out of their networks.
vindin 5mo ago
robots.txt is merely a gentleman’s courtesy at this point. Nobody is obligated to follow it.
- c-hendricks 5mo ago
  always_has_been.jpg
TrackerFF 5mo ago
Is it just me, or is it extra unethical and self-serving when crawlers from say Amazon(Bot) decides to incessantly crawl AWS hosted websites? Same goes for Google and Microsoft crawlers crawling GC and Azure. By that, I mean the types of crawls that can hog up significant usage.
rho138 5mo ago
If it respected the standard then a lack of a robits.txt implies do not crawl, which they openly state they ignore
faangguyindia 5mo ago
if you run Meta Ads, it's notorious for ddosing your website with bots. Basically, their ad manager sends dozens of click for each variant of ad you post.
- lofaszvanitt 5mo ago
  to exhaust your purse or why?
The_President 5mo ago
I identify bots by known address range. Logs are full of Google Bot scanning for resources and endpoints that don't exist. Further, it is using proxy scanners from other geolocations, to identify potential resource availability by region. These proxied scans hit the same non-existant endpoint from a dozen nodes within 2 seconds of the initial request.

171 points by xena 5mo ago

bstsb 5mo ago

> Get Outlook for Mac this bit made me laugh. was the email drafted in Outlook? was it sent to some sort of forwarding mailbox, or did they just BCC every customer in?

jdiff 5mo ago
> Looking at the email headers it has a bunch of Exchange-specific headers so it's probably actually from Outlook for Mac. My guess would be some sort of internal forwarding mailing list, yeah.
hkchad 5mo ago
They can't get out of SES sandbox /s

jacobn 5mo ago

I just complained to them the other day! They were scraping our weather website to no end, very much including the disallowed path prefixes. Did end up just adding them to our WAF blocklist, which is weirdly ironic - hosting on their infra & using their services to block their AI scraper...

BLKNSLVR 5mo ago
I hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still incredibly amateur software with 'Amazon-priorities' rather than 'responsible internet traffic' priorities.
- tardedmeme 5mo ago
  The responsible internet is dead. Every big actor on the internet is selfish now that there's money involved. And has been for 20 years. Google only respected it because blocking Google from crawling your site used to hurt you more than it hurt Google.
  - BLKNSLVR 5mo ago
    Time to switch to allow lists instead of block lists...
adrianvi 5mo ago
step 1: create the problem, step 2: sell the solution, step 3: profit

deleted 5mo ago

[deleted]

namegulf 5mo ago

Robots.txt is lame BTW, there is no way to enforce it. It is up to the bot to decide to crawl or not and most cases they don't care. Cloudflare had a nice technic to address the bot problem (if you use their name servers). It'll respect and use the robots.txt while sending the remaining bots to a deep black hole.

input_sh 5mo ago
Yes, we know, its purpose is to guide the bots, not forcibly block them. That said, one of the biggest websites in the world not respecting it is definitely a noteworthy story. Hopefully another one of the biggest websites in the world (formerly known as Twitter) eventually respects it as well instead of not even disclosing itself via a user agent and pretending to be Safari running on iOS.
- namegulf 5mo ago
  Why down vote a comment? You're talking about one (yes, biggest) but millions of other bots don't follow must be a bigger story.
llbbdd 5mo ago
Yeah, robots.txt is a great herald example of the type of solution invented by people who don't understand incentives whatsoever.
- Ferret7446 5mo ago
  robots.txt is a great herald example of people misunderstanding and misusing a tool. The file was designed to help crawlers, by pointing them to the most valuable to index content and help them avoid wasting resources on useless pages. The people trying to use it to block or limit bots are uninformed and/or misinformed.
marginalia_nu 5mo ago
Robots.txt is great if you're trying to run an above board operation. Much easier than trying to guess how a webmaster wishes the crawler to behave, and then getting angry emails when you guess wrong.

arjie 5mo ago

Huh, I get a lot of traffic from Amazonbot (relative to humans) and try as I might, it would get stuck in a tarpit of no creation because it would sit there and keep blasting every variation of my recent pages because Mediawiki lists many links. I have them appropriately nofollow and warning the bot not to waste its time with robots.txt but it just goes and sticks itself on nonsense internal pages. The traffic isn't a problem. I've got Cloudflare in front and the machine itself is relatively overpowered, and downtime isn't critical. But I'd just like the thing to be able to spider me properly. Someone did point out to me that maybe I wasn't receiving actual Amazonbot but some other spider: https://news.ycombinator.com/item?id=46352723 https://news.ycombinator.com/item?id=46352723

TurdF3rguson 5mo ago

Why does Amazonbot even exist, can someone explain? I don't understand why an ecommerce play would be crawling other websites.

embedding-shape 5mo ago
Amazonbot is specifically the user agent they use for crawling for "provide more accurate information to customers" (whatever that means, could be anything it sounds like) and also when they scrape for data used in AI training, according to https://developer.amazon.com/amazonbot https://developer.amazon.com/amazonbot
reaperducer 5mo ago
AI. Gotta slurp the world.
input_sh 5mo ago
To train AI. Not even a hyperbole, that is the only concrete example they list in their explanation: https://developer.amazon.com/amazonbot https://developer.amazon.com/amazonbot > Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models.
- tardedmeme 5mo ago
  It would be more fun to respond with false data.
tintor 5mo ago
To ensure Amazon marketplace sellers aren't offering lower prices on other ecommerce websites. Also AI.
- b112 5mo ago
  I was wondering about this. And it makes me think this is all mistruth, unless they plan to drop this pricing tactic. They've been getting some heat on it lately, but I find it hard to believe they're going to give up entirely? And if so, what's to stop someone from just flouting their rules on pricing, and then doing the robots.txt thing to prevent issues?

captn3m0 5mo ago

Good place to ask, saw a new AWS User agent in logs today: Amazon-Quick-on-Behalf-of-$HEXID I found a mention on some user agent trackers but no official documentation. Anyone knows if it’s documented? Asking because I am seeing decent traffic (30GB/week) from this.

iLoveOncall 5mo ago
Amazon Quick is the new name of Quicksight, which is the BI tool from AWS. It has AI agents included so I guess this can just come from it searching the web based on user requests.
embedding-shape 5mo ago
Came across this recently too, seems to be from "Amazon Quick" where crawling other's websites is basically a feature of the product: https://docs.aws.amazon.com/quick/latest/userguide/web-crawler-integration.html https://docs.aws.amazon.com/quick/latest/userguide/web-crawl... > Crawling behavior [...] Crawler identification: Identifies itself with user-agent string "aws-quick-on-behalf-of-<UUID>" in request headers. Maybe people found a way of using it as a loophole for something or Amazon Quick is just picking up in usage, and your website is popular amongst whoever uses that sort of stuff.

phdelightful 5mo ago

I just put Anubis in front of my self-hosted forge this morning because AmazonBot had helped itself to 750 GiB (!) of traffic to my public repos this month! At least, it claimed to be AmazonBot…

nathanmills 5mo ago
Do you have a robots.txt?
- xena 5mo ago
  > We are writing to inform you that starting Monday, June 15, 2026, crawl preferences for Amazonbot will be managed solely through the industry-standard directives. They will in the future, but not today.
Bender 5mo ago
Are they in this space? [1] One could map the ranges into a web daemon and rate limit them or just 'ip route add blackhole ${cidr}' each cidr block. [1] - https://ip-ranges.amazonaws.com/ip-ranges.json https://ip-ranges.amazonaws.com/ip-ranges.json
- rnhmjoj 5mo ago
  I just do this for the IP ranges of Amazon, OpenAI, Huawei and other companies that run these insane crawlers: it's 100% effective and it doesn't annoy real users with a captcha or some PoW thing. There's simply no reason for them to reach my homeserver other than to scrape the hell out of it.
- lofaszvanitt 5mo ago
  That list is a tad bit too long. Why don't they enforce a rule on these big corps to publicly state which range does what.
  - Bender 5mo ago
    That would indeed by handy but I think the answer is that people would block specific ranges. By not segmenting into specific groups people are forced to either: - play the game of whack-a-mole - use difficult implementations of user validation checks that potentially cause pain for real humans - block all Amazon CIDR blocks which they know most corporations will not do. This forces the majority to just tolerate whatever comes out of their networks.

vindin 5mo ago

robots.txt is merely a gentleman’s courtesy at this point. Nobody is obligated to follow it.

c-hendricks 5mo ago
always_has_been.jpg

TrackerFF 5mo ago

Is it just me, or is it extra unethical and self-serving when crawlers from say Amazon(Bot) decides to incessantly crawl AWS hosted websites? Same goes for Google and Microsoft crawlers crawling GC and Azure. By that, I mean the types of crawls that can hog up significant usage.

rho138 5mo ago

If it respected the standard then a lack of a robits.txt implies do not crawl, which they openly state they ignore

faangguyindia 5mo ago

if you run Meta Ads, it's notorious for ddosing your website with bots. Basically, their ad manager sends dozens of click for each variant of ad you post.

lofaszvanitt 5mo ago
to exhaust your purse or why?

The_President 5mo ago

I identify bots by known address range. Logs are full of Google Bot scanning for resources and endpoints that don't exist. Further, it is using proxy scanners from other geolocations, to identify potential resource availability by region. These proxied scans hit the same non-existant endpoint from a dozen nodes within 2 seconds of the initial request.