I have experienced the same situation with facet-like pages in the past. Links leading to (for example) the same product with a different color pre-selected on

I have experienced the same situation with facet-like pages in the past. Links leading to (for example) the same product with a different color pre-selected on page load. Or within in a listing of a category products, a link might change from price descending to price ascending. All else equal, even the crawlers don't want to re-index the same page in eighty different ways if they could avoid it. They simply don't know better, and have decided to ignore our attempt to teach them (robots.txt). In the past, we've used this behavior as a signal to identify and block bad bots. These days, they will try again from 2000 separate residential IPs before they give up. But there was a long time where egregious duplicate page views of these faceted pages (against the advice of robots.txt) made detecting certain bad bots much easier.