3 ms·

robots.txt is a shitshow just like user agents. It's been twisted so many ways it doesn't reliably signal actual intent any more.