exactly, but nobody in their sane minds would be using robots.txt to block crawlers as that is SEO (organic traffic from searches) disaster. People could be ad

exactly, but nobody in their sane minds would be using robots.txt to block crawlers as that is SEO (organic traffic from searches) disaster. People could be adding a specific robot user-agent, if they knew openai even existed before yesterday and was stealing their content. but nobody did.