I doubt those crawler rules will be honoured for long. I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whethe

I doubt those crawler rules will be honoured for long. I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whether accessed directly or via these models. It’s too important an innovation to stall, especially considering the US’s competitors (China) won’t respect robots.txt either.