Sites that try to use robots.txt to stay relevant are going to fail spectacularly. All their information is going to leak or be provided by other sources, and

Sites that try to use robots.txt to stay relevant are going to fail spectacularly. All their information is going to leak or be provided by other sources, and they're going to get forgotten. The only people who will be able to make a living selling data will be the ones who can produce data so unique that nobody else can claim to have created it, and for which training on it leaves an unmistakable signature that makes litigation easy.