3 ms·
The only problem is that the other side already ignores robots.txt. What is the difference in processing and indexing something to be returned as a full origina
by righthand 3y ago
The only problem is that the other side already ignores robots.txt. What is the difference in processing and indexing something to be returned as a full original result vs being returned in a permutation? The point of robots.txt is to control indexing, something the LLM/ML businesses are completely ignoring already because they’re “different” and don’t have to play by the rules because AGI.
- viraptor 3y ago> What is the difference in processing and indexing something to be returned as a full original result vs being returned in a permutation? The difference is the work done to implement that system and the cost of losing the data. The request is basically "please make your job harder and result worse because we don't like it". I'm not making a judgement here about what's right, or what should be legal. I'm saying there's zero incentive for the companies to comply here.
- righthand 3y agoExactly which makes robots.txt effectively useless. Because you can block the IPs and the models will probably use a proxy to scrape you still. All of robots.txt is based on promise there’s no incentive to actually comply.