2 ms·
> What is the difference in processing and indexing something to be returned as a full original result vs being returned in a permutation? The difference is th
by viraptor 3y ago
> What is the difference in processing and indexing something to be returned as a full original result vs being returned in a permutation?
The difference is the work done to implement that system and the cost of losing the data. The request is basically "please make your job harder and result worse because we don't like it". I'm not making a judgement here about what's right, or what should be legal. I'm saying there's zero incentive for the companies to comply here.
- righthand 3y agoExactly which makes robots.txt effectively useless. Because you can block the IPs and the models will probably use a proxy to scrape you still. All of robots.txt is based on promise there’s no incentive to actually comply.