2 ms·
We don't actually know what they trained on. If we want to trust them, they've always honored robots.txt, because why wouldn't they? On the other hand, if we wa
by fragmede 3y ago
We don't actually know what they trained on. If we want to trust them, they've always honored robots.txt, because why wouldn't they? On the other hand, if we want to believe they're as capricious as we want to make them out to be, they've got a copy of LibGen, and every single torrent out there to train it in on, on top of using robots.txt to find what you don't want crawled. robots.txt is great - it tells you where your evil scraper should go to get the good stuff.