3 ms·
> Scrappers can be dicks about it too It's not always about individual bad actors. You can have lots of small players causing problems. I wonder how many pyt
by mrkeen 3y ago
> Scrappers can be dicks about it too
It's not always about individual bad actors. You can have lots of small players causing problems. I wonder how many python developers there are right now trying to make their own offline copy of stackoverflow.com.
Wikipedia has a great defence against this. They ask you not to scrape, and at the same time, provide torrents of the data (https://meta.wikimedia.org/wiki/Data_dump_torrents https://meta.wikimedia.org/wiki/Data_dump_torrents)
- rzzzt 3y agoStack Exchange also provides one: https://archive.org/details/stackexchange https://archive.org/details/stackexchange There was a hiccup around June but that seems resolved now: https://meta.stackexchange.com/questions/389922/june-2023-data-dump-is-missing https://meta.stackexchange.com/questions/389922/june-2023-da...
- genewitch 3y agoyeah i thought this was the case. There's a couple of other sites that - i hesitate to state this completely factual phrase - leave dumps for the public to pick up.