3 ms·
Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists. See https:
by simonw 29d ago
Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists.
See https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.
- perching_aix 29d agoIs there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.
- kees99 29d agoThis is exactly the problem, unfortunately. For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0]. Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are. [0] https://dumps.wikimedia.org/ https://dumps.wikimedia.org/
- nubinetwork 29d agoYou can tell Claude to clone from github for Linux stuff all you want... it's still going to try web, and fail, before doing what you asked it to do.