2 ms·
Bulk data access to the historic archive (other than via the public Wayback Machine) is currently only available by special arrangement with research projects.
by gojomo 18y ago
Bulk data access to the historic archive (other than via the public Wayback Machine) is currently only available by special arrangement with research projects. We don't really have a good system for enabling such access, so it happens rarely, on a case-by-case basis.
If you just need fresh web content, it's not hard to collect for yourself quite a bit of broad material in a short period on a small budget, with an open source crawler.
The data from Dotbot might be good, or potential data feeds from Wikia Search/Grub.
- bravura 18y agoIf I'm not mistaken, wikia does not want you to download their crawls using a robot: http://soap.grub.org/arcs/robots.txt http://soap.grub.org/arcs/robots.txt
- gojomo 18y agoThey may not want automated downloads, but that robots.txt is in the wrong place. The standard only provides for robots.txt to be found and respected at the root (/robots.txt), not any subdirectory-lookalike path (/arcs/robots.txt).