3 ms·
Also a list of indexed files would be interesting. I've recently looked for a spider of a couple of large sites (for us to use for demos for potential customer
by wheels 18y ago
Also a list of indexed files would be interesting. I've recently looked for a spider of a couple of large sites (for us to use for demos for potential customers), but I'd like to know what's in there before sorting out space to decompress a 22 GB file.
I notice on the Internet Archive file that you link it mentions that those files are no longer accessible -- are there similar places that you can grab spidered content?
- gojomo 18y agoBulk data access to the historic archive (other than via the public Wayback Machine) is currently only available by special arrangement with research projects. We don't really have a good system for enabling such access, so it happens rarely, on a case-by-case basis. If you just need fresh web content, it's not hard to collect for yourself quite a bit of broad material in a short period on a small budget, with an open source crawler. The data from Dotbot might be good, or potential data feeds from Wikia Search/Grub.
- bravura 18y agoIf I'm not mistaken, wikia does not want you to download their crawls using a robot: http://soap.grub.org/arcs/robots.txt http://soap.grub.org/arcs/robots.txt
- gojomo 18y agoThey may not want automated downloads, but that robots.txt is in the wrong place. The standard only provides for robots.txt to be found and respected at the root (/robots.txt), not any subdirectory-lookalike path (/arcs/robots.txt).