3 ms·

I don't understand. If this virtual machine: - Downloads dying web pages, and - Uploads them to the Wayback Machine at the Internet Archive, That means I'm n

by gcr 14y ago

I don't understand. If this virtual machine: - Downloads dying web pages, and - Uploads them to the Wayback Machine at the Internet Archive, That means I'm not saving the Internet Archive any bandwidth at all, and this is no more efficient than them just downloading the site themselves. The virtual machine itself is 174MB. This times however many volunteers means distributing the virtual machine is probably more stressful than the actual archiving operation.

mark_olson 14y ago
Yahoo and others either rate limit or do temporary IP bans if you access too many pages too quickly. Distributing the tools through a VM distributes the workload in a pretty easy-to-setup way.
pronoiac 14y ago
The Internet Archive crawler is polite, going at a slow rate. With a deadline looming so soon, a more aggressive effort is called for.
lhl 14y ago
As mentioned, the issue is getting around YDOD throttling (and that the shutdown is happening in 10 days). If you have proper boxen, you don't need the VM. Here's how to get it up and running on a clean Ubuntu setup for example: sudo apt-get install build-essential -y sudo apt-get install git -y sudo apt-get install libgnutls-dev -y sudo apt-get install liblua5.1-dev -y sudo apt-get install python-distribute -y sudo easy_install pip sudo pip install seesaw git clone https://github.com/ArchiveTeam/yahoo-upcoming-grab.git cd yahoo-upcoming-grab ./get-wget-lua.sh run-pipeline pipeline.py [YOURNAMEHERE]
- bdonlan 14y ago
  Does that download posterous as well, or only upcoming?
  - lhl 14y ago
    For Posterous you'd need to check out https://github.com/ArchiveTeam/posterous-grab https://github.com/ArchiveTeam/posterous-grab and run that particular script I believe.
deleted 14y ago
[deleted]
nwh 14y ago
It does actually save quite a lot of bandwidth in the process. The giles are gzipped as they are uploaded to the Archive Teams server, so your virtual machine will do a lot more download than it does upload.

by gcr 14y ago

mark_olson 14y ago

Yahoo and others either rate limit or do temporary IP bans if you access too many pages too quickly. Distributing the tools through a VM distributes the workload in a pretty easy-to-setup way.

pronoiac 14y ago

The Internet Archive crawler is polite, going at a slow rate. With a deadline looming so soon, a more aggressive effort is called for.

lhl 14y ago

As mentioned, the issue is getting around YDOD throttling (and that the shutdown is happening in 10 days). If you have proper boxen, you don't need the VM. Here's how to get it up and running on a clean Ubuntu setup for example: sudo apt-get install build-essential -y sudo apt-get install git -y sudo apt-get install libgnutls-dev -y sudo apt-get install liblua5.1-dev -y sudo apt-get install python-distribute -y sudo easy_install pip sudo pip install seesaw git clone https://github.com/ArchiveTeam/yahoo-upcoming-grab.git cd yahoo-upcoming-grab ./get-wget-lua.sh run-pipeline pipeline.py [YOURNAMEHERE]

bdonlan 14y ago
Does that download posterous as well, or only upcoming?
- lhl 14y ago
  For Posterous you'd need to check out https://github.com/ArchiveTeam/posterous-grab https://github.com/ArchiveTeam/posterous-grab and run that particular script I believe.

deleted 14y ago

[deleted]

nwh 14y ago

It does actually save quite a lot of bandwidth in the process. The giles are gzipped as they are uploaded to the Archive Teams server, so your virtual machine will do a lot more download than it does upload.