5 ms·

ArchiveTeam + Yahoo Messages Shuttering + EC2 Spot Instances = MegaCrawl

74 points by duggan 14y ago

vitovito 14y ago
When your spot instance gets killed mid-download, does the Warrior system handle that and re-assign it to someone else immediately? Or was your spot instance assigned some URLs to download with the assumption that your Warrior would be reliable, and now they won't get reassigned until they check them all at the end, which may be too late?
- Cameron_D 14y ago
  They will be reassigned later on
- deleted 14y ago
  [deleted]
lignuist 14y ago
I always hate it, when companies remove user generated content from the internet. Why doesn't Yahoo just send some Dvds with the content to Archive.org?
- knackers 14y ago
  I was thinking the same thing whilst reading the article. There are surely people at Yahoo! who would gladly help if asked.
  - Cameron_D 14y ago
    I'm sure there would be people in Yahoo would want to help, the problem is actually finding and getting in contact with someone who does care.
    - seanp2k2 14y ago
      ...then clearing it with legal and getting approval within the org to actually /do/ this. AFAIK, this would be unprecedented, but would probably win Yahoo! a lot of fans when they need it most. I have no real hope for them or anyone in a similar position to provide user data back to community custodians.
      - ersii 14y ago
        .. as long as you forget all the other user-generated-content sites Yahoo has closed over the years.
conroy 14y ago
If you have boto installed, this is all you'll need to do import boto.ec2 conn = boto.ec2.connect_to_region("us-east-1") conn.request_spot_instances('0.005', 'ami-2400984d', instance_type='t1.micro', user_data='USERNAME')
JustARandomGuy 14y ago
Is there a way we can donate to this? If you can supply a Paypal account, I'd gladly throw in a few bucks.
- MichaelStubbs 14y ago
  Their wiki answers your question: http://archiveteam.org/index.php?title=Posterous#Can_I_donate_some_cash_instead.3F http://archiveteam.org/index.php?title=Posterous#Can_I_donat...
duggan 14y ago
For those having trouble with the EC2 instructions, I thought I'd point out that I think that the Archive Warrior[0] (which is much easier to get up and running on your laptop, etc) running over my tethered cellphone is my most performant client. Yahoo don't appear to rate limit mobile devices / IP blocks as aggressively as everything else (probably because cellular providers tend to have many customers behind one IP). [0] http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri...
- duggan 14y ago
  I've also made the image available in all regions, for those who want to run in additional regions: N. Virginia: ami-2400984d Ireland: ami-d8d2d8ac Tokyo: ami-a361e1a2 Singapore: ami-6e703c3c Sydney: ami-4e0e9f74 Sao Paolo: ami-9d7aa180 N. California: ami-94f6dbd1 Oregon: ami-cf9206ff
seanp2k2 14y ago
Happy to help, but it looks like we're getting throttled hard by Yahoo. Anyone have a contact over there who they can ping about this?
Auguste 14y ago
We're currently at 16,000+ items and 61 GB uploaded. Nice work. Edit: 25 minutes later and we are at 18,700+ items and 67 GB uploaded. Distributed computing at its finest!
kogir 14y ago
Is this whole archival movement the Internet equivalent of hoarding? When is it ok to clean house?
- cleverjake 14y ago
  When bits are essentially free, there isn't much of a reason not to. Could you imagine how fascinating it would be to be able to dive into the everyday culture of 100, or even 1000 years ago? The anthropological impacts of archiving day to day life is huge. For once, history may be written by facts, rather than the victors. That may be a bit naïve, but who cares if they aren't hurting anyone.
- sp332 14y ago
  Several groups tried to salvage as much of Geocities as possible before Yahoo killed it. They got most of it (about 1 terabyte) and you can fix most geocities.com links by changing them to point at reocities.com instead. The main reason is that it's user data and deleting it is rude. Second is that you can do interesting analysis on a terabyte of user accounts during the boom of the internet. http://contemporary-home-computing.org/1tb/archives/3297 http://contemporary-home-computing.org/1tb/archives/3297 The third is that it's history! NSFW example from geocities http://contemporary-home-computing.org/1tb/archives/2736 http://contemporary-home-computing.org/1tb/archives/2736
pronoiac 14y ago
If you'd rather run a script than a vm, check out: https://github.com/ArchiveTeam/yahoomessages-grab https://github.com/ArchiveTeam/yahoomessages-grab
- sp332 14y ago
  That works for this project, but having the warrior makes it easy to join other ArchiveTeam projects now and in the future.
tgeek 14y ago
I built 2 CloudFormation templates to allow you to easily spin up a ton of these things across multiple availability zones. It uses the Amazon Linux AMI that exists in each region instead of the ones listed, and builds up the dependencies and the application on the fly. You can run either of these two templates in any region and it should just work. Download one of these files: With a keypair ( so you can login to the host) http://files.wordsaboutbytes.com/yahoo-messages-save.cf.txt http://files.wordsaboutbytes.com/yahoo-messages-save.cf.txt Without a keypair ( can’t log in locally, but it will run) http://files.wordsaboutbytes.com/yahoo-messages-save-nokeypair.cf.txt http://files.wordsaboutbytes.com/yahoo-messages-save-nokeypa... Then: 1. Open the console ( https://console.aws.amazon.com https://console.aws.amazon.com ) 2. Go to CloudFormation 3. Give your stack a name 4. Browse and select the file you downloaded from above 5. Click Next. 6. Fill in the parameters here ( # of instances, The nick you want to be tracked with at the archive team site, the spot price you are willing to pay, and optionally a keypair if you selected that file). 7. Check the box at the bottom acknowledging that the template will create IAM resources ( used by the host to bootstrap ) 8. Click Continue. 9. Tags if you want, or click continue. 10. Review. Click Continue. 11. Close. This will launch however many instances you told it to, as t1.micro’s, as the spot price you set it to. When you want to stop, you just go and delete the stack in this console and everything should go away. Running this right now in US-West-2, spread across all 3 AZ's there, about 90 instances total, and cranking through things.
rpicard 14y ago
I've got 50 spot requests running and another 50 pending evaluation. I'm really curious to see how much I can contribute with that in 24 hours.

74 points by duggan 14y ago

When your spot instance gets killed mid-download, does the Warrior system handle that and re-assign it to someone else immediately? Or was your spot instance assigned some URLs to download with the assumption that your Warrior would be reliable, and now they won't get reassigned until they check them all at the end, which may be too late?

Cameron_D 14y ago
They will be reassigned later on
deleted 14y ago
[deleted]

lignuist 14y ago

I always hate it, when companies remove user generated content from the internet. Why doesn't Yahoo just send some Dvds with the content to Archive.org?

knackers 14y ago
I was thinking the same thing whilst reading the article. There are surely people at Yahoo! who would gladly help if asked.
- Cameron_D 14y ago
  I'm sure there would be people in Yahoo would want to help, the problem is actually finding and getting in contact with someone who does care.
  - seanp2k2 14y ago
    ...then clearing it with legal and getting approval within the org to actually /do/ this. AFAIK, this would be unprecedented, but would probably win Yahoo! a lot of fans when they need it most. I have no real hope for them or anyone in a similar position to provide user data back to community custodians.
    - ersii 14y ago
      .. as long as you forget all the other user-generated-content sites Yahoo has closed over the years.

conroy 14y ago

If you have boto installed, this is all you'll need to do import boto.ec2 conn = boto.ec2.connect_to_region("us-east-1") conn.request_spot_instances('0.005', 'ami-2400984d', instance_type='t1.micro', user_data='USERNAME')

JustARandomGuy 14y ago

Is there a way we can donate to this? If you can supply a Paypal account, I'd gladly throw in a few bucks.

MichaelStubbs 14y ago
Their wiki answers your question: http://archiveteam.org/index.php?title=Posterous#Can_I_donate_some_cash_instead.3F http://archiveteam.org/index.php?title=Posterous#Can_I_donat...

duggan 14y ago

For those having trouble with the EC2 instructions, I thought I'd point out that I think that the Archive Warrior[0] (which is much easier to get up and running on your laptop, etc) running over my tethered cellphone is my most performant client. Yahoo don't appear to rate limit mobile devices / IP blocks as aggressively as everything else (probably because cellular providers tend to have many customers behind one IP). [0] http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri...

duggan 14y ago
I've also made the image available in all regions, for those who want to run in additional regions: N. Virginia: ami-2400984d Ireland: ami-d8d2d8ac Tokyo: ami-a361e1a2 Singapore: ami-6e703c3c Sydney: ami-4e0e9f74 Sao Paolo: ami-9d7aa180 N. California: ami-94f6dbd1 Oregon: ami-cf9206ff

seanp2k2 14y ago

Happy to help, but it looks like we're getting throttled hard by Yahoo. Anyone have a contact over there who they can ping about this?

Auguste 14y ago

We're currently at 16,000+ items and 61 GB uploaded. Nice work. Edit: 25 minutes later and we are at 18,700+ items and 67 GB uploaded. Distributed computing at its finest!

kogir 14y ago

Is this whole archival movement the Internet equivalent of hoarding? When is it ok to clean house?

cleverjake 14y ago
When bits are essentially free, there isn't much of a reason not to. Could you imagine how fascinating it would be to be able to dive into the everyday culture of 100, or even 1000 years ago? The anthropological impacts of archiving day to day life is huge. For once, history may be written by facts, rather than the victors. That may be a bit naïve, but who cares if they aren't hurting anyone.
sp332 14y ago
Several groups tried to salvage as much of Geocities as possible before Yahoo killed it. They got most of it (about 1 terabyte) and you can fix most geocities.com links by changing them to point at reocities.com instead. The main reason is that it's user data and deleting it is rude. Second is that you can do interesting analysis on a terabyte of user accounts during the boom of the internet. http://contemporary-home-computing.org/1tb/archives/3297 http://contemporary-home-computing.org/1tb/archives/3297 The third is that it's history! NSFW example from geocities http://contemporary-home-computing.org/1tb/archives/2736 http://contemporary-home-computing.org/1tb/archives/2736

pronoiac 14y ago

If you'd rather run a script than a vm, check out: https://github.com/ArchiveTeam/yahoomessages-grab https://github.com/ArchiveTeam/yahoomessages-grab

sp332 14y ago
That works for this project, but having the warrior makes it easy to join other ArchiveTeam projects now and in the future.

tgeek 14y ago

I built 2 CloudFormation templates to allow you to easily spin up a ton of these things across multiple availability zones. It uses the Amazon Linux AMI that exists in each region instead of the ones listed, and builds up the dependencies and the application on the fly. You can run either of these two templates in any region and it should just work. Download one of these files: With a keypair ( so you can login to the host) http://files.wordsaboutbytes.com/yahoo-messages-save.cf.txt http://files.wordsaboutbytes.com/yahoo-messages-save.cf.txt Without a keypair ( can’t log in locally, but it will run) http://files.wordsaboutbytes.com/yahoo-messages-save-nokeypair.cf.txt http://files.wordsaboutbytes.com/yahoo-messages-save-nokeypa... Then: 1. Open the console ( https://console.aws.amazon.com https://console.aws.amazon.com ) 2. Go to CloudFormation 3. Give your stack a name 4. Browse and select the file you downloaded from above 5. Click Next. 6. Fill in the parameters here ( # of instances, The nick you want to be tracked with at the archive team site, the spot price you are willing to pay, and optionally a keypair if you selected that file). 7. Check the box at the bottom acknowledging that the template will create IAM resources ( used by the host to bootstrap ) 8. Click Continue. 9. Tags if you want, or click continue. 10. Review. Click Continue. 11. Close. This will launch however many instances you told it to, as t1.micro’s, as the spot price you set it to. When you want to stop, you just go and delete the stack in this console and everything should go away. Running this right now in US-West-2, spread across all 3 AZ's there, about 90 instances total, and cranking through things.

rpicard 14y ago

I've got 50 spot requests running and another 50 pending evaluation. I'm really curious to see how much I can contribute with that in 24 hours.