6 ms·
I won't lie... I came here hoping for a link to the torrent of the entire collection...
by mryingster 13y ago
I won't lie... I came here hoping for a link to the torrent of the entire collection...
- toomuchtodo 13y agoAlready working on it (putting together a package for the Internet Archive; might as well put a torrent up as well). EDIT: Scraper running. Should be done shortly (collecting small version, original version, XML data).
- toomuchtodo 13y agoUpdate: Still scraping, lots of sleep X to be gentle on Getty.
- dawie 13y agoPlease post the torrent link once it's done.
- toomuchtodo 13y agoI will.
- pandesmos 13y agoI'm not going to give up hope. I still believe.
- toomuchtodo 13y agoWas at Burning Man, this totally fell off my plate. I'm looking for someplace to put the 100GB zip file up right now.
- toomuchtodo 13y agoPutting the torrent up now. It's ~100GB of content, so give it a bit.
- ericcoleman 13y agoYay! Link yet?
- karasinski 13y agoDoesn't look like it! Will be excited to see this though.
- iooi 13y agoDo you mind sharing what tool you use to scrape?
- toomuchtodo 13y agoI was rushing, so it was just curl, lynx, sed, awk, all wrapped with bash. Don't judge; I'd use python and a scraping library if I wasn't rushing.
- fluxon 13y agoThank you for taking the initiative.
- deleted 13y ago[deleted]
- rogerhoward 13y agoThey had the S3 bucket originally set to allow listing - I grabbed it all before I reported it to an engineer there. Not going to post a torrent though - mostly cuz you all can figure it out anyhow. The images are all of the form: NNNNNN01.JPG on that S3 bucket. NNNNNN is the object ID (the internal record ID of the work of art) zero-padded to 6 digits. The object ID is the same value that appears, for instance, in the URL on their online collection: URL for Irises: http://www.getty.edu/art/gettyguide/artObjectDetails?artobj=947 http://www.getty.edu/art/gettyguide/artObjectDetails?artobj=... URL for the full-rez image on S3: http://gettylargeimages.s3.amazonaws.com/00094701.jpg http://gettylargeimages.s3.amazonaws.com/00094701.jpg You figure it out :)
- uniclaude 13y agoNice. That would make a convenient fizzbuzz style interview question.
- hobs 13y agoI dont understand... wouldnt offering a torrent be more friendly than us all downloading it again?
- 6cxs2hd6 13y agoS3 buckets can act as torrents. Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.htm...
- toomuchtodo 13y agoThis is exactly how I plan to serve up the compiled archive.
- hobs 13y agoThats pretty sweet actually, definitely +1. I said my original comment because the idea was to have use learn the power of wget by redownloading all files, which seemed pointless to me.
- 13y ago