5 ms·
I added the raw .gz files to IPFS when Library of Congress announced this last week: https://github.com/ipfs/archives/issues/152 https://github.com/ipfs/archive
by lgierth 9y ago
I added the raw .gz files to IPFS when Library of Congress announced this last week: https://github.com/ipfs/archives/issues/152 https://github.com/ipfs/archives/issues/152
It's slightly more than 100GB and here it is: https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9/ https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtH...
(Note that the filesizes in the directory listing are all wrong -- that's the original index.html from loc.gov/cds/downloads/MDSConnect/)
This makes it a lot easier to use this dataset at e.g. hackathons, where a lot of people would simultaneously pester that LoC server, which already seemed pretty bandwidth-limited on its own when I downloaded the files.
- lgierth 9y agoYou can pin it: `ipfs pin add QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9` Or list it: `ipfs ls QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9` Or copy it into your local filesystem `ipfs get QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`
- toomuchtodo 9y agoDid you push them into the Internet Archive yet? If not, going to grab a beer and start iterating through your IPFS objects.
- lgierth 9y agoGo for it! :) I've never pushed anything to IA so far to be honest.
- toomuchtodo 9y agoFor future reference then! https://github.com/jjjake/internetarchive https://github.com/jjjake/internetarchive
- voltagex_ 9y agoIf someone wants to pay for / donate 100GB worth of bandwidth on a VPS somewhere, I'll do it.
- tty7 9y agoscaleway: €2.99/m 200Mbit/s Unmetered bandwidth
- voltagex_ 9y agoOkay, so who's going to spot me 3 EUR? ;)
- toomuchtodo 9y agoCould TransferWise you the $3.35USD. That work?
- voltagex_ 9y agoHaving never done this before I've got to have a look at the internetarchive tool first, but yes, that would work (I'd hate to take your money then not be able to deliver).
- voltagex_ 9y agoThis thread is getting too long; how do I contact you?
- tombrossman 9y agoThis amount of usage falls well within most VPS companies' free trial / promo codes offerings and should cost you nothing. Use a throwaway email account and drop it after a month. AWS will give you a whole year if you haven't tried them yet and the other popular VPS companies (DO, Linode, etc.) all will give you at least $10 startup credit. This is probably simpler and faster than figuring out how to receive <$4 from some random internet commenter.
- lgierth 9y agoOops, it turns out I somehow ended up with decompressed files, that's why the filesizes are so far off. I'm creating a new fixed dump.
- lgierth 9y agoThe correct dataset (with gzip compression) is here: https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1JAA7HX5 https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1...