3 ms·
A little anecdote that might be interesting here. I worked at The Archive for a little while and one of the projects I worked on was to unpack about 300 TB of
by noufalibrahim 10y ago
A little anecdote that might be interesting here.
I worked at The Archive for a little while and one of the projects I worked on was to unpack about 300 TB of crawl data from the defunct search company cuil.com. It was mostly decent quality data in a standard format and after some grinding, the whole thing was converted into warc files which the wayback machine could use to show the URLs. The end result was that about 60 billion URLs came "back onto the web".
During the work, I was looking at the stones rather than cathedral but after I left and thought about it in detail, it was very satisfying. I was reading the book "A Canticle For Leibowitz" at the time and the general theme of cycles of history was in my head. That dovetailed very well with the work I had done.
If you're interested, you can download the dumps over here https://archive.org/details/cuilcrawl https://archive.org/details/cuilcrawl
- toomuchtodo 10y agoUpvote for "A Canticle For Leibowitz" alone. Thanks for your service at the Internet Archive!
- noufalibrahim 10y agoIt was a privilege to work there and definitely the best period of my career. So many wonderful people and experiences in such a short time. I personally consider Brewster Kahle a severely underrated hero of our age. The Archive's work is extremely valuable but I think the value will be appreciated only by a future generation. As for the book, I think if The Archive had novel for a totem, it'd be "A Canticle For Leibowitz". Very much affected my world view when I read it and this thread has just kindled my interest again. :)
- toomuchtodo 10y agoIf you don't have a paperback copy of A Canticle, I'd be happy to send you mine!
- noufalibrahim 10y agoThat's very generous of you but I do have a treasured copy myself. Thanks! :)