6 ms·
The Library of Congress Acquires Entire Twitter Archive
- jrockway 17y agoOne more government database...
- tptacek 17y agoFirst they want to track all of our books... now this? Next it'll be the newspapers. Just you wait.
- jrockway 17y agoBooks were intended for wide distribution, but not all tweets are. I don't care that this was done, and it's nice that this snapshot of 2008-2010 will be available for future researchers. But I think this sort of dataset is more useful for nefarious purposes (government investigations) than a bunch of books or newspapers, and it would be too expensive to build for just one investigation. "Build it and they will come." It's built.
- pyre 17y agoNotice that they said public tweets. This is nothing that someone couldn't scrape off the site with enough time and energy spent.
- alanh 17y agoTechnically inaccurate, if you were to start today. Only the last 3,200 tweets of a particular user are discoverable without special access. And that assumes you have a list of all the users.
- pyre 17y agoAh. I stand corrected. They rest of my point still stands though. This was all information that was broadcast to the world, so there was hardly an expectation of privacy (or at least a reasonable one).
- axod 17y agoI don't see why this particularly matters. You could likely store the entire twitter archive on a couple of memory sticks, so it doesn't seem to warrant some big library to "house" it.
- pchristensen 17y agoThis is a huge collection of primary sources from lots and lots of people. It's kind of like if you got a copy of every letter sent by people in Philadelphia in the 1770s - most of it would be crap but there would be some pretty neat stuff too.
- pyre 17y agoIt's a little different since these are only public tweets. You won't get the same level of private information that you would get if you were grabbing private letters from the mail service.
- tibbon 17y agoNo. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at least a few terabytes of data. I think this is big for a few reasons. One, if its in the LoC then it is (probably) available to the public. Previously it wasn't. That's pretty big. Also with the LoC having a public archive of it, we can keep better track of politicians who post things online- always a good thing since occasionally they shift their positions or outright lie. As a data analyst, I'm really excited.
- axod 17y agoSure, maybe a few TB uncompressed. Compression would be pretty astoundingly good on that sort of data though - retweets etc.
- Gormo 17y ago55 million tweets/day * 140 characters/tweet ~= 14.5 GB/day, presuming a 16-bit character encoding. That's only about 20 TB for four years worth of data, or about $1500 worth of storage, before compression. That doesn't seem like it would be a significant challenge for the LOC. I'm more interested in understanding why they would want to use that amount of space to archive Twitter, as opposed to anything else.
- lecha 17y agoThis headline reads like one for an Onion news story
- chaosmachine 17y agoI thought for sure this was a late april fools post.
- xelipe 17y agoYeah, I thought the same thing... then the fear that my tweets to trending hashtags like #threewordsforyou or #worstlies are in the Library of Congress for perpetuity.
- pierrefar 17y agoIt's not helped that the announcement about acquiring the Twitter archive is done on facebook.com.
- astine 17y agoI work for the LoC. I think I'd of heard of this. Edit: It is my considered opinion that a lot of what the LoC does is excessively ambitious. The project I'm currently working in is called the Born Digital project, which is an attempt to allow major broadcasters like PBS and CNN to submit broadcast material to the Library in real-time. The tools to do this, do not currently exist.
- rufo 17y agoWhat's wrong with ambitious? Also - I'd thought real-time capture of video sources was something the DVR solved a long time ago... if you can talk about it, what's different about your project?
- deleted 17y ago[deleted]
- jonknee 17y ago> The tools to do this, do not currently exist. Considering that all the major networks beam their broadcasts all over the world in real time I don't see what the challenge is other than storage space. Startups like Justin.tv have figured out how to not only capture but stream thousands of simultaneous live streams, so the tools certainly exist.
- astine 17y agoThe primary difficulty is that the Library is attempting more than just live-capture of video streams. They are going for full archival quality. They're not simply saving streams to tape but but running extensive qc processes on everything and transcoding coding the material into multiple formats. Technically, the technology exists, but the Library does not have the hardware to run the software. (Unless someone knows of some efficient, off the shelf content-validation program that we don't know of...) Other problems include, yes, storage. The projected aggregation rate will outstrip, not only the Libraries current physical facilities, but all planned new facilities, by 2013 (at which point we will be looking at 250PB of storage.) The rate of growth will continue to increase from there. Unless tape technology improves at an exponential rate, this seem unavoidable.
- pyre 17y agoI wonder if the LoC keeps dumps from Wikipedia (in relation to the other frontpage story). I would think that the full 5.6TB Wikipedia dump would probably contain more useful information on average than 20TB of tweets. [ref: http://news.ycombinator.com/item?id=1265138 http://news.ycombinator.com/item?id=1265138]
- protomyth 17y agoHopefully they'll convert the short urls to long and make some arrangements to download pictures from twitphoto and their ilk.
- paulgb 17y agoIf not, this would be a cool project for a third-party distributed project like SETI@home or folding@home.
- tibbon 17y agoIt's a lot of data, but not THAT much data. Nothing that a small amazon cluster Hadooping it couldn't do in just an hour or so. If anything, the bottleneck would probably be the URL shorting services themselves.
- chronomex 17y agoYeah. I have a multithreaded perl script that can basically peg the database on the far end. This usually happens at about 20-30 threads. It sort of leaks memory, to the point that restarting it every day or so is advisable. http://github.com/chronomex/urlteam http://github.com/chronomex/urlteam if you're interested in my code.
- chronomex 17y agoI'm a member of ArchiveTeam (http://www.archiveteam.org/ http://www.archiveteam.org/). One of our current projects is to archive all the URL shorteners we can find. We're going to make the full dataset available for public download.
- joshwa 17y agoIs anyone concerned about copyright issues? I'd have to go back and consult the Twitter TOS... edit: "You agree that this license includes the right for Twitter to make such Content available to other companies, organizations or individuals who partner with Twitter for the syndication, broadcast, distribution or publication of such Content on other media and services, subject to our terms and conditions for such Content use. Such additional uses by Twitter, or other companies, organizations or individuals who partner with Twitter, may be made with no compensation paid to you with respect to the Content that you submit, post, transmit or otherwise make available through the Services. We may modify or adapt your Content in order to transmit, display or distribute it over computer networks and in various media and/or make changes to your Content as are necessary to conform and adapt that Content to any requirements or limitations of any networks, devices, services or media."
- whyenot 17y agoWhat copyright issues? The US Copyright Office is part of the Library of Congress. One of the LOC's responsibilities is to maintain a collection of all copyrighted works. The Library serves as a legal repository for copyright protection and copyright registration, and as the base for the United States Copyright Office. Regardless of whether they register their copyright, all publishers are required to submit two complete copies of their published works to the Library if requested—this requirement is known as mandatory deposit. http://en.wikipedia.org/wiki/Library_of_Congress http://en.wikipedia.org/wiki/Library_of_Congress
- jamesbritt 17y agoEven for non-USA citizens?
- deleted 17y ago[deleted]
- wallflower 17y agoIt will be interesting if they extend their CQL (Contextual Query Language) implementation to support complex searches (e.g. who tweeted first about X) http://www.loc.gov/standards/sru/index.html http://www.loc.gov/standards/sru/index.html
- ja27 17y agoHow interesting is it that they announced that on Facebook?
- cobralibre 17y agoI wonder how the content and quality of discourse on Twitter would have differed if its users had known all along that their posts would be archived for posterity by the LoC. This has been like an Ender's Game for hyperbanality!
- arthur_debert 17y agoIt's very likely that most people using twitter have no idea what the LoC is.
- michael_nielsen 17y agoThis has been confirmed on the Library of Congress blog: http://www.loc.gov/tweet/how-tweet-it-is.html http://www.loc.gov/tweet/how-tweet-it-is.html It is not a fake or a late April Fool joke, despite the fact that the original post is from facebook.
- iamdave 17y agoGreat, now future generations can see how obsessed we are with Justin Beiber.