43 ms·

I don't see why this particularly matters. You could likely store the entire twitter archive on a couple of memory sticks, so it doesn't seem to warrant some bi

by axod 17y ago

I don't see why this particularly matters. You could likely store the entire twitter archive on a couple of memory sticks, so it doesn't seem to warrant some big library to "house" it.

pchristensen 17y ago
This is a huge collection of primary sources from lots and lots of people. It's kind of like if you got a copy of every letter sent by people in Philadelphia in the 1770s - most of it would be crap but there would be some pretty neat stuff too.
- pyre 17y ago
  It's a little different since these are only public tweets. You won't get the same level of private information that you would get if you were grabbing private letters from the mail service.
tibbon 17y ago
No. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at least a few terabytes of data. I think this is big for a few reasons. One, if its in the LoC then it is (probably) available to the public. Previously it wasn't. That's pretty big. Also with the LoC having a public archive of it, we can keep better track of politicians who post things online- always a good thing since occasionally they shift their positions or outright lie. As a data analyst, I'm really excited.
- axod 17y ago
  Sure, maybe a few TB uncompressed. Compression would be pretty astoundingly good on that sort of data though - retweets etc.
- Gormo 17y ago
  55 million tweets/day * 140 characters/tweet ~= 14.5 GB/day, presuming a 16-bit character encoding. That's only about 20 TB for four years worth of data, or about $1500 worth of storage, before compression. That doesn't seem like it would be a significant challenge for the LOC. I'm more interested in understanding why they would want to use that amount of space to archive Twitter, as opposed to anything else.
  - FlemishBeeCycle 17y ago
    Combination of low-hanging fruit plus inflated idea of twitter's usefulness (much like other people largely unfamiliar with today's technology culture)?
  - tibbon 17y ago
    There's a good bit of metadata (about another 300 characters or so on average) attached to each tweet that makes it useful. Not a huge amount more, but considering the growth curve... yea 20TB is about right.

by axod 17y ago

I don't see why this particularly matters. You could likely store the entire twitter archive on a couple of memory sticks, so it doesn't seem to warrant some big library to "house" it.

pchristensen 17y ago

This is a huge collection of primary sources from lots and lots of people. It's kind of like if you got a copy of every letter sent by people in Philadelphia in the 1770s - most of it would be crap but there would be some pretty neat stuff too.

pyre 17y ago
It's a little different since these are only public tweets. You won't get the same level of private information that you would get if you were grabbing private letters from the mail service.

tibbon 17y ago

No. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at least a few terabytes of data. I think this is big for a few reasons. One, if its in the LoC then it is (probably) available to the public. Previously it wasn't. That's pretty big. Also with the LoC having a public archive of it, we can keep better track of politicians who post things online- always a good thing since occasionally they shift their positions or outright lie. As a data analyst, I'm really excited.

axod 17y ago
Sure, maybe a few TB uncompressed. Compression would be pretty astoundingly good on that sort of data though - retweets etc.
Gormo 17y ago
55 million tweets/day * 140 characters/tweet ~= 14.5 GB/day, presuming a 16-bit character encoding. That's only about 20 TB for four years worth of data, or about $1500 worth of storage, before compression. That doesn't seem like it would be a significant challenge for the LOC. I'm more interested in understanding why they would want to use that amount of space to archive Twitter, as opposed to anything else.
- FlemishBeeCycle 17y ago
  Combination of low-hanging fruit plus inflated idea of twitter's usefulness (much like other people largely unfamiliar with today's technology culture)?
- tibbon 17y ago
  There's a good bit of metadata (about another 300 characters or so on average) attached to each tweet that makes it useful. Not a huge amount more, but considering the growth curve... yea 20TB is about right.