4 ms·

Given that tweets are only 280 characters and there are perhaps a couple hundred accounts that the LoC archives, I fail to understand why the premier archiving

Given that tweets are only 280 characters and there are perhaps a couple hundred accounts that the LoC archives, I fail to understand why the premier archiving project in the country can't keep a slowly growing fistful of gigabytes on a hard drive somewhere.

cevn 9y ago
Agreed. I have to wonder if they are doing it manually.
falcolas 9y ago
From someone who worked with the Twitter firehose briefly: A handful of gigs per day. More like hundreds of terabytes of information total, headed into petabyte territory. First, there's a lot of tweets. Also, the message itself is not all the data available with a tweet. Toss in any semantic processing or tagging and each message grows quickly. The company I worked with spent a non-trivial amount of money storing historical tweets. I'd even go so far to say that was the majority of the IAAS costs - even more than the compute required to process them in real time.
- Uehreka 9y ago
  OK fair enough, I didn't account for metadata. I'm gonna try some back of the envelope math, tell me if I go off the rails anywhere: First, how many accounts are we talking about here? Between POTUS, VP, their spokespeople, cabinet members and official agency accounts, I'm going to assume the executive branch has about 50 accounts that ought to be archived. For congress, I think it's reasonable to archive each member's account and their spokesperson: 200 accounts for senate, 870 accounts for congress. In the interest of Fermi estimation (and because I'm not sure if every one of these people has an account) let's call it 1000 accounts. I'm going to go with a mean of 10 tweets per day (again in the name of Fermi estimation). With 280 characters + metadata, I'm going to round up to 1KB per tweet. 1000 accounts * 10 tweets/day * 1KB/tweet = ~10MB/day = ~3.65GB/year = ~40GB for the current lifetime of Twitter If you're drinking from the firehose to archive tweets for a huge userbase (and feed them into models or perform semantic analysis) I could see this getting expensive and costly. If you're just try to archive tweets from a certain group of users and keep them on a disk (with a tarball or zip file released quarterly) it feels a bit more doable. That said, this little thought exercise has gotten me thinking a lot more about what I expect of the LoC and the National Archives. I'd be happy with a "cold storage" record of the tweets being preserved for posterity, but they may see their mission as making the tweets into a tagged searchable collection. I also think there are arguments to be had about how many people's account really need to be archived (perhaps states could handle archiving their own reps). In the end I guess I'm cool with the LoC scaling back collection as long as they're transparent about how they're doing it.
  - rajivm 9y ago
    They were doing all accounts previously, not just government officials. Their new selection is likely to still include government officials.
  - ghostbrainalpha 9y ago
    It's not just metadata. You can tweet photos, videos and GIFS. My average gif is 500kb. If these were excluded I suspect it would make the archive much easier to manage.
cooper12 9y ago
Any archival project worth its salt would never store just the tweet text, but also all associated metadata. The best way I've seen to do this is to just save the JSON returned by an API as it's easily parseable. You also have to consider that the utility of Twitter isn't in the single tweets, but rather the whole comment chains. Lastly, Twitter has images as well.
advisedwang 9y ago
> there are perhaps a couple hundred accounts that the LoC archives The article states that it was archiving _all_ of public twitter. It is moving to the selectivity that you have assumed was status quo.
hanbura 9y ago
All of twitter, simplz gz compressing the firehose stream, is at least 4tb/month. Likely more

cevn 9y ago

Agreed. I have to wonder if they are doing it manually.

falcolas 9y ago

From someone who worked with the Twitter firehose briefly: A handful of gigs per day. More like hundreds of terabytes of information total, headed into petabyte territory. First, there's a lot of tweets. Also, the message itself is not all the data available with a tweet. Toss in any semantic processing or tagging and each message grows quickly. The company I worked with spent a non-trivial amount of money storing historical tweets. I'd even go so far to say that was the majority of the IAAS costs - even more than the compute required to process them in real time.

Uehreka 9y ago
OK fair enough, I didn't account for metadata. I'm gonna try some back of the envelope math, tell me if I go off the rails anywhere: First, how many accounts are we talking about here? Between POTUS, VP, their spokespeople, cabinet members and official agency accounts, I'm going to assume the executive branch has about 50 accounts that ought to be archived. For congress, I think it's reasonable to archive each member's account and their spokesperson: 200 accounts for senate, 870 accounts for congress. In the interest of Fermi estimation (and because I'm not sure if every one of these people has an account) let's call it 1000 accounts. I'm going to go with a mean of 10 tweets per day (again in the name of Fermi estimation). With 280 characters + metadata, I'm going to round up to 1KB per tweet. 1000 accounts * 10 tweets/day * 1KB/tweet = ~10MB/day = ~3.65GB/year = ~40GB for the current lifetime of Twitter If you're drinking from the firehose to archive tweets for a huge userbase (and feed them into models or perform semantic analysis) I could see this getting expensive and costly. If you're just try to archive tweets from a certain group of users and keep them on a disk (with a tarball or zip file released quarterly) it feels a bit more doable. That said, this little thought exercise has gotten me thinking a lot more about what I expect of the LoC and the National Archives. I'd be happy with a "cold storage" record of the tweets being preserved for posterity, but they may see their mission as making the tweets into a tagged searchable collection. I also think there are arguments to be had about how many people's account really need to be archived (perhaps states could handle archiving their own reps). In the end I guess I'm cool with the LoC scaling back collection as long as they're transparent about how they're doing it.
- rajivm 9y ago
  They were doing all accounts previously, not just government officials. Their new selection is likely to still include government officials.
- ghostbrainalpha 9y ago
  It's not just metadata. You can tweet photos, videos and GIFS. My average gif is 500kb. If these were excluded I suspect it would make the archive much easier to manage.

cooper12 9y ago

Any archival project worth its salt would never store just the tweet text, but also all associated metadata. The best way I've seen to do this is to just save the JSON returned by an API as it's easily parseable. You also have to consider that the utility of Twitter isn't in the single tweets, but rather the whole comment chains. Lastly, Twitter has images as well.

advisedwang 9y ago

> there are perhaps a couple hundred accounts that the LoC archives The article states that it was archiving _all_ of public twitter. It is moving to the selectivity that you have assumed was status quo.

hanbura 9y ago

All of twitter, simplz gz compressing the firehose stream, is at least 4tb/month. Likely more