2 ms·

This is exciting to me; does anyone know how Twitter will go about this? Will there be a public dataset available for download? A research contract through the

This is exciting to me; does anyone know how Twitter will go about this? Will there be a public dataset available for download? A research contract through the recently-acquired GNIP? Or just firehose access for future streams?

umanwizard 12y ago
Doesn't Twitter make a fair bit of money from selling access to various slices of their data? I'd be surprised if they released it all to the general public. I imagine scientists would have to be under some sort of NDA.
- JamesMcMinn 12y ago
  The Twitter terms of service prohibit sharing the data in the Tweets. Researchers are allowed to share tweet IDs and User IDs which can be used to identify a Tweet. Currently, Twitter collections are shared using this method -- I recently released a dataset of 120 million Tweet IDs which cover a sample of a months worth of data, and numerous other researchers have used these IDs to crawl Twitter and obtain the same dataset as I used in my experiments.
beejiu 12y ago
Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.
- StavrosK 12y ago
  Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?
  - vschiavoni 12y ago
    Have you ever heard about the 'Human Brain Project' ? Well, they produce 100 GB of data _per second_ Welcome to the real big data world.
    - vijayr 12y ago
      Woah, 100 GB per second???!! What exactly do they do? Edit: Never mind, this is what they are doing "We would like to develop some kind of ‘google' brain where we can zoom in and out, see it from different perspectives and understand how brain structure and function is related."
  - Houshalter 12y ago
    Maybe it includes images.
    - cmyr 12y ago
      it does not include images. A tweet object does include a bunch of metadata though: https://dev.twitter.com/docs/platform-objects/tweets https://dev.twitter.com/docs/platform-objects/tweets
  - maxerickson 12y ago
    Pick a topic you are familiar with. Open up a twitter search for it. Wait a while. See the inevitable storm of tweet-spam that sort of looks like social sharing. Thousands of fake accounts are tweeting out nonsense all the time. Another example, there are multiple accounts that tweet items from HN (and presumably lots of other rss feeds).
  - Wilya 12y ago
    Depends how much metadata there is with every tweet. I took a random tweet on my timeline, and the JSON representation, with added metadata, weighs 8740 bytes. With this figure, you would "only" need 500 tweets/sec to get 400GB a day. There's a lot of people on Twitter, and a lot of bots, so that doesn't seem unreasonable to me.
  - nevinera 12y ago
    Most of that data is not the content of the tweet itself, but the metadata associated with it. When I last checked, we were storing about a kilobyte of data for every tweet. Also, tweets are limited to 140 characters, not bytes - chinese tweets typically take about 200-250 bytes, for example.
    - StavrosK 12y ago
      Yeah, I figured 300 bytes per Tweet, to be generous, but didn't realize it would take 1 kb of metadata. Thanks for that detail.
  - JamesMcMinn 12y ago
    I have access to the Twitter gardenhose (which is equal to slightly less than 10% of the full volume). These are the RX and TX statistics from the machine that I've been using to gather data for a few months now: RX bytes:18053724505080 (16.4 TiB) TX bytes:2686623557042 (2.4 TiB) It works out at around 70GB/day, so I'd actually think that the full firehose would use considerably more data than 400GB/day (likely closer to 800GB).
- shahocean 12y ago
  hell lot of data! First they have to find some alternative to zip these much of data!

umanwizard 12y ago

Doesn't Twitter make a fair bit of money from selling access to various slices of their data? I'd be surprised if they released it all to the general public. I imagine scientists would have to be under some sort of NDA.

JamesMcMinn 12y ago
The Twitter terms of service prohibit sharing the data in the Tweets. Researchers are allowed to share tweet IDs and User IDs which can be used to identify a Tweet. Currently, Twitter collections are shared using this method -- I recently released a dataset of 120 million Tweet IDs which cover a sample of a months worth of data, and numerous other researchers have used these IDs to crawl Twitter and obtain the same dataset as I used in my experiments.

beejiu 12y ago

Considering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.

StavrosK 12y ago
Jeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?
- vschiavoni 12y ago
  Have you ever heard about the 'Human Brain Project' ? Well, they produce 100 GB of data _per second_ Welcome to the real big data world.
  - vijayr 12y ago
    Woah, 100 GB per second???!! What exactly do they do? Edit: Never mind, this is what they are doing "We would like to develop some kind of ‘google' brain where we can zoom in and out, see it from different perspectives and understand how brain structure and function is related."
- Houshalter 12y ago
  Maybe it includes images.
  - cmyr 12y ago
    it does not include images. A tweet object does include a bunch of metadata though: https://dev.twitter.com/docs/platform-objects/tweets https://dev.twitter.com/docs/platform-objects/tweets
- maxerickson 12y ago
  Pick a topic you are familiar with. Open up a twitter search for it. Wait a while. See the inevitable storm of tweet-spam that sort of looks like social sharing. Thousands of fake accounts are tweeting out nonsense all the time. Another example, there are multiple accounts that tweet items from HN (and presumably lots of other rss feeds).
- Wilya 12y ago
  Depends how much metadata there is with every tweet. I took a random tweet on my timeline, and the JSON representation, with added metadata, weighs 8740 bytes. With this figure, you would "only" need 500 tweets/sec to get 400GB a day. There's a lot of people on Twitter, and a lot of bots, so that doesn't seem unreasonable to me.
- nevinera 12y ago
  Most of that data is not the content of the tweet itself, but the metadata associated with it. When I last checked, we were storing about a kilobyte of data for every tweet. Also, tweets are limited to 140 characters, not bytes - chinese tweets typically take about 200-250 bytes, for example.
  - StavrosK 12y ago
    Yeah, I figured 300 bytes per Tweet, to be generous, but didn't realize it would take 1 kb of metadata. Thanks for that detail.
- JamesMcMinn 12y ago
  I have access to the Twitter gardenhose (which is equal to slightly less than 10% of the full volume). These are the RX and TX statistics from the machine that I've been using to gather data for a few months now: RX bytes:18053724505080 (16.4 TiB) TX bytes:2686623557042 (2.4 TiB) It works out at around 70GB/day, so I'd actually think that the full firehose would use considerably more data than 400GB/day (likely closer to 800GB).
shahocean 12y ago
hell lot of data! First they have to find some alternative to zip these much of data!