8 ms·
Twitter to Release All Tweets to Scientists
- apetresc 12y agoThis is exciting to me; does anyone know how Twitter will go about this? Will there be a public dataset available for download? A research contract through the recently-acquired GNIP? Or just firehose access for future streams?
- umanwizard 12y agoDoesn't Twitter make a fair bit of money from selling access to various slices of their data? I'd be surprised if they released it all to the general public. I imagine scientists would have to be under some sort of NDA.
- JamesMcMinn 12y agoThe Twitter terms of service prohibit sharing the data in the Tweets. Researchers are allowed to share tweet IDs and User IDs which can be used to identify a Tweet. Currently, Twitter collections are shared using this method -- I recently released a dataset of 120 million Tweet IDs which cover a sample of a months worth of data, and numerous other researchers have used these IDs to crawl Twitter and obtain the same dataset as I used in my experiments.
- beejiu 12y agoConsidering there's at least 400GB of data generated per day, I don't think it'll be readily available for the public as a download.
- StavrosK 12y agoJeez, 400 GB of text per day? How the hell? EDIT: That's 11.5k tweets/sec. How do you get eleven thousand people to tweet every second?
- vschiavoni 12y agoHave you ever heard about the 'Human Brain Project' ? Well, they produce 100 GB of data _per second_ Welcome to the real big data world.
- vijayr 12y agoWoah, 100 GB per second???!! What exactly do they do? Edit: Never mind, this is what they are doing "We would like to develop some kind of ‘google' brain where we can zoom in and out, see it from different perspectives and understand how brain structure and function is related."
- Houshalter 12y agoMaybe it includes images.
- cmyr 12y agoit does not include images. A tweet object does include a bunch of metadata though: https://dev.twitter.com/docs/platform-objects/tweets https://dev.twitter.com/docs/platform-objects/tweets
- maxerickson 12y agoPick a topic you are familiar with. Open up a twitter search for it. Wait a while. See the inevitable storm of tweet-spam that sort of looks like social sharing. Thousands of fake accounts are tweeting out nonsense all the time. Another example, there are multiple accounts that tweet items from HN (and presumably lots of other rss feeds).
- Wilya 12y agoDepends how much metadata there is with every tweet. I took a random tweet on my timeline, and the JSON representation, with added metadata, weighs 8740 bytes. With this figure, you would "only" need 500 tweets/sec to get 400GB a day. There's a lot of people on Twitter, and a lot of bots, so that doesn't seem unreasonable to me.
- 12y ago
- shahocean 12y agohell lot of data! First they have to find some alternative to zip these much of data!
- JoshTriplett 12y agoThis would likely make a great natural language data set for compression algorithms.
- pipeep 12y agoTrue, but doesn't Twitter already provide an API for access to a fraction of the firehose? Surely that would be enough data. If Twitter doesn't have a good API, Reddit allows full access to all comments through their API (although Reddit has orders of magnitude less data).
- minimaxir 12y agoTwitter's API is too limited for historical data. (you'll hit the rate limits quickly for any meaningful volume). Reddit's rate limits, however, let you process a million comments every day.
- hyperbovine 12y ago@JoshTriplett tweets r alrdy #compressed. hth
- loceng 12y agoUpvoted for much needed humour on HN.
- stokedmartin 12y agoTwitter had initiated granting of datasets some time back (now closed)[0] on the merits of a short proposal. The number of groups who eventually got access to the data were very few[1]. I hope in the future they increase the number of grants. [0] https://blog.twitter.com/2014/introducing-twitter-data-grants https://blog.twitter.com/2014/introducing-twitter-data-grant... [1] https://blog.twitter.com/2014/twitter-datagrants-selections https://blog.twitter.com/2014/twitter-datagrants-selections
- alexleavitt 12y agoYes, I am pretty sure this article is just rehashing the Twitter grants (I believe there were only 6 to 8 rewards), rather than announcing full open data to any researchers (thereby making the title misleading).
- chbrown 12y agoI've heard that before. * Library of Congress: http://blogs.loc.gov/loc/2013/01/update-on-the-twitter-archive-at-the-library-of-congress/ http://blogs.loc.gov/loc/2013/01/update-on-the-twitter-archi... * Twitter Data grants: https://blog.twitter.com/2014/introducing-twitter-data-grants https://blog.twitter.com/2014/introducing-twitter-data-grant... I'll admit, I haven't applied for access through either one, but neither have I seen any papers cite access through those venues—and I read quite a few NLP + Twitter papers.
- etiam 12y agofrom http://www.loc.gov/today/pr/2013/files/twitter_report_2013jan.pdf http://www.loc.gov/today/pr/2013/files/twitter_report_2013ja... "Transfer of Data to the Library In December, 2010, Twitter named a Colorado-based company, Gnip, as the delivery agent for moving data to the Library. Shortly thereafter, the Library and Gnip began to agree on specifications and processes for the transfer of files - "current" tweets - on an ongoing basis. In February 2011, transfer of "current" tweets was initiated and began with tweets from December 2010. On February 28, 2012, the Library received the 2006-2010 archive through Gnip in three compressed files totaling 2.3 terabytes. When uncompressed the files total 20 terabytes. The files contained approximately 21 billion tweets, each with more than 50 accompanying metadata fields, such as place and description. As of December 1, 2012,the Library has received more than 150 billion additional tweets and corresponding metadata, for a total including the 2006-2010 archive of approximately 170 billion tweets totaling 133.2 terabytes for two compressed copies." I find the quantities hilarious. But since they haven't been able to cope with providing access yet I get pessimistic about their prospects of doing so at all any time soon. Can we do something to help them? I've been thinking maybe GPU-accelerated databases like MapD, could mitigate the cost issue for them, but I'm pretty sure that doesn't go all the way to solving the problem...
- denzil_correa 12y agoThis article is just talking about Twitter Data Grants for which 6 universities were decided as winners [0]. You won't see papers through these grants as yet because well, the winners were announced about 40 days back! [0] https://blog.twitter.com/2014/twitter-datagrants-selections https://blog.twitter.com/2014/twitter-datagrants-selections
- theg2 12y agoA release for journalists would be nice too...
- deleted 12y ago[deleted]
- of 12y agoWho cares? Isn't it already available?
- callesgg 12y agoYes it is, however not in excel. (written so a non tech person could understand)
- NamTaf 12y agoI can't wait to see someone legitimately design a better sewerage system by using twitter's geolocation.
- nevinera 12y agoTheir geo-data is utter crap. The vast majority of it is based on 'profile location', which means that there are almost a million people tweeting from the exact center of Atlanta. It's a crowded spot, must be a Starbucks there or something.
- Nicholas_C 12y agoJust find those mass locations and remove them from the data set.
- nevinera 12y agoYou end up with about 0.01% of tweets having locations after that. It's basically just iphone users.
- izzydata 12y agoIt is all available to the public to begin with anyway. I don't see the dilemma here.
- namenotrequired 12y agoNot all tweets are public.
- jonknee 12y agoSurely they aren't coughing up private tweets?
- namenotrequired 12y agoThat was my interpretation of "all", but reading back it seems there's another difference: "Although a majority of tweets are public, if scientists want to freely search the lot, they do it through Twitter's application programming interface, which currently scours only 1 percent of the archive. But that is about to change: in February the company announced that it will make all its tweets, dating back to 2006, freely available to researchers."
- marincounty 12y agoI think all the data should be available.
- flycaliguy 12y agoThere is no magic discovery about the nature of man hidden away in that data. Nothing you're average stand up comedian hasn't already written a bit about.
- coherentpony 12y agoThat's a little cynical. I like to think we can predict big historical social events (regime changes, major protests, climate change?) based on Twitter's data. You never know unless you try.
- TallGuyShort 12y agoAre you familiar with Atrocity Watch? If not, you should look them up - that's exactly what they do.
- coherentpony 12y agoI am not. Checking them out now; looks pretty cool.
- dirtyaura 12y agoThat was a good one. On a more serious tone, there is one area of research that Twitter data is very valuable for: how information and disinformation is created and spread during major news events: wars, catastrophes, uprisings, school shootings etc. My gut feeling is that today a lot of quality journalism happens outside of the traditional journalistic organisations. The downside is that also a lot of wild speculation and rumours are spread, but it would be valuable to see how good this modern "crowd" journalism is. A skilled research group can use Twitter data and Internet Archive to track down the original sources of information pretty well.
- marincounty 12y agoAnd momentum stock trades. I've always felt a adept programmer who knew how to data mine and worked for Twitter could make a fortune in the stock market?
- extesy 12y agoArticle date is Jun 1, 2014. Is the author from the future?
- _RPM 12y agoSo much for the "protected" tweet illusion.
- unclesaamm 12y agoCan anyone find a primary source?
- mike415 12y agoThere is a link to apply for a "Data Grant" here: https://engineering.twitter.com/research https://engineering.twitter.com/research. Unfortunately, it looks like submissions are closed.
- scalene 12y agoNot to be skeptical, but I'm pretty sure one of these scientists may happen to work for the NSA.
- Smulv 12y agoIt appears as if the data is only available to those scientists who apply for the data grant and win it. Furthermore, applications for the grant have been closed since midway through March. Yea, I'm not surprised Twitter isn't making its historical data public. That would literally end Gnip, which is a revenue source for Twitter not based on advertising to users.
- uptown 12y agoOne thing I've wondered. Is it possible to follow "everyone" on Twitter? If-not, what type of cap does Twitter enforce on the number of accounts you're allowed to follow? I realize it'd be difficult to know which new accounts to add as people join, but how far could you push a roll-your-own stream of the Twitter firehose?
- jonknee 12y agohttp://lmgtfy.com/?q=twitter+maximum+follow+limit&l=1 http://lmgtfy.com/?q=twitter+maximum+follow+limit&l=1
- freehunter 12y agoHypothetically: I'm sure you could do it algorithmically; if your program sees a retweet from someone who is not on your following list, you then follow them. You might miss a few, but you would get most everyone.
- jebus989 12y agoHow about just loosening the API rate limits, or making a better token request process with resource allocation e.g. I'd like 1500 requests per 15 min window (as opposed to 15, for some things) for 72 hours. I guess this could be limited to those with a academic email address if they insist.