5 ms·

How are they getting the data? Twitter's APIs seem so limited these days.

by tibbon 14y ago

FranckErnewein 14y ago
Data come from the Twitter Streaming API https://dev.twitter.com/docs/streaming-apis https://dev.twitter.com/docs/streaming-apis
- jevinskie 14y ago
  Is the site getting the firehose stream though? How do you get that level of access?
  - huskyr 14y ago
    They're probably using the 'sample' firehose, which is not protected and still serves a few tweets per second.
    - tariqr 14y ago
      that looks like a lot more data than a few tweets per second though.
    - lucaspiller 14y ago
      The free version gives access to 1% [1] of all tweets, which seems about right for this. [1] https://dev.twitter.com/discussions/2752 https://dev.twitter.com/discussions/2752
mmahemoff 14y ago
It struck me seeing this how rare it is now to see any kind of Twitter experiment. Complete 180 from a few years back.
- mochizuki 14y ago
  There's so much potential information there, and some very neat things you can do with it. But for me at least, their new API policies completely turned me off of making cool things with it. Not that the API isn't still powerful.
- tibbon 14y ago
  Yep. I was part of the Web Ecology Project in 2009-2010 and we were able to get pretty awesome access to Twitter at the time. I had a good dialog on a weekly basis it seemed with their API team and they were super responsive and supportive of us. Although, whenever I asked what we needed to pay for full Firehose (which some companies were getting) I couldn't get an answer from them, and I think they only wanted to talk with huge companies. Then that all stopped. No more whitelisted IPs for search, no more new whitelisted accounts for API calls, and then they stripped all my whitelisted accounts of their 20k/hr API calls and put them all back at the standard 150/hr or whatever. And what's worse is the network is bigger than ever. Instead of scaling API access up with network size, they scaled it down. This is really a shame as I think Twitter is terribly interesting to with it from an academic standpoint, but unless you have huge money to shell out for API calls or direct access you're screwed.

How are they getting the data? Twitter's APIs seem so limited these days.

by tibbon 14y ago

How are they getting the data? Twitter's APIs seem so limited these days.

Data come from the Twitter Streaming API https://dev.twitter.com/docs/streaming-apis https://dev.twitter.com/docs/streaming-apis

jevinskie 14y ago
Is the site getting the firehose stream though? How do you get that level of access?
- huskyr 14y ago
  They're probably using the 'sample' firehose, which is not protected and still serves a few tweets per second.
  - tariqr 14y ago
    that looks like a lot more data than a few tweets per second though.
  - lucaspiller 14y ago
    The free version gives access to 1% [1] of all tweets, which seems about right for this. [1] https://dev.twitter.com/discussions/2752 https://dev.twitter.com/discussions/2752

mmahemoff 14y ago

It struck me seeing this how rare it is now to see any kind of Twitter experiment. Complete 180 from a few years back.

mochizuki 14y ago
There's so much potential information there, and some very neat things you can do with it. But for me at least, their new API policies completely turned me off of making cool things with it. Not that the API isn't still powerful.
tibbon 14y ago
Yep. I was part of the Web Ecology Project in 2009-2010 and we were able to get pretty awesome access to Twitter at the time. I had a good dialog on a weekly basis it seemed with their API team and they were super responsive and supportive of us. Although, whenever I asked what we needed to pay for full Firehose (which some companies were getting) I couldn't get an answer from them, and I think they only wanted to talk with huge companies. Then that all stopped. No more whitelisted IPs for search, no more new whitelisted accounts for API calls, and then they stripped all my whitelisted accounts of their 20k/hr API calls and put them all back at the standard 150/hr or whatever. And what's worse is the network is bigger than ever. Instead of scaling API access up with network size, they scaled it down. This is really a shame as I think Twitter is terribly interesting to with it from an academic standpoint, but unless you have huge money to shell out for API calls or direct access you're screwed.