5 ms·

So what did they pay for? I’m confused, do Twitter rate limit free accounts?

by xevb3k 8y ago

testplzignore 8y ago
The free API is not a full firehose. It's been many years since the full firehose was publicly available.
- dumbfounder 8y ago
  It is very easy to setup multiple streams to get a LOT of data. Not everything, but a metric shit ton. I have collected over 10 billion tweets for Twicsy over the years, and that is only because I was targeting pictures. If I had cast the net wide I could easily have gathered 10x that, and if I was super aggressive maybe 50x.
- spullara 8y ago
  It is 1% of all tweets which is about 50 tweets per second. Not the full firehose but still very useful.
  - dumbfounder 8y ago
    It is up to 1% per stream. With 2 streams you can stream up to 2% (minus overlaps). This is theoretical, practice isn't quite that good, but not far off. In reality you can collect about 4-5 million tweets per day with a stream. If you look for common terms like http,and,for,the, etc you can collect 10's of millions per day easily with 20 or so streams. If you take the time to develop some tech and pull hundreds of streams designed to have minimal overlap you can pull 100+ million per day. And how do you pull more than 1 stream, you ask? The limit of streams isn't per app, it is per authorization. If you develop an app that has sign in with Twitter and you collect auth keys for 100 people you can have 100 streams because the auth limits are per user, not per app.
    - solarkraft 8y ago
      Interested dummy here: You can get a 1% stream per user authorisation (sounds weird to me)? How do you minimize overlap? I once had access to a 1% stream but thought it was a fire hose test version (and everyone would get the same 1% to avoid combining like you describe).
      - dumbfounder 8y ago
        There is a sample stream which is supposed to be 1%, but then the keyword based filter streams are also theoretically up to 1% of tweets. You can setup streams for different words and broaden your coverage. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter https://developer.twitter.com/en/docs/tweets/filter-realtime...
        spullara 8y ago
        I could probably figure out the optimal set of keywords from all the twitter sample hose data I have...
existencebox 8y ago
Yes. (less one-wordily, 3500 posts or 30 days history per user, if I'm remembering right. (or something to that extent) There are a bunch of ratelimits. And if you really get into it, the "firehose", the full stream of all tweets, certainly isn't available to a normal user.
dumbfounder 8y ago
Yes, there are limits that are easily worked around to gather data en masse.
- xevb3k 8y ago
  But I assume all these work arounds would be against the terms of service. So Cambridge Analytica were paying for legal access to this data. I think that makes it a bit more significant (though the situation is still very different than Facebook).

So what did they pay for? I’m confused, do Twitter rate limit free accounts?

by xevb3k 8y ago

So what did they pay for? I’m confused, do Twitter rate limit free accounts?

testplzignore 8y ago

The free API is not a full firehose. It's been many years since the full firehose was publicly available.

dumbfounder 8y ago
It is very easy to setup multiple streams to get a LOT of data. Not everything, but a metric shit ton. I have collected over 10 billion tweets for Twicsy over the years, and that is only because I was targeting pictures. If I had cast the net wide I could easily have gathered 10x that, and if I was super aggressive maybe 50x.
spullara 8y ago
It is 1% of all tweets which is about 50 tweets per second. Not the full firehose but still very useful.
- dumbfounder 8y ago
  It is up to 1% per stream. With 2 streams you can stream up to 2% (minus overlaps). This is theoretical, practice isn't quite that good, but not far off. In reality you can collect about 4-5 million tweets per day with a stream. If you look for common terms like http,and,for,the, etc you can collect 10's of millions per day easily with 20 or so streams. If you take the time to develop some tech and pull hundreds of streams designed to have minimal overlap you can pull 100+ million per day. And how do you pull more than 1 stream, you ask? The limit of streams isn't per app, it is per authorization. If you develop an app that has sign in with Twitter and you collect auth keys for 100 people you can have 100 streams because the auth limits are per user, not per app.
  - solarkraft 8y ago
    Interested dummy here: You can get a 1% stream per user authorisation (sounds weird to me)? How do you minimize overlap? I once had access to a 1% stream but thought it was a fire hose test version (and everyone would get the same 1% to avoid combining like you describe).
    - dumbfounder 8y ago
      There is a sample stream which is supposed to be 1%, but then the keyword based filter streams are also theoretically up to 1% of tweets. You can setup streams for different words and broaden your coverage. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter https://developer.twitter.com/en/docs/tweets/filter-realtime...
      - spullara 8y ago
        I could probably figure out the optimal set of keywords from all the twitter sample hose data I have...

existencebox 8y ago

Yes. (less one-wordily, 3500 posts or 30 days history per user, if I'm remembering right. (or something to that extent) There are a bunch of ratelimits. And if you really get into it, the "firehose", the full stream of all tweets, certainly isn't available to a normal user.

dumbfounder 8y ago

Yes, there are limits that are easily worked around to gather data en masse.

xevb3k 8y ago
But I assume all these work arounds would be against the terms of service. So Cambridge Analytica were paying for legal access to this data. I think that makes it a bit more significant (though the situation is still very different than Facebook).