4 ms·

Twitter also gives away their data to anyone that wants it for free. You can be streaming millions of tweets per day within minutes of creating an account.

xevb3k 8y ago
So what did they pay for? I’m confused, do Twitter rate limit free accounts?
- testplzignore 8y ago
  The free API is not a full firehose. It's been many years since the full firehose was publicly available.
  - dumbfounder 8y ago
    It is very easy to setup multiple streams to get a LOT of data. Not everything, but a metric shit ton. I have collected over 10 billion tweets for Twicsy over the years, and that is only because I was targeting pictures. If I had cast the net wide I could easily have gathered 10x that, and if I was super aggressive maybe 50x.
  - spullara 8y ago
    It is 1% of all tweets which is about 50 tweets per second. Not the full firehose but still very useful.
    - dumbfounder 8y ago
      It is up to 1% per stream. With 2 streams you can stream up to 2% (minus overlaps). This is theoretical, practice isn't quite that good, but not far off. In reality you can collect about 4-5 million tweets per day with a stream. If you look for common terms like http,and,for,the, etc you can collect 10's of millions per day easily with 20 or so streams. If you take the time to develop some tech and pull hundreds of streams designed to have minimal overlap you can pull 100+ million per day. And how do you pull more than 1 stream, you ask? The limit of streams isn't per app, it is per authorization. If you develop an app that has sign in with Twitter and you collect auth keys for 100 people you can have 100 streams because the auth limits are per user, not per app.
      - solarkraft 8y ago
        Interested dummy here: You can get a 1% stream per user authorisation (sounds weird to me)? How do you minimize overlap? I once had access to a 1% stream but thought it was a fire hose test version (and everyone would get the same 1% to avoid combining like you describe).
        dumbfounder 8y ago
        There is a sample stream which is supposed to be 1%, but then the keyword based filter streams are also theoretically up to 1% of tweets. You can setup streams for different words and broaden your coverage. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter https://developer.twitter.com/en/docs/tweets/filter-realtime...
        spullara 8y ago
        I could probably figure out the optimal set of keywords from all the twitter sample hose data I have...
- existencebox 8y ago
  Yes. (less one-wordily, 3500 posts or 30 days history per user, if I'm remembering right. (or something to that extent) There are a bunch of ratelimits. And if you really get into it, the "firehose", the full stream of all tweets, certainly isn't available to a normal user.
- dumbfounder 8y ago
  Yes, there are limits that are easily worked around to gather data en masse.
  - xevb3k 8y ago
    But I assume all these work arounds would be against the terms of service. So Cambridge Analytica were paying for legal access to this data. I think that makes it a bit more significant (though the situation is still very different than Facebook).
mindcrash 8y ago
That used to be the case, sure. But AFAIK these days you won't get access to the firehose without full authorization from Twitter (and there's a price tag involved).
- deleted 8y ago
  [deleted]
- rco8786 8y ago
  The firehouse has been under lock and key (except to those with enough $$) for a really long time now, like maybe more than a decade.
  - dumbfounder 8y ago
    You can get A LOT of data for free through their standard streaming API. If you are creative you can pull in about 100 million tweets per day.
chimeracoder 8y ago
> Twitter also gives away their data to anyone that wants it for free. You can be streaming millions of tweets per day within minutes of creating an account. That's news to me. I maintain the Twitter client library for Go[0], and I still haven't been able to get through the process for getting access to the new streaming API (for testing the library). It requires an application stating your intended purpose and usage. I applied explaining the situation and was rejected (with a request to provide more information - I still have to resubmit that, but it's time and I just haven't gotten around to that yet). It's possible to stream millions of tweets per day, but good luck trying to do it "within minutes of creating an account". And it's definitely not free. For full context: I'm already a verified user on Twitter (ie, "blue check mark") and was clear in my application that I was requesting access so I could develop and test the library that other users of the Twitter API develop with. I'm not exactly complaining that they didn't approve the request initially, but if that's not sufficient to get access, it's pretty clear that they're not "giving away their data to anyone that wants it for free". [0] https://github.com/ChimeraCoder/anaconda https://github.com/ChimeraCoder/anaconda
- dumbfounder 8y ago
  Maybe you are referring to their paid service, Gnip? Yes, you need to pay for that one, but they have a free version that you can indeed access within minutes. Try Twitter4J. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html https://developer.twitter.com/en/docs/tweets/filter-realtime...
  - chimeracoder 8y ago
    > Maybe you are referring to their paid service, Gnip? Yes, you need to pay for that one, but they have a free version that you can indeed access within minutes. The streaming API is heavily rate-limited, so you can't realistically stream "millions of tweets per day within minutes", in the context of the original article. The new Account Activity API, which replaces Site Streams, is what I was specifically referring to. https://twittercommunity.com/t/account-activity-api-beta-access/104722 https://twittercommunity.com/t/account-activity-api-beta-acc...
ryanmccullagh 8y ago
Firehouse data is not the complete stream. It states this in their docs. It's curated.

xevb3k 8y ago

So what did they pay for? I’m confused, do Twitter rate limit free accounts?

testplzignore 8y ago
The free API is not a full firehose. It's been many years since the full firehose was publicly available.
- dumbfounder 8y ago
  It is very easy to setup multiple streams to get a LOT of data. Not everything, but a metric shit ton. I have collected over 10 billion tweets for Twicsy over the years, and that is only because I was targeting pictures. If I had cast the net wide I could easily have gathered 10x that, and if I was super aggressive maybe 50x.
- spullara 8y ago
  It is 1% of all tweets which is about 50 tweets per second. Not the full firehose but still very useful.
  - dumbfounder 8y ago
    It is up to 1% per stream. With 2 streams you can stream up to 2% (minus overlaps). This is theoretical, practice isn't quite that good, but not far off. In reality you can collect about 4-5 million tweets per day with a stream. If you look for common terms like http,and,for,the, etc you can collect 10's of millions per day easily with 20 or so streams. If you take the time to develop some tech and pull hundreds of streams designed to have minimal overlap you can pull 100+ million per day. And how do you pull more than 1 stream, you ask? The limit of streams isn't per app, it is per authorization. If you develop an app that has sign in with Twitter and you collect auth keys for 100 people you can have 100 streams because the auth limits are per user, not per app.
    - solarkraft 8y ago
      Interested dummy here: You can get a 1% stream per user authorisation (sounds weird to me)? How do you minimize overlap? I once had access to a 1% stream but thought it was a fire hose test version (and everyone would get the same 1% to avoid combining like you describe).
      - dumbfounder 8y ago
        There is a sample stream which is supposed to be 1%, but then the keyword based filter streams are also theoretically up to 1% of tweets. You can setup streams for different words and broaden your coverage. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter https://developer.twitter.com/en/docs/tweets/filter-realtime...
        spullara 8y ago
        I could probably figure out the optimal set of keywords from all the twitter sample hose data I have...
existencebox 8y ago
Yes. (less one-wordily, 3500 posts or 30 days history per user, if I'm remembering right. (or something to that extent) There are a bunch of ratelimits. And if you really get into it, the "firehose", the full stream of all tweets, certainly isn't available to a normal user.
dumbfounder 8y ago
Yes, there are limits that are easily worked around to gather data en masse.
- xevb3k 8y ago
  But I assume all these work arounds would be against the terms of service. So Cambridge Analytica were paying for legal access to this data. I think that makes it a bit more significant (though the situation is still very different than Facebook).

mindcrash 8y ago

That used to be the case, sure. But AFAIK these days you won't get access to the firehose without full authorization from Twitter (and there's a price tag involved).

deleted 8y ago
[deleted]
rco8786 8y ago
The firehouse has been under lock and key (except to those with enough $$) for a really long time now, like maybe more than a decade.
- dumbfounder 8y ago
  You can get A LOT of data for free through their standard streaming API. If you are creative you can pull in about 100 million tweets per day.

chimeracoder 8y ago

> Twitter also gives away their data to anyone that wants it for free. You can be streaming millions of tweets per day within minutes of creating an account. That's news to me. I maintain the Twitter client library for Go[0], and I still haven't been able to get through the process for getting access to the new streaming API (for testing the library). It requires an application stating your intended purpose and usage. I applied explaining the situation and was rejected (with a request to provide more information - I still have to resubmit that, but it's time and I just haven't gotten around to that yet). It's possible to stream millions of tweets per day, but good luck trying to do it "within minutes of creating an account". And it's definitely not free. For full context: I'm already a verified user on Twitter (ie, "blue check mark") and was clear in my application that I was requesting access so I could develop and test the library that other users of the Twitter API develop with. I'm not exactly complaining that they didn't approve the request initially, but if that's not sufficient to get access, it's pretty clear that they're not "giving away their data to anyone that wants it for free". [0] https://github.com/ChimeraCoder/anaconda https://github.com/ChimeraCoder/anaconda

dumbfounder 8y ago
Maybe you are referring to their paid service, Gnip? Yes, you need to pay for that one, but they have a free version that you can indeed access within minutes. Try Twitter4J. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html https://developer.twitter.com/en/docs/tweets/filter-realtime...
- chimeracoder 8y ago
  > Maybe you are referring to their paid service, Gnip? Yes, you need to pay for that one, but they have a free version that you can indeed access within minutes. The streaming API is heavily rate-limited, so you can't realistically stream "millions of tweets per day within minutes", in the context of the original article. The new Account Activity API, which replaces Site Streams, is what I was specifically referring to. https://twittercommunity.com/t/account-activity-api-beta-access/104722 https://twittercommunity.com/t/account-activity-api-beta-acc...

ryanmccullagh 8y ago

Firehouse data is not the complete stream. It states this in their docs. It's curated.