9 ms·
Twitter also sold data access to Cambridge Analytica researcher
- dumbfounder 8y agoTwitter also gives away their data to anyone that wants it for free. You can be streaming millions of tweets per day within minutes of creating an account.
- xevb3k 8y agoSo what did they pay for? I’m confused, do Twitter rate limit free accounts?
- testplzignore 8y agoThe free API is not a full firehose. It's been many years since the full firehose was publicly available.
- dumbfounder 8y agoIt is very easy to setup multiple streams to get a LOT of data. Not everything, but a metric shit ton. I have collected over 10 billion tweets for Twicsy over the years, and that is only because I was targeting pictures. If I had cast the net wide I could easily have gathered 10x that, and if I was super aggressive maybe 50x.
- spullara 8y agoIt is 1% of all tweets which is about 50 tweets per second. Not the full firehose but still very useful.
- dumbfounder 8y agoIt is up to 1% per stream. With 2 streams you can stream up to 2% (minus overlaps). This is theoretical, practice isn't quite that good, but not far off. In reality you can collect about 4-5 million tweets per day with a stream. If you look for common terms like http,and,for,the, etc you can collect 10's of millions per day easily with 20 or so streams. If you take the time to develop some tech and pull hundreds of streams designed to have minimal overlap you can pull 100+ million per day. And how do you pull more than 1 stream, you ask? The limit of streams isn't per app, it is per authorization. If you develop an app that has sign in with Twitter and you collect auth keys for 100 people you can have 100 streams because the auth limits are per user, not per app.
- solarkraft 8y agoInterested dummy here: You can get a 1% stream per user authorisation (sounds weird to me)? How do you minimize overlap? I once had access to a 1% stream but thought it was a fire hose test version (and everyone would get the same 1% to avoid combining like you describe).
- dumbfounder 8y agoThere is a sample stream which is supposed to be 1%, but then the keyword based filter streams are also theoretically up to 1% of tweets. You can setup streams for different words and broaden your coverage. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter https://developer.twitter.com/en/docs/tweets/filter-realtime...
- spullara 8y agoI could probably figure out the optimal set of keywords from all the twitter sample hose data I have...
- existencebox 8y agoYes. (less one-wordily, 3500 posts or 30 days history per user, if I'm remembering right. (or something to that extent) There are a bunch of ratelimits. And if you really get into it, the "firehose", the full stream of all tweets, certainly isn't available to a normal user.
- dumbfounder 8y agoYes, there are limits that are easily worked around to gather data en masse.
- xevb3k 8y agoBut I assume all these work arounds would be against the terms of service. So Cambridge Analytica were paying for legal access to this data. I think that makes it a bit more significant (though the situation is still very different than Facebook).
- mindcrash 8y agoThat used to be the case, sure. But AFAIK these days you won't get access to the firehose without full authorization from Twitter (and there's a price tag involved).
- deleted 8y ago[deleted]
- rco8786 8y agoThe firehouse has been under lock and key (except to those with enough $$) for a really long time now, like maybe more than a decade.
- dumbfounder 8y agoYou can get A LOT of data for free through their standard streaming API. If you are creative you can pull in about 100 million tweets per day.
- chimeracoder 8y ago> Twitter also gives away their data to anyone that wants it for free. You can be streaming millions of tweets per day within minutes of creating an account. That's news to me. I maintain the Twitter client library for Go[0], and I still haven't been able to get through the process for getting access to the new streaming API (for testing the library). It requires an application stating your intended purpose and usage. I applied explaining the situation and was rejected (with a request to provide more information - I still have to resubmit that, but it's time and I just haven't gotten around to that yet). It's possible to stream millions of tweets per day, but good luck trying to do it "within minutes of creating an account". And it's definitely not free. For full context: I'm already a verified user on Twitter (ie, "blue check mark") and was clear in my application that I was requesting access so I could develop and test the library that other users of the Twitter API develop with. I'm not exactly complaining that they didn't approve the request initially, but if that's not sufficient to get access, it's pretty clear that they're not "giving away their data to anyone that wants it for free". [0] https://github.com/ChimeraCoder/anaconda https://github.com/ChimeraCoder/anaconda
- dumbfounder 8y agoMaybe you are referring to their paid service, Gnip? Yes, you need to pay for that one, but they have a free version that you can indeed access within minutes. Try Twitter4J. https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html https://developer.twitter.com/en/docs/tweets/filter-realtime...
- chimeracoder 8y ago> Maybe you are referring to their paid service, Gnip? Yes, you need to pay for that one, but they have a free version that you can indeed access within minutes. The streaming API is heavily rate-limited, so you can't realistically stream "millions of tweets per day within minutes", in the context of the original article. The new Account Activity API, which replaces Site Streams, is what I was specifically referring to. https://twittercommunity.com/t/account-activity-api-beta-access/104722 https://twittercommunity.com/t/account-activity-api-beta-acc...
- ryanmccullagh 8y agoFirehouse data is not the complete stream. It states this in their docs. It's curated.
- Talyen42 8y agoPublic tweets. You know, the ones everyone can see, all the time. Big difference.
- mickael-kerjean 8y agoFor how much time? When you consider youporn sends ping to twitter throught a weird domain (with a cookie on https://syndication.twitter.com/ https://syndication.twitter.com/), it makes me wondering how much time it will takes before the porn life of everyone on twitter will be reveal. I don't want to be the alarmist but it's not like twitter has a good track record when it comes to database leak and god knows how many influent politics are using it ...
- crunchlibrarian 8y agoSeems like every web page pings all the social networks nowadays. The app world is worse, much more difficult to see who is phoning whom. You need to install three extensions just to battle all this abuse. Getting a browser ready to surf the web in 2018 is like preparing to invade Normandy, and quite absurd.
- zamber 8y agoThat's why when I setup a browser for anyone that's not too technical I select pretty much all social blocking lists (among other). No one yet complained that they can't Tweet while watching porn.
- imglorp 8y agoMore than that. They're also selling engagement and audience on their API. This means they keep track of who you follow and what you click on.
- m1el 8y agoThe follow list is public as well.
- deleted 8y ago[deleted]
- testplzignore 8y agoIf anyone is interested in how to buy public Twitter data, here are the available APIs: http://support.gnip.com/apis/ http://support.gnip.com/apis/ https://developer.twitter.com/en/docs/tweets/batch-historical/overview https://developer.twitter.com/en/docs/tweets/batch-historica... (new doc for Gnip API) https://developer.twitter.com/en/docs/tweets/search/overview/enterprise https://developer.twitter.com/en/docs/tweets/search/overview... (new doc for Gnip API)
- stingraycharles 8y agoYes I’m surprised all this is shocking people. Guess people knew about this, but only now are suddenly realising the implications ?
- superkuh 8y agohttps://xkcd.com/743/ https://xkcd.com/743/
- TAForObvReasons 8y agoFacebook's public response destroyed every social media company's presumption of good faith. > “In 2015, GSR did have one-time API access to a random sample of public tweets from a five-month period from December 2014 to April 2015,” Twitter said in a statement to Bloomberg. “Based on the recent reports, we conducted our own internal review and did not find any access to private data about people who use Twitter.” Facebook kept changing their message regarding the affected accounts, from nothing to 87M and now possibly way more. There's a possibility that Twitter might have sold DM information as well. Under any other circumstance they would have some presumption of good faith, but given what happened with Facebook there's a good chance we will hear more.
- detaro 8y ago> There's a possibility that Twitter might have sold DM information as well. Do you have any specific reason to mention that or are we just listing all data they have?
- viyu 8y agoIt's looking a lot like a (crumbling) house of cards...
- viyu 8y agoSorry I don't understand the severe downvote? I was referring to all the "under the table" things that happened with Cambridge Analytica and similar stories, that will surface in the next few weeks/months. Hence the crumbing house of cards...
- JumpCrisscross 8y ago> I don't understand the severe downvote? The analogy adds nothing new nor interesting to the discussion. It's a content-free observation.
- viyu 8y agoI see. I'm new here and I thought others might relate to what I said. Thanks anyway.
- reitanqild 8y agoFurthermore complaining about being downvoted is against the guidelines. Not saying you complained but some people seems to use that as an excuse for downvoting every metaquestion that touches into these stupid markers of approval/appreciation or whatever it signals : )
- pmlnr 8y agoComplaining != asking for clarification. Sometimes the reason for downvotes is not obvious.
- viyu 8y agoAgreed. I'm at -7 and I didn't do anything to deserve this. It's quite rude and unfriendly. But it clearly makes some people feel good to have the power to do that, certainly because they can't do it IRL. Poor sapiens.
- listentojohan 8y agoI find that the whole premise of Twitter, is that the data is public.
- ComodoHacker 8y agoSo we may assume he found a way to correlate users between Twitter and Facebook.
- anoncoward778 8y agoOf course they did. Jack been a bad boy.
- blklivesmatter 8y agoMy private data sold again. Just like facebook and google. When people understand to avoid these liars ?? Please do not trust twitter facebook or google with your privste data. Really scary is how much these dna research companies are with google.
- kyleblarson 8y agoNewsflash: a for-profit company takes actions that maximize profits.
- stochastic_monk 8y agoCancer cells maximize growth. That doesn’t mean we should leave them unregulated and let them metastasize and kill the host.
- zamber 8y agoMakes you think someone should setup some rules for handling, gathering and processing user information. These companies are too big to fail. You can't fight with cancer by slapping it on the tendrils with fines. If you can't cut the cancer out then cut yourself from it. The effect is comparable (if we forget about or actively fight shadow profiles). FB, Twitter and other social/media companies are for-profit and your persona is the currency they're after. But hey, at least we'll get the perfect brand of socks/bed sheets/mattresses in the ads we're served.
- stochastic_monk 8y agoI’m with you; github is my social network. I love that their business model is about supporting open source products but providing infrastructure for pay for companies that need privacy. That way, I’m not a product, my work is shared with the world, and they get to make money.
- fusiongyro 8y agoAlmost more like "McDonalds sells hamburger to Cambridge Analytica researcher."
- downandout 8y agoThis title is misleading. By saying “Twitter also sold data access...” it implies that Facebook sold him data access. This of course is not the case. Facebook didn’t sell him anything, he produced an app and abused the platform.
- TheForumTroll 8y agoHe produced an app and used the platform.
- zaroth 8y agoIf they violated the ToS — which as I understand it they did — that would mean it’s more accurate to say “abused” here.
- dwild 8y agoDidn't they get more than what was expected from the API? The user accepted his profile information but they also got his friends information. Based on Wikipedia: > Abuse is the improper usage or treatment of an entity, often to unfairly or improperly gain benefit. That seems like they used the API improperly to access his friends information and got the friends information. For sure Facebook shouldn't have given that information, but it was also clear that information wasn't supposed to be there, that Cambridge Analytica were aware of it and abused it.
- downandout 8y agoThe user accepted his profile information but they also got his friends information. That was expected and agreed to behavior by all parties (users, developers, and Facebook) at the time. I know it’s out of fashion in 2018 to acknowledge that TOS exist and that users should have read them before agreeing to them (or at least before complaining after the fact), but this is the reality. The “abuse” part was that you were supposed to only access data that was necessary for the operation of your app. Accessing friends’ profile information to make an app that shows people who work at the same company as you, for example, was completely acceptable and a good use case. Collecting it for purposes unrelated to the stated purpose of the app was and is abuse. The Obama For America app also abused the platform in this way, but on a vastly larger scale - they pulled the entire US social graph (about 4x as many profiles as Kogan had) after less than 1 million people actually authorized the app and used it in all sorts of ways outside their app. This technique was widely celebrated in the press, right up until the “wrong” candidate won.
- AzzieElbab 8y agoWhat!? Did Cambridge Analytics buy that bridge in Brooklyn too?
- Yizahi 8y agoNow is the moment. The moment where you eat your tie/ hat/whatever. Yes, you, who blindly defended Twitter that "did no harm" and "Twitter is not like FB". Come on people, company that hosts insanely large free webservice 24/7 for years doesn't sell user data to the highest bidder? Do you even believe it?
- jimnotgym 8y agoA bot of a tangent, but a year ago I loved Twitter and found it engaging. I tweeted mildly interesting comments and sometimes mildly amusing anecdotes. Then the algorithms took hold. Now most of what I tweet goes totally unnoticed. Now perhaps I am more boring now (even though I have a lot more folowers), but it doesn't seem to be the case, I think I was just as inane before. On the other hand, if I get maybe four or five friends to retweet me, all of a sudden I have exposure and I get read, liked and retweeted a lot. This has just pushed Twitter more into the hands of bots. If I had a bot army of maybe 10 bots I would get a fair bit of exposure. There is also a momentum thing, if I take a couple of weeks off Twitter it doesn't love me any more when I come back. This has ruined Twitter for me.
- munchor 8y agoI am pretty sure you can disable the algorithmic feed in the settings. It's under "Show the best Tweets first" checkbox in the "Account" page of the Settings.
- jimnotgym 8y agoThat won't fix my experience, unless you can get everyone else to turn it off too
- rainbowmverse 8y agoThis is only available on older accounts and doesn't seem to work anyway.
- weka 8y agoYou know what ruined Twitter for me? Having them clog up my twitter timeline of people I don't _even_ follow with their tweets or the LIKES of people I do. They distorted what a timeline was and thus the follow button essentially became ruined.
- arca_vorago 8y agoWhat really pisses me off is that now I get message notifications for "you might have missed what X said"... Twitter, thats not a fucking message!
- dymk 8y agoInaccurate headline. There's no "also"; Facebook didn't sell that data to CA.
- fiatjaf 8y agoTwitter is supposed to be public.
- greggarious 8y agoAt least I thought carefully about what I tweeted since I knew it would be public. I'm much more upset that private Facebook data, never intended to be distributed to the public, was taken from me.
- textmode 8y ago"The ethics panel member added: "Facebook is rather deceptive on this and creates the appearance of a cosy and confidential peer group environment, as a means of gulling users into disclosing private information that they then sell to advertisers, but this doesn't make it right to an ethical researcher to follow their lead."" https://www.theguardian.com/technology/2018/apr/24/cambridge-university-rejected-facebook-study-over-deceptive-privacy-standards https://www.theguardian.com/technology/2018/apr/24/cambridge... The article discusses the Cambridge University ethics committee rejection of proposal for use of Facebook app to gather data. Anyone have a non-scribd link to the full comments from the ethics committee members? When the Facebook CTO recently testified before the UK Parliament, at one point they laughed at him for being so blatently insincere. Facebook is now copying Reddit/HN up/downvotes, trial in NZ/AUS.