17 ms·

Twitter shut down Emojitracker's access

293 points by exolymph 11y ago

dajohnson89 11y ago
Aw that's too bad, I just discovered Emojitracker yesterday and thought it was really cool. Also: is the service being removed colloquially known as the "Twitter Firehose", or something else? I recently implemented a feature that uses the standard rate-limited API. I went through a lot of pain to get the most out of the rate limits. And people would always ask me "why don't you just get all the tweets through the Streaming API?". I wonder if I had done that, how screwed I would be now.
- exolymph 11y ago
  It's hard to sort out the colloquial terms for Twitter features from what the company calls them =/
  - mattzito 11y ago
    Okay, here we go: - Twitter Firehose - full stream of all Tweets, commercial - Twitter Decahose - 10% sample of Tweets for analysis, commercial - Twitter Powertrack - real-time streaming of posts based on query (country, language, keyword, etc.), commercial - Twitter Full-archive API - search interface for entire corpus of Tweets, commercial - Twitter Streaming API - a mini-powertrack of posts based on basic keyword searches (up to 400 keywords), free - Twitter Streaming Sample - a small sampling of Tweets for statistical analysis, free There's a couple more esoteric product offerings from a data perspective, but these are the relevant ones. I think it's a pretty broad breadth of different options, with different functionality at the free and commercial options.
- xutopia 11y ago
  What did emojitracker do?
  - wpietri 11y ago
    http://www.emojitracker.com/ http://www.emojitracker.com/
  - spiznnx 11y ago
    It still does, for a few more weeks. http://emojitracker.com/ http://emojitracker.com/ Basically live tracks tweets that use emoji, so emoji are ranked by frequency.
  - NSAID 11y ago
    Emojitracker gives live, realtime usage numbers for all emoji used on twitter. http://emojitracker.com/ http://emojitracker.com/
- mattzito 11y ago
  So, Twitter has long had two different classes of access: - "Twitter Developer" access - which is what you find here: https://dev.twitter.com/ https://dev.twitter.com/ - that includes a streaming API that is relatively limited in terms of volume of data and operators that are allowed, but is free and open - Gnip Twitter access - this is commercial access that would include the Firehose option (though that's not very common). But basically this is a paid offering that offers much more functionality than the developer platform. In this specific case, the author is just losing elevated access to the streaming API: https://dev.twitter.com/streaming/reference/post/statuses/filter https://dev.twitter.com/streaming/reference/post/statuses/fi... I'm guessing they had extra keywords or multiple connections available so they could track those keywords. Nothing to do with the firehose or anything else. Just a user who had an exception to the official policy that's being taken away.
  - cmyr 11y ago
    For a long time twitter has offered specific developers who were doing projects they thought were interesting the ability to consumer a higher volume of traffic using the standard streaming endpoints.
aioprisan 11y ago
Another example of Twitter taking steps to kill their platform dreams and push developers away. So much for Jack's renewed push for improved developer relations.
- yeukhon 11y ago
  Well this is on the front-page now. maybe he will scream at the team and say "don't shut it down."
  - pc86 11y ago
    I don't think Jack gets his news from HN.
    - yeukhon 11y ago
      He may or may not, but I wouldn't doubt their PR team already pinned HN as a source to consume. Plus there is https://twitter.com/newsycombinator https://twitter.com/newsycombinator. #twitter, definitely.
  - aioprisan 11y ago
    You mean like they shut down the Twitter share counter data right after Jack had a big conference once he came back as CEO promising better developer relations? So, I doubt they care about the state of developer relations, as a platform.
- mattzito 11y ago
  This is a situation where one developer had a special exception carved out for them for a non-commercial project that everyone thought was cool. That access persisted for three years until someone made a decision that giving free access to something that normally they charge quite a lot of money for is not worth it. Twitter offers several free endpoints for developers, including search, streaming, a raw sample of tweets, and the like. They all have rate limits and the like attached, but they're real, free, access to their data. For people who want bigger, broader, deeper access, they sell access to that data. That's been the case for the last few years. In this case, Emojitracker unfortunately falls into the cracks in between, where they need deeper access to do what they want, but it's not a commercial enterprise. I agree, it would be nice of Twitter to continue giving Emojitracker free access, but are they obligated to do that? They are a business, and providing access to millions of Tweets for free seems like an unfair obligation to ask them to continue indefinitely.
joeblau 11y ago
Based on http://www.emojitracker.com http://www.emojitracker.com, there is a lot of Joy, and Love on Twitter.
TaylorGood 11y ago
With @jack having tweeted about this just months ago, talk about an internal disconnect..
duaneb 11y ago
> So even if Twitter were to give me free access to the Gnip APIs, others then wouldn’t be able to take my code and build upon it I mean, isn't this the case anyway? Few people have "elevated access to the Streaming API that Emojitracker depends on in order to operate at its high volume"
grandalf 11y ago
I've often wondered why Twitter seems to do this again and again in spite of what appears to be common sense and good stewardship of its platform and ecosystem. I think it's because Twitter is actually a successful, fairly open ecosystem that attracts all kinds of use. What seems like ham-handed or draconian decision making is just Twitter misunderstanding its role as infrastructure provider. What we all implicitly want from Twitter is for it to embody the values of net neutrality. If someone can build something cool on top of Twitter, why not let it happen? Twitter's investors have pressured the company to monetize, and most of the bad platform decisions seem to be tied to those efforts. Twitter's folly is assuming that usage patterns are "bad". It should view all usage in a content-neutral manner and provide and support APIs to let the ecosystem grow. It is the hubris of caring too much about content and content engagement metrics that is driving this folly. Twitter is a very simple broadcast graph. This is incredibly powerful, and while it's not quite as lucrative to own as Facebook (with users habituated to seeing the same boring stuff from the same boring people every day and clicking on sponsored content), it's far more powerful and is likely to change society more than other social media platforms. All this is why Twitter should pivot and become a decentralized infrastructure... of course this will never happen. I'm predicting that Twitter will die a slow death as it tries to copy Facebook's revenue generation approach. It is also copying Facebook's real name initiative by caring too much about content attribution and verified accounts (which users don't care about). Eventually we may see a decentralized broadcast graph, but until then it makes sense to expect Twitter to behave like a cognitively impaired dictator with respect to its ecosystem. I think most of this is driven by intense cost-cutting measures and intensely misguided focus on revenue metrics in terms of content and engagement. Twitter is not a product, it's an infrastructure.
- mschuster91 11y ago
  > It is also copying Facebook's real name initiative by caring too much about content attribution and verified accounts (which users don't care about). I fully agree with you, except on this point. With all the shitstorming, the mis-attribution of quotes, massive and massive amounts of fake accounts etc. going on, the Verified by Twitter program is a very good thing especially as it does not force you to publically post your real name. However, it could be expanded for greater reach.
roosterjm2k2 11y ago
This is a very entitled piece... and a very entitled thought process... When you are using someone else's platform, they still make the rules... this line in particular: "it wouldn’t help other developers who I believe deserve to get the same level of opportunity as me." ... nobody "deserves" access to any API... period. I don't understand the entitlement in so many tech projects, who get upset when a 3rd party they are freely using changes their service... you were never entitled to it in the first place, you merely benefited from its existence and should be appreciative of what you were allowed to do instead of feel entitled to it.
- nonanako 11y ago
  I think it comes from many companies branding themselves as altruistic and developer-friendly. I don't think people would be surprised at this move had Twitter said: "Using our platform? We own you. We will pursue all potential profit deriving from our platform to the fullest extent of the law."
- mroth 11y ago
  Author here. To be clear, I don't believe I deserve free or special access to anything. The point I was trying to make was actually more in the opposite direction: I wouldn't want to receive free/special access as an exception now simply because the project has become "famous", if the same opportunity doesn't exist for other independent developers when their projects are starting out -- I just want to encourage a level playing field. (When Emojitracker originally received elevated access 3 years ago, the project was not famous or well known at all, it was just a slightly more formal version of Me saying: "hey check this out, I think this might be a cool hack but I need access to 900 keywords instead of 400, that ok?" and Twitter: "yeah cool, no problem dude.")
  - Bedon292 11y ago
    So you are still accessing the streaming endpoint, but you have more keywords than they normally allow? So you are not accessing the entire Firehose and parsing from there, right? Do you regularly update it to include new emoji when they come out? I am thinking about ways to do it on the free stream still, just trying to see where you are already at.
drinchev 11y ago
Yeah. I was recently building a simple bot for Twitter ( a favor for a friend of mine ). Things worked out, but everywhere I go I read "API Rate limited to ... per ..." I know ... It's tremendous amount of data going on there, so I just don't want to imagine how hard it is to scale it and offer that to everyone. Nevertheless it's so sad that they break promises. Twitter nowadays is a bot driven, marketing platform ( with some sort of journalism touch ). I still remember the days that I was excited when I got a new follower. Now I think it's just about "hashtags".
willu 11y ago
I have no love for Twitter, Gnip, or the way they've treated API devs over the years but this particular use case (ranking emojis by frequency of use) seems like it could be just as well served by the free sampled stream. I can't imagine anyone needing the exact usage counts vs. the relative popularity. Also, the author's stated goal of making it open and immediately usable by other developers would be better served by using the access level that is freely available to everyone.
- TheHappyRock 11y ago
  Agreed, I don't think the full stream is needed either. It think being built of the free stream will serve the needs of making it open and still provide interesting data. Relative(sampling) usage graphs even seem more digestable and interesting to me.
kmfrk 11y ago
I'm surprised there are still developers left for Twitter to piss off.
bobbyadamson 11y ago
I don't understand why people continue to build things for services that have a track record of shutting things down.
stevebmark 11y ago
This makes sense to me. Granting free firehose access to a small, fun project like this seems like unnecessary overhead for Twitter. I think their mistake was granting it in the first place.
- Razengan 11y ago
  Exactly. Wasn't HN almost unanimously in support of GitHub throttling CocoaPods barely a week ago? How is this any different? Honest question.
  - spriggan3 11y ago
    CocoaPods was throttled not shut down. Twitter has an history of shutting down access to its API without notice or warning when a project becomes too popular.
    - Razengan 11y ago
      What baffles me is that CocoaPods was actually providing something of value and their usage fit in with the purpose of GitHub (serving code.) EmojiTracker appears to be essentially frivolous and puts stress on Twitter's platform for no good reason. GitHub and Twitter both provided a free service and then modified it for a specific user without warning. I would say throttling CocoaPod affected more "legit" users than "shutting down" EmojiTracker did. (I put shutting down in quotes because it seems EmojiTracker still has limited access to their API?) How come Twitter gets called out as unconditionally Evil, yet GitHub was just looking after itself?
- jacquesm 11y ago
  But then once you do grant it you should only revoke it for a really good reason (such as abuse or some other violation of the terms of service). Otherwise it should simply continue.
JamesMcMinn 11y ago
Everyone with elevated access is losing it in June as far as I'm aware - at least that's the case for academic uses. There will be academic pricing for Gnip, although we don't know how much it'll cost, I'm doubtful it'll be low enough for most academics to afford. They've been telling us that they planned to shut down free elevated access down for well over a year now, so this doesn't come as a huge surprise. Twitter seems to do regular audits like this, and at least for our academic work, we've had to tell them what we're working on etc. every 6 months or so to keep elevated access. Having said that, looking at the volume of data that Emojitracker is pulling down, it doesn't seem like it would be too difficult to spread the tracking over 2 or 3 separate connections. The UserStream API, whilst not designed for server-to-server connections, would probably allow Emojitracker to spread the ~900 emoji they track over 3 streaming connections using only a single account.
CoolAssPuppy 11y ago
Hi, I'm Prashant. I run Developer Relations at Twitter. I want to clarify a few things here. In the past, Twitter had little formality associated with granting elevated access. It was very much, "Hey, I want to do this cool thing!" "Okay, sure, here you go." As Twitter's platform business has matured to include many businesses building billions of dollars worth of social media monitoring and other tools, we've formalized the process to becoming a Twitter Official Partner (partners.twitter.com). Over the last six months, we've started contacting all of those API key holders with elevated access and asked them to clarify what they're doing. In some cases, we know the business and business owner and reach out personally. In other cases, like this one, the business is listed in our systems as "N/A", so we send the template mail. In the email, we encourage people who believe their app is within the bounds of acceptable use cases on Twitter to contact us directly, and provide a link to do so. The owner of this app elected to blog publicly about the situation before contacting us, which is unfortunate. We have contacted the owner of this app and hope to resolve this situation, as we do with hundreds of other developers on our platform. We do occasionally provide exceptions for apps that are non-commercial (not-for-profit, no ads, etc.) Note also that in this instance, the notice is NOT about shutting down this app. It merely inquires about why the developer needs elevated access, something that is typically reserved for our business partners. None of this is related to continuing to use the Twitter API or our commitment to enabling developers to build on our platform (Fabric, Gnip, our Ads platform, and the Twitter API).
- abritinthebay 11y ago
  Simply put - your "template mail" is terrible. If what you just said is what it the email meant to communicate then it fails at every level. Please don't try to shift the blame for Twitters poor communication to the recipient because they "elected to blog publicly about the situation". The "situation" is entirely created by Twitter. Twitter has a (recent) history of terrible communication with developers and it would not surprise anyone if what seemed to be the case WAS the case. If it is NOT the case, as you say, then you really need to improve your company communication.
- bby 11y ago
  prashant this won't save you.
subdane 11y ago
I'm "confused," what in the world are we supposed to make of this tweet https://twitter.com/jack/status/693296738145140736 https://twitter.com/jack/status/693296738145140736 and this statement? http://recode.net/2015/10/21/twitter-ceo-jack-dorsey-apologized-to-developers-for-past-confusion/ http://recode.net/2015/10/21/twitter-ceo-jack-dorsey-apologi...
- aioprisan 11y ago
  It means Twitter as a platform company doesn't care about keeping successful applications on their platform. Words (Tweets?) are cheap, actions speak louder. In this case, it's a resounding and massive door closing on an application that Jack said he loves and probably thought is/was good use of the platform.
- developer2 11y ago
  The telling quote from that article is this: > “Developers took to our service from day one and [gave] Twitter a much more global reach." Twitter would not be where Twitter is today without outside developers building their company for them, through the creation of tools that users actually want to use to interface with Twitter. Twitter as a standalone product does not have a future. Every change they make that turns more developers away from the platform continues to reduce the number of users globally who are using Twitter in some fashion. It's a downward spiral. Twitter's demise is inevitable if they continue down this path. Developers should assist them by abandoning the platform in droves. There won't be much of a platform left to work with anyway, so why bother? Let Twitter be the next Yahoo.
williamcotton 11y ago
Is any of this to be expected? Twitter is a private company and they have to create artificial scarcity to sell access to their platform. We still live in a market economy, no matter how much everyone on Twitter would like to think otherwise. They're just trying to pay the bills. Treating Twitter like a public platform is a disservice both to actual public infrastructure as well as the roll of private enterprise.
- halestock 11y ago
  Twitter is a public company.
  - williamcotton 11y ago
    What you mean to say is Twitter is a publicly traded company owned and operated by private individuals. Those private individuals have rights and protections granted by the government jurisdictions in which Twitter operates.
Lazare 11y ago
> In the email, we encourage people who believe their app is within the bounds of acceptable use cases on Twitter to contact us directly [...] the notice is NOT about shutting down this app. It merely inquires about why the developer needs elevated access, something that is typically reserved for our business partners. What? This is absurdly, obviously, completely wrong. Did you read the email you guys sent? Either it's forged, or you're talking out of your ass. Your response is great, reasonable, and, as a developer, would improve my opinion of Twitter if it had the slightest link to reality. > The owner of this app elected to blog publicly about the situation before contacting us, which is unfortunate. It's unfortunate they took your very clear email at face value and reacted in the way it was clearly intended to invoke? "Hey, when we told him to fuck off and die, we meant 'why don't you drop by the offices and have some cake!', why are you guys so rude?"
- dang 11y ago
  You can't comment like this here, regardless of how good a point you may have. If you can't be civil, please abstain from posting until you can. We detached this comment from https://news.ycombinator.com/item?id=11300071 https://news.ycombinator.com/item?id=11300071 and marked it off-topic.
tangled_zans 11y ago
This is just... wow. The guy spends three years developing a product on Twitter's platform and they just kill it off with one e-mail? It's fair enough if they need to change their API policy, but at the very least they could have acknowledged all the work that he's done and found a way to help him and the other developers in the same position to make the transition as painlessly as possible. Not... whatever corporate BS this was. Given how much Twitter is struggling already, is this really the time to be alienating more of its community? Personally, I've had a really nifty side project that I've been working on since last year that I wanted to launch on Twitter's platform, but as of a month ago I've all but given up on actively developing it. All the people that I know on Twitter with whom I wanted to share it have either left already, or are only sticking around because as-of-yet there's nowhere else to go that does what Twitter does. So I'm curious to hear some perspectives on this: clearly the direction that Twitter has taken is making a lot of devs unhappy, but what would be the right thing to do? What sort of API access would people ideally want to see it provide? What would be the ideal best-case scenario necessary for the devs to want to start coming back and putting their time into building the ecosystem?
wslh 11y ago
Three hours later and have you realized that CoolAssPuppy (Twitter itself) hasn't answered any specific question? If you write in HN you expect a conversation happening.
- JoblessWonder 11y ago
  Probably because Twitter's PR people probably reached out to him and told him he wasn't helping anything and needed to stop posting. I wouldn't be surprised if they walked to his office and unplugged his keyboard. It was bad. He posted something that made him look like an idiot or a liar. I'm not saying he is either one of those, but the post he wrote makes almost no sense when compared to the e-mail presented in the post. It is like he didn't even read it. If he was trying to say that they never sent that e-mail, then that point wasn't clear. I'm sure he had the best of intentions but it really blew up in his face.
- skeletonjelly 11y ago
  I used to subscribe to a service that emailed me when I got a response on HN but that stopped for some reason. HN Notify I think it was called. What do people use apart from refreshing their own comment pages?
  - kbenson 11y ago
    Dan Grossman, who's active in this very story, set up a replacement[1] a while back that I've been using and it seems to work well. 1: http://hnreplies.com/ http://hnreplies.com/
    - skeletonjelly 11y ago
      Legend thank you!
rdl 11y ago
Twitter fucked over every developer who trusted them a few years ago, and is now trying to recast their business as developer tools and services (since the core Twitter business probably is not a $20b business...). One of their biggest roadblocks is getting developers to trust them again. And then they do this?
- x1024 11y ago
  The only way for them to get developers on their platform is to wait for the next generation of developers. Literally nobody will base anything on their stuff now, and this has been the case for a good few years.
grizzles 11y ago
Why do people keep building stuff on Twitter? All the app developers that have been fenced off should create a simple protocol to make Twitter irrelevent. Brent Simmons suggested a great way do it awhile back: http://inessential.com/2012/06/29/matthew_on_twitter_restrictions http://inessential.com/2012/06/29/matthew_on_twitter_restric...
- pmlnr 11y ago
  Why do people keep building stuff on any silo? Take a look at Known (https://withknown.com/ https://withknown.com/) and Woodwind (https://woodwind.xyz/ https://woodwind.xyz/): one to have an actual website, the second to have a stream of content you want. Done. Forget silos.
PostThisTooFast 11y ago
This is why you don't build your enterprise on that of a third party who can dick you over and waste years of effort in an instant.
spriggan3 11y ago
At that point anybody whose business relies on Twitter "free API" deserves to be in that situation. It's a well proven fact that Twitter will shut down every 3rd party client or app based on their API which is a bit too popular. Never build a business on top of a "free" API, especially Twitter, because there is 100% chances Twitter will shut down you access.
jonesb6 11y ago
Has anyone here built a product that relies on Twitter's API for business critical features? How would you describe your risk tolerance, how long have you been in business, and do you have a personal or professional relationship with Twitter or Twitter's employees?
siegecraft 11y ago
This just the extract value before exit phase, right?
egfx 11y ago
I developed a product that connects Twitter and Facebook. I always thought that Twitter would be the enabler because of it's open street like atmosphere and Facebook would be difficult to work with because it was a walled garden. The exact opposite was true. Over time, Twitter would get more restrictive and difficult to work with while Facebook enhanced the look and feel of the stream and how the tweets appeared, emphasizing and highlighting them. In essence I think the product spoke to the nature of the whole issue. The tweets left Twitter to seed Facebook with good interesting, connected data. Twitter would know little about the context while Facebook would get enriched context from the tweets. Twitter doesn't want to be a dumb pipe to make apps smarter and better. They want to be the bucket like Facebook.

293 points by exolymph 11y ago

Aw that's too bad, I just discovered Emojitracker yesterday and thought it was really cool. Also: is the service being removed colloquially known as the "Twitter Firehose", or something else? I recently implemented a feature that uses the standard rate-limited API. I went through a lot of pain to get the most out of the rate limits. And people would always ask me "why don't you just get all the tweets through the Streaming API?". I wonder if I had done that, how screwed I would be now.

exolymph 11y ago
It's hard to sort out the colloquial terms for Twitter features from what the company calls them =/
- mattzito 11y ago
  Okay, here we go: - Twitter Firehose - full stream of all Tweets, commercial - Twitter Decahose - 10% sample of Tweets for analysis, commercial - Twitter Powertrack - real-time streaming of posts based on query (country, language, keyword, etc.), commercial - Twitter Full-archive API - search interface for entire corpus of Tweets, commercial - Twitter Streaming API - a mini-powertrack of posts based on basic keyword searches (up to 400 keywords), free - Twitter Streaming Sample - a small sampling of Tweets for statistical analysis, free There's a couple more esoteric product offerings from a data perspective, but these are the relevant ones. I think it's a pretty broad breadth of different options, with different functionality at the free and commercial options.
xutopia 11y ago
What did emojitracker do?
- wpietri 11y ago
  http://www.emojitracker.com/ http://www.emojitracker.com/
- spiznnx 11y ago
  It still does, for a few more weeks. http://emojitracker.com/ http://emojitracker.com/ Basically live tracks tweets that use emoji, so emoji are ranked by frequency.
- NSAID 11y ago
  Emojitracker gives live, realtime usage numbers for all emoji used on twitter. http://emojitracker.com/ http://emojitracker.com/
mattzito 11y ago
So, Twitter has long had two different classes of access: - "Twitter Developer" access - which is what you find here: https://dev.twitter.com/ https://dev.twitter.com/ - that includes a streaming API that is relatively limited in terms of volume of data and operators that are allowed, but is free and open - Gnip Twitter access - this is commercial access that would include the Firehose option (though that's not very common). But basically this is a paid offering that offers much more functionality than the developer platform. In this specific case, the author is just losing elevated access to the streaming API: https://dev.twitter.com/streaming/reference/post/statuses/filter https://dev.twitter.com/streaming/reference/post/statuses/fi... I'm guessing they had extra keywords or multiple connections available so they could track those keywords. Nothing to do with the firehose or anything else. Just a user who had an exception to the official policy that's being taken away.
- cmyr 11y ago
  For a long time twitter has offered specific developers who were doing projects they thought were interesting the ability to consumer a higher volume of traffic using the standard streaming endpoints.

aioprisan 11y ago

Another example of Twitter taking steps to kill their platform dreams and push developers away. So much for Jack's renewed push for improved developer relations.

yeukhon 11y ago
Well this is on the front-page now. maybe he will scream at the team and say "don't shut it down."
- pc86 11y ago
  I don't think Jack gets his news from HN.
  - yeukhon 11y ago
    He may or may not, but I wouldn't doubt their PR team already pinned HN as a source to consume. Plus there is https://twitter.com/newsycombinator https://twitter.com/newsycombinator. #twitter, definitely.
- aioprisan 11y ago
  You mean like they shut down the Twitter share counter data right after Jack had a big conference once he came back as CEO promising better developer relations? So, I doubt they care about the state of developer relations, as a platform.
mattzito 11y ago
This is a situation where one developer had a special exception carved out for them for a non-commercial project that everyone thought was cool. That access persisted for three years until someone made a decision that giving free access to something that normally they charge quite a lot of money for is not worth it. Twitter offers several free endpoints for developers, including search, streaming, a raw sample of tweets, and the like. They all have rate limits and the like attached, but they're real, free, access to their data. For people who want bigger, broader, deeper access, they sell access to that data. That's been the case for the last few years. In this case, Emojitracker unfortunately falls into the cracks in between, where they need deeper access to do what they want, but it's not a commercial enterprise. I agree, it would be nice of Twitter to continue giving Emojitracker free access, but are they obligated to do that? They are a business, and providing access to millions of Tweets for free seems like an unfair obligation to ask them to continue indefinitely.

joeblau 11y ago

Based on http://www.emojitracker.com http://www.emojitracker.com, there is a lot of Joy, and Love on Twitter.

TaylorGood 11y ago

With @jack having tweeted about this just months ago, talk about an internal disconnect..

duaneb 11y ago

> So even if Twitter were to give me free access to the Gnip APIs, others then wouldn’t be able to take my code and build upon it I mean, isn't this the case anyway? Few people have "elevated access to the Streaming API that Emojitracker depends on in order to operate at its high volume"

grandalf 11y ago

I've often wondered why Twitter seems to do this again and again in spite of what appears to be common sense and good stewardship of its platform and ecosystem. I think it's because Twitter is actually a successful, fairly open ecosystem that attracts all kinds of use. What seems like ham-handed or draconian decision making is just Twitter misunderstanding its role as infrastructure provider. What we all implicitly want from Twitter is for it to embody the values of net neutrality. If someone can build something cool on top of Twitter, why not let it happen? Twitter's investors have pressured the company to monetize, and most of the bad platform decisions seem to be tied to those efforts. Twitter's folly is assuming that usage patterns are "bad". It should view all usage in a content-neutral manner and provide and support APIs to let the ecosystem grow. It is the hubris of caring too much about content and content engagement metrics that is driving this folly. Twitter is a very simple broadcast graph. This is incredibly powerful, and while it's not quite as lucrative to own as Facebook (with users habituated to seeing the same boring stuff from the same boring people every day and clicking on sponsored content), it's far more powerful and is likely to change society more than other social media platforms. All this is why Twitter should pivot and become a decentralized infrastructure... of course this will never happen. I'm predicting that Twitter will die a slow death as it tries to copy Facebook's revenue generation approach. It is also copying Facebook's real name initiative by caring too much about content attribution and verified accounts (which users don't care about). Eventually we may see a decentralized broadcast graph, but until then it makes sense to expect Twitter to behave like a cognitively impaired dictator with respect to its ecosystem. I think most of this is driven by intense cost-cutting measures and intensely misguided focus on revenue metrics in terms of content and engagement. Twitter is not a product, it's an infrastructure.

mschuster91 11y ago
> It is also copying Facebook's real name initiative by caring too much about content attribution and verified accounts (which users don't care about). I fully agree with you, except on this point. With all the shitstorming, the mis-attribution of quotes, massive and massive amounts of fake accounts etc. going on, the Verified by Twitter program is a very good thing especially as it does not force you to publically post your real name. However, it could be expanded for greater reach.

roosterjm2k2 11y ago

This is a very entitled piece... and a very entitled thought process... When you are using someone else's platform, they still make the rules... this line in particular: "it wouldn’t help other developers who I believe deserve to get the same level of opportunity as me." ... nobody "deserves" access to any API... period. I don't understand the entitlement in so many tech projects, who get upset when a 3rd party they are freely using changes their service... you were never entitled to it in the first place, you merely benefited from its existence and should be appreciative of what you were allowed to do instead of feel entitled to it.

nonanako 11y ago
I think it comes from many companies branding themselves as altruistic and developer-friendly. I don't think people would be surprised at this move had Twitter said: "Using our platform? We own you. We will pursue all potential profit deriving from our platform to the fullest extent of the law."
mroth 11y ago
Author here. To be clear, I don't believe I deserve free or special access to anything. The point I was trying to make was actually more in the opposite direction: I wouldn't want to receive free/special access as an exception now simply because the project has become "famous", if the same opportunity doesn't exist for other independent developers when their projects are starting out -- I just want to encourage a level playing field. (When Emojitracker originally received elevated access 3 years ago, the project was not famous or well known at all, it was just a slightly more formal version of Me saying: "hey check this out, I think this might be a cool hack but I need access to 900 keywords instead of 400, that ok?" and Twitter: "yeah cool, no problem dude.")
- Bedon292 11y ago
  So you are still accessing the streaming endpoint, but you have more keywords than they normally allow? So you are not accessing the entire Firehose and parsing from there, right? Do you regularly update it to include new emoji when they come out? I am thinking about ways to do it on the free stream still, just trying to see where you are already at.

drinchev 11y ago

Yeah. I was recently building a simple bot for Twitter ( a favor for a friend of mine ). Things worked out, but everywhere I go I read "API Rate limited to ... per ..." I know ... It's tremendous amount of data going on there, so I just don't want to imagine how hard it is to scale it and offer that to everyone. Nevertheless it's so sad that they break promises. Twitter nowadays is a bot driven, marketing platform ( with some sort of journalism touch ). I still remember the days that I was excited when I got a new follower. Now I think it's just about "hashtags".

willu 11y ago

I have no love for Twitter, Gnip, or the way they've treated API devs over the years but this particular use case (ranking emojis by frequency of use) seems like it could be just as well served by the free sampled stream. I can't imagine anyone needing the exact usage counts vs. the relative popularity. Also, the author's stated goal of making it open and immediately usable by other developers would be better served by using the access level that is freely available to everyone.

TheHappyRock 11y ago
Agreed, I don't think the full stream is needed either. It think being built of the free stream will serve the needs of making it open and still provide interesting data. Relative(sampling) usage graphs even seem more digestable and interesting to me.

kmfrk 11y ago

I'm surprised there are still developers left for Twitter to piss off.

bobbyadamson 11y ago

I don't understand why people continue to build things for services that have a track record of shutting things down.

stevebmark 11y ago

This makes sense to me. Granting free firehose access to a small, fun project like this seems like unnecessary overhead for Twitter. I think their mistake was granting it in the first place.

Razengan 11y ago
Exactly. Wasn't HN almost unanimously in support of GitHub throttling CocoaPods barely a week ago? How is this any different? Honest question.
- spriggan3 11y ago
  CocoaPods was throttled not shut down. Twitter has an history of shutting down access to its API without notice or warning when a project becomes too popular.
  - Razengan 11y ago
    What baffles me is that CocoaPods was actually providing something of value and their usage fit in with the purpose of GitHub (serving code.) EmojiTracker appears to be essentially frivolous and puts stress on Twitter's platform for no good reason. GitHub and Twitter both provided a free service and then modified it for a specific user without warning. I would say throttling CocoaPod affected more "legit" users than "shutting down" EmojiTracker did. (I put shutting down in quotes because it seems EmojiTracker still has limited access to their API?) How come Twitter gets called out as unconditionally Evil, yet GitHub was just looking after itself?
jacquesm 11y ago
But then once you do grant it you should only revoke it for a really good reason (such as abuse or some other violation of the terms of service). Otherwise it should simply continue.

JamesMcMinn 11y ago

Everyone with elevated access is losing it in June as far as I'm aware - at least that's the case for academic uses. There will be academic pricing for Gnip, although we don't know how much it'll cost, I'm doubtful it'll be low enough for most academics to afford. They've been telling us that they planned to shut down free elevated access down for well over a year now, so this doesn't come as a huge surprise. Twitter seems to do regular audits like this, and at least for our academic work, we've had to tell them what we're working on etc. every 6 months or so to keep elevated access. Having said that, looking at the volume of data that Emojitracker is pulling down, it doesn't seem like it would be too difficult to spread the tracking over 2 or 3 separate connections. The UserStream API, whilst not designed for server-to-server connections, would probably allow Emojitracker to spread the ~900 emoji they track over 3 streaming connections using only a single account.

CoolAssPuppy 11y ago

Hi, I'm Prashant. I run Developer Relations at Twitter. I want to clarify a few things here. In the past, Twitter had little formality associated with granting elevated access. It was very much, "Hey, I want to do this cool thing!" "Okay, sure, here you go." As Twitter's platform business has matured to include many businesses building billions of dollars worth of social media monitoring and other tools, we've formalized the process to becoming a Twitter Official Partner (partners.twitter.com). Over the last six months, we've started contacting all of those API key holders with elevated access and asked them to clarify what they're doing. In some cases, we know the business and business owner and reach out personally. In other cases, like this one, the business is listed in our systems as "N/A", so we send the template mail. In the email, we encourage people who believe their app is within the bounds of acceptable use cases on Twitter to contact us directly, and provide a link to do so. The owner of this app elected to blog publicly about the situation before contacting us, which is unfortunate. We have contacted the owner of this app and hope to resolve this situation, as we do with hundreds of other developers on our platform. We do occasionally provide exceptions for apps that are non-commercial (not-for-profit, no ads, etc.) Note also that in this instance, the notice is NOT about shutting down this app. It merely inquires about why the developer needs elevated access, something that is typically reserved for our business partners. None of this is related to continuing to use the Twitter API or our commitment to enabling developers to build on our platform (Fabric, Gnip, our Ads platform, and the Twitter API).

abritinthebay 11y ago
Simply put - your "template mail" is terrible. If what you just said is what it the email meant to communicate then it fails at every level. Please don't try to shift the blame for Twitters poor communication to the recipient because they "elected to blog publicly about the situation". The "situation" is entirely created by Twitter. Twitter has a (recent) history of terrible communication with developers and it would not surprise anyone if what seemed to be the case WAS the case. If it is NOT the case, as you say, then you really need to improve your company communication.
bby 11y ago
prashant this won't save you.

subdane 11y ago

I'm "confused," what in the world are we supposed to make of this tweet https://twitter.com/jack/status/693296738145140736 https://twitter.com/jack/status/693296738145140736 and this statement? http://recode.net/2015/10/21/twitter-ceo-jack-dorsey-apologized-to-developers-for-past-confusion/ http://recode.net/2015/10/21/twitter-ceo-jack-dorsey-apologi...

aioprisan 11y ago
It means Twitter as a platform company doesn't care about keeping successful applications on their platform. Words (Tweets?) are cheap, actions speak louder. In this case, it's a resounding and massive door closing on an application that Jack said he loves and probably thought is/was good use of the platform.
developer2 11y ago
The telling quote from that article is this: > “Developers took to our service from day one and [gave] Twitter a much more global reach." Twitter would not be where Twitter is today without outside developers building their company for them, through the creation of tools that users actually want to use to interface with Twitter. Twitter as a standalone product does not have a future. Every change they make that turns more developers away from the platform continues to reduce the number of users globally who are using Twitter in some fashion. It's a downward spiral. Twitter's demise is inevitable if they continue down this path. Developers should assist them by abandoning the platform in droves. There won't be much of a platform left to work with anyway, so why bother? Let Twitter be the next Yahoo.

williamcotton 11y ago

Is any of this to be expected? Twitter is a private company and they have to create artificial scarcity to sell access to their platform. We still live in a market economy, no matter how much everyone on Twitter would like to think otherwise. They're just trying to pay the bills. Treating Twitter like a public platform is a disservice both to actual public infrastructure as well as the roll of private enterprise.

halestock 11y ago
Twitter is a public company.
- williamcotton 11y ago
  What you mean to say is Twitter is a publicly traded company owned and operated by private individuals. Those private individuals have rights and protections granted by the government jurisdictions in which Twitter operates.

Lazare 11y ago

> In the email, we encourage people who believe their app is within the bounds of acceptable use cases on Twitter to contact us directly [...] the notice is NOT about shutting down this app. It merely inquires about why the developer needs elevated access, something that is typically reserved for our business partners. What? This is absurdly, obviously, completely wrong. Did you read the email you guys sent? Either it's forged, or you're talking out of your ass. Your response is great, reasonable, and, as a developer, would improve my opinion of Twitter if it had the slightest link to reality. > The owner of this app elected to blog publicly about the situation before contacting us, which is unfortunate. It's unfortunate they took your very clear email at face value and reacted in the way it was clearly intended to invoke? "Hey, when we told him to fuck off and die, we meant 'why don't you drop by the offices and have some cake!', why are you guys so rude?"

dang 11y ago
You can't comment like this here, regardless of how good a point you may have. If you can't be civil, please abstain from posting until you can. We detached this comment from https://news.ycombinator.com/item?id=11300071 https://news.ycombinator.com/item?id=11300071 and marked it off-topic.

tangled_zans 11y ago

This is just... wow. The guy spends three years developing a product on Twitter's platform and they just kill it off with one e-mail? It's fair enough if they need to change their API policy, but at the very least they could have acknowledged all the work that he's done and found a way to help him and the other developers in the same position to make the transition as painlessly as possible. Not... whatever corporate BS this was. Given how much Twitter is struggling already, is this really the time to be alienating more of its community? Personally, I've had a really nifty side project that I've been working on since last year that I wanted to launch on Twitter's platform, but as of a month ago I've all but given up on actively developing it. All the people that I know on Twitter with whom I wanted to share it have either left already, or are only sticking around because as-of-yet there's nowhere else to go that does what Twitter does. So I'm curious to hear some perspectives on this: clearly the direction that Twitter has taken is making a lot of devs unhappy, but what would be the right thing to do? What sort of API access would people ideally want to see it provide? What would be the ideal best-case scenario necessary for the devs to want to start coming back and putting their time into building the ecosystem?

wslh 11y ago

Three hours later and have you realized that CoolAssPuppy (Twitter itself) hasn't answered any specific question? If you write in HN you expect a conversation happening.

JoblessWonder 11y ago
Probably because Twitter's PR people probably reached out to him and told him he wasn't helping anything and needed to stop posting. I wouldn't be surprised if they walked to his office and unplugged his keyboard. It was bad. He posted something that made him look like an idiot or a liar. I'm not saying he is either one of those, but the post he wrote makes almost no sense when compared to the e-mail presented in the post. It is like he didn't even read it. If he was trying to say that they never sent that e-mail, then that point wasn't clear. I'm sure he had the best of intentions but it really blew up in his face.
skeletonjelly 11y ago
I used to subscribe to a service that emailed me when I got a response on HN but that stopped for some reason. HN Notify I think it was called. What do people use apart from refreshing their own comment pages?
- kbenson 11y ago
  Dan Grossman, who's active in this very story, set up a replacement[1] a while back that I've been using and it seems to work well. 1: http://hnreplies.com/ http://hnreplies.com/
  - skeletonjelly 11y ago
    Legend thank you!

rdl 11y ago

Twitter fucked over every developer who trusted them a few years ago, and is now trying to recast their business as developer tools and services (since the core Twitter business probably is not a $20b business...). One of their biggest roadblocks is getting developers to trust them again. And then they do this?

x1024 11y ago
The only way for them to get developers on their platform is to wait for the next generation of developers. Literally nobody will base anything on their stuff now, and this has been the case for a good few years.

grizzles 11y ago

Why do people keep building stuff on Twitter? All the app developers that have been fenced off should create a simple protocol to make Twitter irrelevent. Brent Simmons suggested a great way do it awhile back: http://inessential.com/2012/06/29/matthew_on_twitter_restrictions http://inessential.com/2012/06/29/matthew_on_twitter_restric...

pmlnr 11y ago
Why do people keep building stuff on any silo? Take a look at Known (https://withknown.com/ https://withknown.com/) and Woodwind (https://woodwind.xyz/ https://woodwind.xyz/): one to have an actual website, the second to have a stream of content you want. Done. Forget silos.

PostThisTooFast 11y ago

This is why you don't build your enterprise on that of a third party who can dick you over and waste years of effort in an instant.

spriggan3 11y ago

At that point anybody whose business relies on Twitter "free API" deserves to be in that situation. It's a well proven fact that Twitter will shut down every 3rd party client or app based on their API which is a bit too popular. Never build a business on top of a "free" API, especially Twitter, because there is 100% chances Twitter will shut down you access.

jonesb6 11y ago

Has anyone here built a product that relies on Twitter's API for business critical features? How would you describe your risk tolerance, how long have you been in business, and do you have a personal or professional relationship with Twitter or Twitter's employees?

siegecraft 11y ago

This just the extract value before exit phase, right?

egfx 11y ago

I developed a product that connects Twitter and Facebook. I always thought that Twitter would be the enabler because of it's open street like atmosphere and Facebook would be difficult to work with because it was a walled garden. The exact opposite was true. Over time, Twitter would get more restrictive and difficult to work with while Facebook enhanced the look and feel of the stream and how the tweets appeared, emphasizing and highlighting them. In essence I think the product spoke to the nature of the whole issue. The tweets left Twitter to seed Facebook with good interesting, connected data. Twitter would know little about the context while Facebook would get enriched context from the tweets. Twitter doesn't want to be a dumb pipe to make apps smarter and better. They want to be the bucket like Facebook.