11 ms·
Twitter Cuts Off DataSift To Step Up Its Own Big Data Business
- ColinWright 12y agoNever, ever trust another company for your business model. Here's a previous comment: https://news.ycombinator.com/item?id=9262245 https://news.ycombinator.com/item?id=9262245 That was about FaceBook, but exactly the same holds here. Quoting: Certainly I'd be incorporating an exit strategy, and my game plan would be to leverage off FB to start with, to grow as fast as possible, develop my own community and eco-system, and then be able to shed FB and replace it with something else.
- vdaniuk 12y ago>Never, ever trust another company for your business model. This is just another thought-terminating cliche not unlike "if it is free, you are a product" and I can't help but respond. Large technological companies create ecosystems where smaller players can coexist with "hosts" in a mutually beneficial relationship. Ecosystem companies are not able to predict the future of the ecosystem, its evolution for all intents and purposes is mostly unpredictable. Sometimes the relationship works out, sometimes it doesn't. Startups have 70-90% failure* rate, the possibility of the host cutting of some parts of the ecosystem to integrate their food chains into the corporate borg is factored in the business and investments strategies of the smaller companies. You should not trust another company for your business model, but building your tech business in the ecosystem of a large fish is an absolutely valid and quite possibly lucrative strategy. *definitions vary EDIT success/failure
- raverbashing 12y ago> Startups have 70-90% success Don't you mean failure rate? > You should not trust another company for your business model, but building your tech business in the ecosystem of a large fish is an absolutely valid and quite possibly lucrative strategy Yes, as long as if you have a Plan B if/when your relationship is cut (a valid Plan B is to close shop and try to do something else with the expertise gained)
- vdaniuk 12y ago>Don't you mean failure rate? Sure, thx >Plan B Certainly. Datasift has 25 various information sources. Twitter is really important, but doesn't seem to be critical for their platform.
- bane 12y agoIt is if it's 90% of their business by revenue. I've worked for small companies in the past that had dozens of partners, but most partnerships don't really do anything except give you brand logo to mutually share on each other's web pages. It's like anything else, partners provide revenue to you following a power-law distribution, with a long-tail that provides nothing and a very select few that make up the lion's share.
- ColinWright 12y agoI disagree that it's a "thought-terminating cliche" - it certainly shouldn't be. There's a difference between trusting someone, versus using them to your advantage but having an exit strategy. I agree entirely that there is value, sometimes significant, in exploiting the ecosystem created by a larger fish. You'll see that in my comment, wherein it says: ... leverage off XXXX to start with ... be able to shed [it] and replace it with something else. You are saying the same thing I've said, but emphasizing that starting in the jaws of a larger fish is sometimes worth doing, and sometimes they'll deign to let you survive there. Consider, though. If you become successful - which is surely your aim - then you will come to their attention, and they will do whatever they like. So I repeat: Don't trust them. Use them, yes, but don't trust them.
- vdaniuk 12y agoSometimes trust is essential and diversifying would be a loosing strategy as the larger partner will react specifically to your diversification but will help develop your business until you are not looking for something on the side. Generic business strategy rules don't work anymore, the conditions are changing too fast. I agree, however, that blind trust is always a bad path.
- frabcus 12y agoI think this needs more detail as to what stage in the tech lifecycle the businesses are in. Obviously, I'm going to trust an electricity company to provide my business with electricity. I'm even going to trust laptop manufacturers to continue to provide laptops I can code on. In both those cases, the thing I'm depending on is a commodity, where there is also either strong competition or regulation. There isn't much risk in depending on them. The stage to be wary of is when a whole industry is still at product stage and people are trying to move them into being platforms, or using them to Innovate/Leverage/Commoditise (ILC, see Simon Wardley) to play ecosystem games. i.e. Where the business models and how the industry will be vertically and horizontally sliced are not yet settled. Those are the ones you should be wary of dependence on.
- jackgavigan 12y ago
- jacquesm 12y agoIt's not a thought terminating cliche at all, it is very sound business advice and you will ignore it at your peril. Not all that long ago I did tech dd on a company that had exactly one very large company as their main unwilling partner. They claimed that they were 'too big to fail', guess how that ended?
- vdaniuk 12y agoYeah, Gnip founders ignored this "very sound business advice" and raked in ~$130M. Startups are risky, startups that are growing in the ecosystem of one large megacorp are even riskier but they are prime acquisition targets. You decrease success chance for higher yields. This is like business 101. The advice "do not base your business model on another company" evaluates to "do not increase your risks" which would be an absolutely absurd advice. If one qualifies the original statement with several conditions for different contexts, it may become a solid advice. As it stands this model of understanding business strategies is useless.
- jacquesm 12y agoSuper for gnip. But basing your business model on being allowed access to other companies' data fails far more frequently than that it succeeds, and it fails far more frequently than the already pretty bad chances that you have as a start-up to begin with. See the thread subject that you're commenting in for how Gnip's competitor fared.
- blumkvist 12y agoYeah sure, the countless resellers of MSFT, Oracle, IBM, VMWare, Redhat, EMC all do not understand business at all. If only some of the uptards knew how little they know, yet how much they run their mouth.
- jacquesm 12y agoI think you're mis-understanding the issue, but that's fine. Let me try to explain it a bit better: if you use documented interfaces and have an established supplier role with a larger company that's fine. But if you are taking data from some API and you're re-purposing that data in a way that the company supplying you with that data may not agree with then you may one day find your access terminated. You probably should not include RedHat in a list like that.
- gtirloni 12y agoThe problem is that there is nothing preventing the "ecosystem host" from deciding that a niche an "ecosystem player" is exploring is lucrative and worth pursuing. You can fly under the radar for a while but if Twitter/Facebook/Google/etc decide your area is lucrative, they will go after it, whether that will kill one of their "ecosystem players" or not. There is no partnership involved, they provide a service as a huge "blob" that is making them money and they don't care about 1 out of 1 million of their regular users. Your small company cannot call itself a partner, you've as much leverage as any user when it comes down to deciding to sacrifice you in the name of something else more lucrative. I'd say explore the benefits to bootstrap but have a strategy to depend less and less on them or your time will come. And if you really want to tie your business survival to one of these ecosystems, get yours and theirs responsibility down in writing (and even that won't help much because a breach of contract for them will mean nothing in terms of dollars, while for you it'll be your existence).
- droopyEyelids 12y agoAh. So you mean, "If you want to be perfectly secure for all future time you must be completely self sufficient." Can't really argue with that. Maybe not useful advice, but it's self-evidently true.
- ColinWright 12y agoIt's more like: If you build a great business based on something over which you have no control, and which is such that the ones who do have control actually may have an interest in cloning what you do and terminating your facility without warning, be sure to have a "Plan B". Yes, sounds obvious, but so many people seem to ignore it, and in some cases actively disagree with it. You seem to be disagreeing with it - what would you suggest as more useful advice and a more reasonable stance? Honestly, would you build a business based on a Twitter or Facebook facility, relying entirely on them not arbitrarily terminating your access? What is your advice?
- lifeisstillgood 12y ago
- rorykoehler 12y agoAllowing your business model be beholden to the whims of another company is setting yourself up for failure. Sure using an ecosystem built by another company can be a great way to gain traction but by limiting yourself to this ecosystem you are also severely limiting your growth potential. Not only that but you will be easily and quickly dispersed with when you become successful enough. Having validated your idea the larger eco-system company will simply replace you with a clone that they own. Unless you have some serious in-house advantage (patents and people) that is not easily replicated then you will fail.
- vdaniuk 12y agoOr be acquired by the larger ecosystem company what would be the desired endgame for many startups. This alone may be worth all the risk of being squeezed out of the market. The example is obvious: Gnip vs Datasift.
- rorykoehler 12y agoAs I mentioned unless you have some compelling reason to be bought (people or patents) you don't stand a chance.
- jusben1369 12y agoThere's always risk but it varies greatly by platform. Twitter is by now notorious for doing this behavior. On the other hand Apple has a flourishing ecosystem of app developers. And FB has improved greatly moving from being Twitter like to closer to Apple. It's a function of maturity (young startups have growing pains on what they want to be and how they monetize) which can definitely create collateral damage. Mature companies like Apple don't experience the same degree of angsnt. It's no accident that Twitter, which has generally underperformed and disappointed more than its peers, is still screwing up and over its ecosystem far more than others.
- JustSomeNobody 12y agoWhy not? Remora survive by not getting too close to the shark's mouth. Same applies to business.
- room271 12y agoI worry media companies are effectively in this position with regard to social.
- untog 12y agoThis is terrible for Datasift, but I really hope they saw this coming. Twitter buying Gnip should have been the writing on the wall.
- frabcus 12y agoI'm sure Datasift saw it coming. I researched this last summer. Even before I was explicitly told all the original firehose contracts were five years long, I could infer it from attitudes and behaviours. I'm excited that Datasift are launching interesting new Facebook features. They're a great company, with good technology. I hope Twitter's "no data resale" attitude leads to them becoming irrelevant, as it is so destructive of much innovation.
- bane 12y agoYeah, I'm sure it was significant source of talk within the company for a while. But what can you do? It's not like there are other Twitters out there. The value of Twitter is that there's only one. It looks and sounds like they're trying to diversify out to other sources, but none-of-them are Twitter and for whatever reason Tweets are like crack to market research firms.
- frabcus 12y agoWell, Instagram and Twitter are converging in terms of feature set. They're pretty well indistinguishable now. They've not totally converged in terms of audience, but I think have surprising overlaps. Lots of Twitter data use cases you can get from Instagram, I bet.
- MichaelGG 12y agoAnd Facebook/Instagram has no official data policy, right? They don't offer paid data from what I can see. You can sign up as a dev, email them if you need more access, otherwise...? So as far as data access goes, Twitter and FB/Instagram are pretty much on the same page. Except Twitter's docs are way more accurate and nicer, probably because it makes money for them and isn't a "throw it out there and see what happens" kinda thing.
- frabcus 12y agoAt ScraperWiki, we had our Twitter to spreadsheet tool cut off (at a much more modest size!). I wrote up an analysis of where you can get Twitter data from and why which is relevant: https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/ https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...
- bane 12y ago> He said that before this, the pair had been discussing a renewal of the deal. This should be a case study for anybody looking to get into a BD role. A partnership is fundamentally coming from a position of weakness, and the relationship is a way for both partners to partially fill in whatever gap they believe they have in their strategy. Sometimes partnerships go south, especially when the relationship is highly asymmetric. In this case Twitter was the stronger of the two partners and they decided not to renew. That sucks, but that's life. Don't build your entire business around a single partnership, because that signals to your partner that all they have to do to get rid of you is replicate what you do and cut you off. Once you've gone through the blood sweat and tears to prove the market, you've just given your larger partner free market development. DataSift's website lists a number of data sources [1], but Twitter comes up first. More importantly, I suspect these sources are slightly massaged to look like they have more sources coming in than are really useful for their business model. e.g. is Wikipedia really in the same space as Twitter? They also have a bunch of minor sources but they're listed like they all rank equally. This is fine, most businesses would do the same to make themselves look better -- it's just advertising. Another part of business partnerships is that you have to expose your weakness to your partner. Twitter is going to know at some point that DataSift's business model is heavily built around their firehose. They might even have some sense about what volume of business all the rest of those sources makes up to DataSift's business. Twitter also, through their other partnerships, has a better sense of the size of the Twitter Analytics market. Even if DataSift's piece is 15% of the overall market, Twitter has other partners it can use to get a feel for the other 85%. Partnerships are dangerous. Know what you're getting into before you get into them, and try to only do them with similar-sized partners or you'll find yourself getting screwed over/screwing somebody else over. 1 - http://www.datasift.com/platform/datasources/ http://www.datasift.com/platform/datasources/
- jackgavigan 12y ago> Sometimes partnerships go south, especially when the relationship is highly asymmetric. It's arguably not really a partnership if the relationship is highly asymmetric.
- steven2012 12y agoAnyone doing business with Twitter is a fool. There are few companies that show as much disdain to their partners as Twitter does. With partners like these, who needs enemies?
- DanielBMarkham 12y agoTwitter is quickly approaching the point where it has lost my goodwill and I will actively look for something to replace it. (Apple has already passed that point) I understand the way big ecosystems and walled gardens work. I also understand that every large technology company that has come down the pike in the last 30 years has wanted to conquer the known universe. That does not mean I have to actively assist them in this activity.
- tl 12y agoI'm losing faith that this is a viable approach. For example, laptops are a domain where your options are Apple, Lenovo, Dell, or a company that whose future in the laptop market is bleak. All of the options have done goodwill losing things in the past. Can't we all just "conquer the known universe" together? We need a framework for educating these companies that open solutions have greater reach than closed solutions even if they are less directly profitable. The people making decisions are often successful enough this should be possible.
- vijayr 12y agoI am not sure these companies don't actually already know "open solutions have greater reach than closed solutions". They are run by extremely smart and capable people - but they are more concerned with pleasing the shareholders and hence short term profits than creating truly open systems (with some exceptions here and there) and we can't blame them for it. A more effective approach would be to create/support open systems and educate consumers instead.
- fourstar 12y agoTwitter is done. I give it 5 more years shelf life.
- ransom1538 12y ago2008 - Facebook had a meeting with us Crowdstar describing how our current business model (quizzes) was getting their own section in their TOS, and would cease to exist. 2010 - After switching to a game model and doing quite well - neck and neck with Zynga. We were walked into a Facebook meeting room and told a bizarre story how Mark was upset about friend feed forms. They explained that we could only share (by means of post feed forms) our games with people that already had our games. Pleading did nothing, they ended that model. Zynaga is still flailing around. 2012 - I changed companies and focused on mobile advertising. Another decent path with 12 employees or so. We got a phone call . [EDIT REMOVED REST OF STORY] 2015 - I received another phone call from a large company (They deal in search) . I wont go into this one. * Phone calls from large companies are extremely bad. That means they don't want things in writing. So sure, have fun with working with large companies. Just remember one thing: they hate you.
- harperlee 12y agoHonest question, because I'm neiher american nor familiar with their common law: Let's call the company that called us [redacted] Does that really hold water, legally speaking? Because we all know what are you implying...
- ransom1538 12y agoThanks removed.
- mcintyre1994 12y agoFYI you haven't edited the dupe post (if you can), I assume it's the same as the original here.
- dang 12y agoSince it was a duplicate anyway, we'll delete it.
- raverbashing 12y ago
- deleted 12y ago[deleted]
- njyx 12y agoHave some fairly strong feelings about how much of a bad idea this is for twitter in the long run - posted here: http://www.3scale.net/2015/04/loosing-innovation-twitters-firehose-mistake/ http://www.3scale.net/2015/04/loosing-innovation-twitters-fi... I have no idea if DataSift and NTT were good partners, but not having anyone else access the Firehose suggests they think they can cover all significant use-cases. This almost always turns out to be wrong. It also sends another chilling signal to say "we're a platform, but when we feel we want to roll over part of the ecosystem we'll do it".
- danmaz74 12y agoTwitter is still selling data to analysis companies, they are "only" forcing them to buy the data directly from Twitter (aka Gnip).
- MichaelGG 12y agoIt's not like Twitter can't offer some slower Firehose or more advanced filtering or whatever. Or offer some EC2 images to spin up to handle it. It's only 500M items a day. At that level you've got plenty of options as far as hardware and software. (You could even record it into a normalass SQL DB to log each one if you wanted.) You may need some useful buffering for high spikes, perhaps some message queues with lots of RAM.
- ultimape 12y agoThis made me giggle
- MichaelGG 12y agoWhy? A couple years ago I was running a telecom, and we had around that volume of calls a day. Each call required not only routing instructions that hit several DBs, but also a transactional INSERT/UPDATE to account balances. Single node SQL Server handled it fine. (Though it was less than optimal at doing non-OLTP stuff.) Twitter can handle this for people (like DataSift, providing historical lookback), or they can provide template to help them spin up. It's not gonna cost thousands of dollars a month in hardware.
- phesse14 12y agoWonder how you have done it, but this post was already published. A breach on HN?
- zkhalique 12y agoThis is why we are building a new kind of social platform! https://qbix.com/blog/index.php/2013/04/a-new-kind-of-platform/ https://qbix.com/blog/index.php/2013/04/a-new-kind-of-platfo... One where you won't be cut off if you're a developer. And one which organizations can host themselves.
- tlrobinson 12y agoWhy will you succeed where App.net, tent.io, etc hasn't?
- Karunamon 12y agoBoth of those services are still alive - seems a little bit silly to say they "weren't successful" unless your definition of success means having as many users as Twitter.
- tlrobinson 12y agoFor general purpose social networks, yes, that's my definition of success. They aren't very useful otherwise.
- zkhalique 12y agoWell, for one we already have millions of users. Why did Wordpress succeed for decentralized blogs? Why did Bitcoin succeed for decentralized currency? Many organizations run it.
- zkhalique 12y agoI am curious whether the downvoters would actually post why they are downvoting this in particular. Are you really saying you want just a few monopolistic companies to control people's social experiences online?
- wslh 12y agoI feel bad for DataSift. I always understood that their business was really risky. More when Twitter acquired Gnip.
- weirdsift 12y agoI hope some of the people got good salaries. And maybe got some of the almost $80M raised. DataSift was proud of the money they raised. The said it made them "more difficult to acquire than Gnip". I was amazed when I heard that. It sounds like they are happy they took on so much money and were moved passed by Gnip. But it is worse, I think. For many services they require you to provide API keys. Rate limits are a problem? No! Create more API keys and give them us. Why are they not blocked? If any big service decides to block API key rotation ... game over? If we do it in-house, then we can use many proxies and pretend to be separate. Maybe. Or maybe not. Maybe because of VC connection DataSift can rotate API keys and with impunity. Who knows? But yes, to me it is very very risky business, but not too difficult technology. (Maybe that is why I have not had a big hit. Too little risk.) So does HN think it is successful to raise money like them? Do the founders make more than $500K/year total from this venture?
- code_duck 12y agoI was involved in a company that heavily depended on a popular site's API. We could clearly see they thought of the API's best value for them as filling in features before they had time to create them, and as a testing ground for ideas. It became harder and harder to stay ahead as our business grew, as at any time our features could be replaced with an official version. We decided to stop development. Now, a few years later our largest competitor has been doing quite well with a product along the same style... but now, without warning, they've been cut off from API access and are facing requests for hundreds of refunds. Glad it wasn't us.
- ergest 12y agoTalking to Nick Halstead, the CEO and founder of DataSift, he said that his company was “blindsided” by Twitter’s announcement, which it made without any warning to DataSift. He should have totally seen it coming and been prepared. When you don't think about the future and don't have the foresight to build competencies to take advantage of what you discover, you end up being blindsided. This is not about hating on Twitter or siding with them, it's pure competition.
- cinquemb 12y agoI wonder if using the hundreds if not thousands of twitter api credentials checked in on public repos, one could mitigate against things like this? Though I suspect companies not in the jurisdiction of the US or its "partners" will probably be more likely to get away with an approach like this if caught or public about it, while still providing value added services based on walled gardens like facebook and twitter.
- jackgavigan 12y agoIt's great for companies like Twitter and Facebook when startups build on their platform. The startups innovate and experiment with new features and business models (at their founders' and investors' expense). The lousy ideas fail, the good ones survive and the platform owner can cherry-pick by either replicating the successful feature/business model themselves or buying their favourite. Then, by cutting off others' access to the platform, the platform owner can make their own service a monopoly and capture a significant proportion (if not all) of the others' customers and revenues. Phase 3: Profit!
- MichaelGG 12y agoDataSift was simply reselling Twitter's data, along with whatever enhancements and cleaned up API niceties they offered. And getting 80% of the revenue for that (according to the article). How does this benefit Twitter? How does this benefit startups wanting to use Twitter? The data is still available, just not through resellers.
- clay_to_n 12y agoDataSift and Gnip had access to the Twitter firehose (every single tweet posted, live). Normal businesses / developers can't get access to that. Twitter's "streaming" API is a tiny fraction of the firehose. So DataSift wasn't just reselling their data - they, along with Gnip, had a partnership that allowed them to get access to the firehose. For many social media companies, the firehose is a necessity, so they bought contracts with Gnip or DataSift. Then Twitter bought Gnip, and is cutting off DataSift's access to the firehose.
- MichaelGG 12y agoRight so if we say Gnip is Twitter, then anyone can go buy data from Twitter (even if it's a wholly owned subcompany). DataSift didn't have much of a value add (OK, historical data, common filters for Twitter and Reddit and others, etc.) But nothing that Twitter can't just in-house now that the market got nice and established.
- Animats 12y agoThat's not surprising. Twitter is very controlling about their service. They had an open API only until they got market share. Then the jaws gradually closed on Twitter-related programs. Twitter strongly discourages client programs other than their own. I once wrote a Twitter client with spam and ad blocking, just to see what it would be like, but I knew Twitter would never approve it.
- vram22 12y agoI was once invited to be a co-founder of a startup (by the other founder), whose business model was to build something using the API of <insert-big-social-network-company-here>. I declined, for the same sort of reasons mentioned in this comment by georgemcbay - https://news.ycombinator.com/item?id=9364391 https://news.ycombinator.com/item?id=9364391 - i.e. "yes, this fact should really give pause to people who are thinking about basing an entire company around an API, portal or other service that is 100% controlled by a large company. Any such dependency should be seen as the huge risk factor that it is." Have read of multiple other such cases where the rug was pulled from under such startups that relied on a single point of failure (SPOT) - http://en.wikipedia.org/wiki/Single_point_of_failure http://en.wikipedia.org/wiki/Single_point_of_failure I told that founder the reason why I thought his model was not a good idea, after thinking for a while and then declining the offer.
- grizzles 12y agoTheir strategy to handle this business risk was to ...accept it? It seems kind of shortsighted given twitter has ALREADY did a version of this to a bunch of other companies/apps. At it's core, what twitter provides is no big deal. The ability to post 140 chars. They could have been quietly funding an open twitter protocol (it will happen eventually) but they chose to keep their heads in the sand.
- hellbanner 12y ago"Parasites are not crushed by the strong, they feed off of them" but sometimes the strong simply buck them off. Apple did this with their introduction of Gamecenter, killing OpenFeint, and a couple others (can't remember their utility).
- bayonetz 12y agoIt's a shame, for sure, for DataSift. We have been working with DataSift for about a year now though and it is pretty damn clunky. I'd say GNIP is probably even worse at the moment but maybe not for long? As a user, if GNIP can capitalize on being part of Twitter, I gladly welcome the improvements this could bring. I'm not super confident though as now Twitter/GNIP will also be a monopoly in the space with the usual disincentives that come along with that. Also, our use case is strongly biased towards the "historic query" features and neither service seems as interested in that as they are in the real time streaming features. Also, I had considered earlier developing a niche competitor solely focused on the historics space for companies with research-oriented use cases like ours. I suspected Twitter would be dropping this hammer though so glad I didn't waste my time!
- bsdpython 12y agoFor all the great things going on in open source software, truly open source data isn't all that common. I love playing around with third party APIs as much as anyone but as software developers we need to be extra careful that we are not building on sand. Just the thought of being cut off limits lots of interesting ideas and side projects for me. In some ways I want to build a platform just to get 100% access to some really fun data thrown off from the platform in which to build fun things on top of.
- paulhauggis 12y agoThis doesn't surprise me at all. If you are going to use the infrastructure of an existing business to start your own, you will need to have a plan for figuring out other sources of users and diversifying your risk. Otherwise, you will be out of business if you ever get too big or you do something that the larger business disapproves of. This is a good lesson in business.