4 ms·
A "firehose" API for the net would be awesome, but it probably needs to be Open Source, in order to avoid being driven towards monetization. Of course, without
by SwellJoe 14y ago
A "firehose" API for the net would be awesome, but it probably needs to be Open Source, in order to avoid being driven towards monetization. Of course, without the drive to make money, Twitter may have never taken off quite the way it did, and may have never had the money to spend on SMS and other expenses they were dealing with in the beginning to create that firehose of data.
The Open Source twitter clones (like identi.ca) have modeled what Twitter became, mostly, rather than what Twitter could have been...so, as far as I know, there isn't much out there that answers this description. I'd started working on something I was calling SYSRSS years ago, which was to be a realtime systems human and machine readable data stream protocol and implementation, but never got a minimum viable product out the door, as I could never quite figure out the right niche for such a thing. I think it may have been too specific, though building on top of an existing popular data format and protocol was probably a valid choice.
In short, I agree that Twitter could have been more awesome than what it is. But, I don't know if I could have made it better, or that having the API guys win would have succeeded. I haven't done any research into that field lately...I wonder if there are any new developments in the Open Source world for this kind of thing?
- cloudmike 14y agoYou should check out Simperium (disclaimer: I'm the co-founder and Twitter's former platform lead is one of our advisors/investors). While it's not immediately apparent yet, we're pushing in this direction: it's easy to create a firehose for your app, and we're working on ways for apps to easily interconnect with each other in a less file-centric, more API-centric way. The niche we chose to start with is device syncing, based on our experience building Simplenote (which is to Evernote as Twitter is to Facebook). We know there's a need for something more flexible and powerful than iCloud in this regard, so that's what we focused on initially. But we believe a good solution for moving data between devices is also a good solution for moving data between people and 3rd-party apps. Simplenote's 3rd-party ecosystem looks similar to Twitter's in the earlier days. We're taking the path they didn't take. Rather than shoring up Simplenote vertically, we're focused on extending our backend data layer horizontally and generalizing it for a variety of apps and services (though specifically not as an all-encompassing "backend-as-a-service"). We'll be announcing our open source strategy soon. This is all really hard though, both technically and as a business.
- paulsutter 14y agoCan you think of examples of any open source standards that emerged from pasting on an open source strategy later? I ask because I can't think of any, but I may be wrong. I'm contemplating a realtime data sharing system now, but it's hard for me to consider using a commercial platform from a startup. If your API was already open source, with a real open source community that spans several organizations already in place, I'd be a lot more eager to use your service. Basically, an open source api is only appealing to me if it seems impossible for it to be hijacked by one sponsor, no matter how well meaning. Who knows, you could get hit by a bus and wake up as your own evil twin. Love to hear your thoughts
- cloudmike 14y agoWe only launched Simperium a few weeks ago. While it's production-ready (powering Simplenote), it still has a beta label for the simple reason that we haven't announced pricing or our open source strategy. I agree, "pasting on" an open source strategy in a reactionary way doesn't have a great history of success, but similarly, I think it's also hard to build very ambitious open source solutions that are entirely open from the moment of conception. Not impossible, but in any case the path we chose was to build a first version behind closed doors according to a strong vision based on experience and talking to fellow app developers. Having done that, we're excited to start opening it up and we don't view this as pasting on a strategy. We've been proactively considering the implications for our technology and business from the beginning. I think a great, open circulating system for the internet can be as significant as an operating system like Linux.
- paulsutter 14y ago> I think it's also hard to build very ambitious open source solutions that are entirely open from the moment of conception Yes it's hard to build an open source standard using an open development model. On the other hand some people think its impossible to build an open source standard without an open development model. Open source advocates argue that the very barrier to something becoming an open standard is your wish to own a strong consistent vision. They also argue that having a single host company that has a head start over others will discourage others from jumping on board. I sincerely hope that such an approach could lead to a real open standard. I can't think of a case that has, though I do hope to be proven wrong. Cool project, for sure. I bet you can succeed as a commercial platform, you're solving some hard problems and will make life easier for lots of developers. Would be cool if you can drive an open standard too.
- politician 14y agoTelehash is the Open Source "firehose API" you seek. The proposal describes a DHT that routes JSON. However, it seems like I've been waiting for it for years. Maybe it's the obscure telephone lingo that the API is described in terms of that's discouraging would-be implementers, but it's nothing that a good old facade can't fix. http://telehash.org/ http://telehash.org/
- jchrisa 14y agoI like to think of Telehash as the answer for p2p routing. It's powerful and there's working implementations. The other piece of the puzzle is REST in the UDP context, for that, CoAP looks really well put together: http://tools.ietf.org/html/draft-ietf-core-coap-10 http://tools.ietf.org/html/draft-ietf-core-coap-10 I have a weekend project going to blend the two and make magic, but I haven't put enough time into it yet.
- jeremie 14y agoIs there decent CoAP implementations? That's unfortunately one of the big stumbling blocks with universal p2p transport yet :/
- demachina 14y agoTelehash is radically decentralized. How do you get a firehose out of that? It would be more like a point to point garden hose wouldnt it? I would like to be proved wrong, but I think Twitter/centralized and Telehash/decentralized will end up looking and acting like two very different services. If the Twitter as an API group had won out the two best outcomes I can think of are either a buy out from Google, or a move to a foundation like Wikipedia. Google could afford to carry it, like it does Youtube, just so they could own a huge firehose of information and use it elsewhere in search ads, without trashing the stream with ads. Needless to say founders looking to get rich wouldn't go for the low returns of the foundation model but it would have been the best possible outcome for the well being of the future Internet having and open, censorship resistant message bus and API, assuming you could get enough donors to support it.
- chipsy 14y agoOne of the most ambitious projects I've seen in this vein is psyced ( http://www.psyced.org/ http://www.psyced.org/ ) which starts from a redesign of IRC and has gone on to accumulate a kitchen-sink's worth of protocol compatibility(from the front page: Jabber™/XMPP S2S, IRC, TELNET, HTTP, SMTP, OAuth, XML, RSS; limited/experimental code for XMPP C2S, Java Applets, Status.Net, Twitter API, WAP, NNTP.) The core protocol is a gorgeous "Right Thing" design which has had many years to mature, and attempts to cover all venues: IM, chat room, microblogging, mail, profiles... However, like with all of these open-source efforts, traction as a network seems to be missing.
- J_Dalton 14y agoactually the core PSYC folks have moved on to develop http://secushare.org http://secushare.org which is a P2P social beast "done right" with technologies and syntaxes much faster than XMPP or JSON. secushare is getting plenty of attention in the european hacker community and projects like freedombox. it's the missing link basically.