10 ms·
Show HN: Deepstream.io – open-source real-time server with pub/sub and data sync
- espadrine 11y agoWhat is the algorithm used for JSON synchronization? Operational Transformation, CRDT, diff/patch?
- wolframhempel 11y agoCurrently if a conflict occurs it’s reported back to the client that triggered it and its up to it to resolve it. We are currently working on configurable merge strategies on a per record level
- Kiro 11y agoWhy would you need some complicated algorithm for this?
- pka 11y agoSo what is an easy algorithm then? Keep in mind, "newest update wins" doesn't even begin to cover all possible use cases.
- sebak 11y agoActually, for most use cases, newest update wins is sufficient if it can be done on a fine granular basis (one property of an object/document). It's what web applications have been doing for forever and being a 'realtime' framework doesn't change this if your use case isn't something like google docs.
- d0m 11y agoEasiest way I found is to use an "append-only" mentality to avoid conflict entirely. So, to take the vote example, using: [ {ts: <upvoted_time>, author: <id_of_author>. action: 'up'}, {ts: <upvoted_time>, author: <id_of_author>, action: 'down'}, {ts: <upvoted_time>, author: <id_of_author>, action: 'up'} ] You can sync from the server the changes after a certain timestamp, and the client can calculate the accurate value.
- espadrine 11y agoThat common approach is both elegant and tricky. It requires very precise clocks and time synchronization, as some non-commutative operations are order-dependent (such as list insertion). That is why Google's TrueTime API (introduced in Spanner[0]) is such a big deal. [0]: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/spanner-osdi2012.pdf http://static.googleusercontent.com/external_content/untrust...
- izym 11y agoUser A and B both have document X version 1. They both edit this before synchronisation and thus they both have their own version 2. What do?
- espadrine 11y agoGreat question. The granularity of your synchronization impacts the user experience, and sometimes the product offering itself. It can go from "replace the whole object" all the way to "string operations are merged without conflicts to match user intention", with "centralized resource locking" somewhere along the way. The goal product-wise is to ensure intention preservation, but that is hard to achieve in general as intention is tied to what the product does. For instance, if you have an integer and an increment operation, say, in a HN-like app to track upvotes. If your synchronization simply resets modified values with a newest-win approach, the following situation would lose an upvote: you have 5 upvotes, A and B both upvote you simultaneously, each setting the number of upvotes to 6. The server tells A that the number of upvotes has been set to 6, then tells B the same thing, and they are both happy. However, product-wise, it should have been 7 instead.
- elcct 11y agoLooks like "deep" is new "cloud" :)
- majke 11y agoI skimmed the docs, but I'm unsure about the semantics. The pubsub, what happens to slow clients? The same for synchronization. Basically - how it behaves when the clients are not behaving.
- wolframhempel 11y agoRecords use a version number for every update to keep track of missing messages. Events are currently unversioned and won't be queued etc. while a client is offline.
- awjr 11y agoIs this similar to firebase https://www.firebase.com/ https://www.firebase.com/?
- wolframhempel 11y agoYes, but there are a number of core differences - Deepstream.io is a free open source server, not a PaaS offering - Deepstream offers pub-sub, request-response and web-rtc call management in addition to data-sync - Firebase’s data-sync approach is based on one large chunk of JSON data that allows you to observe and manipulate sub-paths. Deepstream does the same, but breaks the data down into individual units, called records - Deepstream uses a functional permissioning model, allowing you to interface with other systems (data-bases / active directory) for user-management, as opposed to firebase’ configuration based permissioning approach
- awjr 11y agoDoes look very good. Are their any stress tests available? How many "devices" can an instance handle (yes I know this is arbitrary).
- yasserf 11y agoYes, you can find the results at http://deepstream.io/info/performance-single-node-vs-cluster.html http://deepstream.io/info/performance-single-node-vs-cluster..., as well as a link to the test harness if you want to run the tests yourself
- splintercell 11y agoQuestion, how can I expand Deepstream.io? For instance I want it to become an install-able API server for blockchain (like Ethereum). How do I do it? Is there a plugin architecture in place? Would I have to modify the main source code?
- wolframhempel 11y agodeepstream can connect to data-bases, caches and message busses using a simple plugin architecture. Please find a list of available ones at http://deepstream.io/download/ http://deepstream.io/download/ tutorials how to use them at http://deepstream.io/tutorials/connectors-and-deployment.html http://deepstream.io/tutorials/connectors-and-deployment.htm... and help with writing your own at http://deepstream.io/tutorials/writing-storage-cache-connector.html http://deepstream.io/tutorials/writing-storage-cache-connect... http://deepstream.io/tutorials/writing-messaging-connector.html http://deepstream.io/tutorials/writing-messaging-connector.h...
- zubairq 11y agoNice, I'll investigate if Deepstream makes sense for AppShare, another realtime framework: https://github.com/zubairq/AppShare https://github.com/zubairq/AppShare
- _yy 11y agoIf you like this, also take a look at RethinkDB: https://www.rethinkdb.com/ https://www.rethinkdb.com/
- wolframhempel 11y agothey wrote a blogpost about how to use both together: https://rethinkdb.com/blog/deepstream/ https://rethinkdb.com/blog/deepstream/
- liviu 11y agoThey also have a nice RethinkDB storage connector and search provider. [0] - https://github.com/deepstreamIO/deepstream.io-storage-rethinkdb https://github.com/deepstreamIO/deepstream.io-storage-rethin... [1] - https://github.com/deepstreamIO/deepstream.io-provider-search-rethinkdb https://github.com/deepstreamIO/deepstream.io-provider-searc...
- akanet 11y agoInteresting self-hosted alternative to Firebase. I have some questions (as someone seriously evaluating this for use)! * How mature would you say your code is? * Is anyone notable using this in production? * What kinds of bugs have you seen so far, and what do you think the biggest obstacles will be for you in the future? * Is anyone working on this full-time? Do you plan to make money off of this?
- wolframhempel 11y agoGreat questions * How mature would you say your code is? quite :-) * Is anyone notable using this in production? Yes, there are quite a few production use-cases. The latest app going into production using deepstream io for both web and mobile is https://briteback.com/ https://briteback.com/ * What kinds of bugs have you seen so far, and what do you think the biggest obstacles will be for you in the future? Mobile devices with flaky connections are always a challenge. So are many users simultaneously working on very small datasets * Is anyone working on this full-time? Do you plan to make money off of this? Yes. We're a full time company, based in Berlin. We'll eventually add a PaaS offering based on deepstream (deepstreamhub.com), but that is still quite a bit down the line.
- chinathrow 11y ago"Yes. We're a full time company, based in Berlin. " I think you need an imprint according german law then.
- jondubois 11y agoThere is also Meatier https://github.com/mattkrick/meatier https://github.com/mattkrick/meatier which is a more loosely-coupled alternative to Meteor.
- ChrisCinelli 11y agoLooks interesting. What is the maturity of it?
- jondubois 11y agoIt's a new project, but it's made up of very established modules and libraries. It's an aggregation of a bunch of tools which work well together but without locking you in to any of them.
- polskibus 11y agoHow does this compare to EventSource in terms of pubsub functionality? Are messages persisted in any way?
- wolframhempel 11y agoAPI-wise deepstream.io's pub-sub mechanism is more comparable to javascript event emitters. Events in deepstream are a one-off messaging concept, for persistent data, just use data-sync with records
- sebak 11y agoHow does it compare to Meteor? And about the claim: 'exceptionally fast'. What does this mean? Compared to what? Any benchmarks?
- wolframhempel 11y agodeepstream.io (in difference to Meteor) isn't a full framework. The server can easily connect to a wide range of databases, caches or messaging systems. Similarly the client library can be used with React, Angular, Knockout or whatever else your heart desires. In terms of speed, please find performance test results here http://www.deepstream.io/info/performance-overview.html http://www.deepstream.io/info/performance-overview.html as well as a test harness that allows you to replicate them.
- akumpf 11y agoThe "Under the hood" visualization is awesome. I think more projects need to take the time to communicate details like this (without just big blobs of text or links to big piles of source code without a high-level overview). cheers! Edit: link directly to the diagram here -- http://www.deepstream.io/#segment-diagram http://www.deepstream.io/#segment-diagram
- amelius 11y agoPersonally, I think the time spent on such visualizations is better spent on proper documentation, which is often inadequate. And by proper documentation, I don't mean one sentence per function. Also more useful, I think, is to enhance the documentation with comments for feedback or general user discussions. In that case, if the documentation is lacking, at least the user base can educate itself. Just my personal opinion.
- stingraycharles 11y agoA picture tells a thousand words. If anything, most opensource projects fail to do proper marketing. This a great counterexample. Having said that, they of course aren't mutually exclusive, but I can see both sides of the argument.
- amelius 11y ago> If anything, most opensource projects fail to do proper marketing. Can you explain why opensource projects need such visual marketing? Shouldn't we choose the tools which are simply better in a technical sense, rather than the tools which have the flashiest pictures?
- nmcfarl 11y agoNot the parent. But I can't count the number of times I've gone to an open-source project and couldn't even figure out what it did. And I don't consider that to be the lowest bar – any project that wants to get decent adoption should not only tell me what it does, but tell me why I should use it and not it competition. And do both of these things without me having to read the source code. Just this week - well perhaps last - I was at the homepage of some new language – and I couldn't find any of this nor a code example. This might be a technically fantastic language but if they don't communicate that adoption is going to suffer - and when it comes to communicating pictures have a place.
- wiradikusuma 11y agointeresting! does anyone know anything like this but for the jvm?
- yasserf 11y agoThe deepstream client can be written in any language due to it's messaging spec. There actually happens to be someone writing the client in Java now as well! You can find more info here: https://github.com/deepstreamIO/deepstream.io/issues/66 https://github.com/deepstreamIO/deepstream.io/issues/66
- curryhowardiso 11y agoIs there some way to get this working with Safari's terrible audio (don't even get me started on video) streaming api? I know this isn't your fault, and I'm not holding my breath. But the landing page is so beautiful, it felt like there was a chance you had answered my prayers haha!
- wolframhempel 11y agoExcellent question :-) Most important tip for any WebRTC call: Mute your own audio output, that completely ruins it. Other than that, here's a list of tips we've found useful when building the WebRTC feature: http://deepstream.io/tutorials/webrtc-tips.html http://deepstream.io/tutorials/webrtc-tips.html
- curryhowardiso 11y agoI guess I was wondering if it was possible to request an audio stream from a safari user (rather than play one to them). My understanding is that the html5 api calls for this return null (i.e. they are unimplemented). I guess I was hoping that you guys had polyfilled in a (ew) flash plugin which is the only way to do this at the moment, giving google hangouts a monopoly. Maybe I have to write a framework ha! Even better, expose an api for me to build this into via a community package and make wrappers for meteor! (lol)
- treenyc 11y ago@wolframhempel or whoever is making this. Thanks for the making it. However, your webrtc demo doesn't work: both of the links are broken on github: https://deepstream.io/tutorials/webrtc.html https://deepstream.io/tutorials/webrtc.html Can we see some live working demo. Also the example on that page is also not working because adaptor.js is not found, but it says that your browser is not webrtc compatible.
- wolframhempel 11y agoCheers, will look into this
- deleted 11y ago[deleted]
- chinathrow 11y ago"We know you're worried about security, and so are we." Please enable https for your website. Even the fact that I am looking at your site might interest others on or near my network. Thank you.
- facepalm 11y agoCan https hide that you look at a web site? How?
- goldbrick 11y agoIt can. The domain name is not sent in the clear in an HTTPS request. Instead, the connection is opened against the IP address and the domain name is moved to a Host: header which is encrypted.
- esailija 11y agoSNI sends the server name in plain text
- goldbrick 11y agoCorrect (I considered explaining SNI but ultimately didn't bother), and a determined attacker could probably figure out a large number of the sites you visit by the IP as well.
- orthecreedence 11y agoRight, and even still it's pretty trivial to reverse a domain from an IP address in most cases.
- noir_lord 11y agoThere is demo video of the parents organisations product. https://vimeo.com/143728632 https://vimeo.com/143728632 There are times I see stuff done in the browser that makes me feel inadequate, this is one of those times. That interface is incredible.
- moondowner 11y agoIncredible indeed. Is there any info whether Deepstream is used in the stock trading platform and in which scenarios?
- wolframhempel 11y agoIt is used for all communication, including live prices, trade execution and sending historic chart data via RPCs. We are ramping down Hoxton One though to fully concentrate on deepstream.io (and deepstreamhub.com in the future)
- noir_lord 11y agoThat's a shame but damn if that isn't an impressive demo. What does the future look like for Golden Layout?
- wolframhempel 11y agoThat's a challenging question: It's almost exclusively bought by large enterprise customers with a multi step request for quote acquisition flow - so in order to stay economically viable it's either ramping the price up or go for a consultancy model. What do you think?
- noir_lord 11y agoI guess it depends on the cost of acquisition, I've not dealt with large companies for a while but they can be an epic pain to sell to, on the other hand the consultancy model is a hard thing to get right, the better your product is documented and easier to use the less likely you are to get the consulting work which is a kinda perverse incentive (cynically some of the the enterprise stuff I've run across seems to have lots of value and horrible documentation). I guess it comes down to what percentage of the larger companies would want to pay the consulting/support rates, We don't pay support for anything since we largely use stuff that is well documented or very popular largely because we've been burnt on the consulting stuff before (no issue with paying a consultant on a product but when they are less technically competent than the person they are supposed to be helping that stings). Since you are the author that is extremely unlikely to be an issue though. I guess the upside of going the pure consultancy route is in adoption, if a company can adopt your product commercially and then it becomes a core part of their offering they are more likely to pay since it becomes a critical part of their business, that's always a nice place to be - ask Microsoft/IBM :).
- EGreg 11y agoHow is it better than socket.io or http://socketcluster.io/ http://socketcluster.io/
- wolframhempel 11y agoHave a look at this comment https://news.ycombinator.com/item?id=11074511 https://news.ycombinator.com/item?id=11074511 he explains it in a very good way
- Jonovono 11y agoFunny to see so many pubsub things on the home page today. I just finished making a NodeJS CLI app that syncs the song you are playing on spotify to others (https://github.com/jonovono/spotluck https://github.com/jonovono/spotluck) I wanted a pubsub service so I could send the songs around. I wanted to be able to use it in a open source app and ideally client to client. The one I ended up going with was Faye (http://faye.jcoglan.com/ http://faye.jcoglan.com/) but I tried a few others. Will take a look at this! Looks neat.
- known 11y agoI like it
- brudgers 11y agoPrevious comments: https://news.ycombinator.com/item?id=9698858 https://news.ycombinator.com/item?id=9698858
- solipsism 11y agoAm I the only one who thinks we need a better name for this kind of thing than realtime server ?
- Svenskunganka 11y agoReally nice to see Deepstream being on the front page of HN! I'm currently using Deepstream in production for one of our clients in-house applications, so unfortunately I can't share a link. I'd like to give a few words on why I chose Deepstream. TL;DR Deepstream is really good! When I first found Deepstream and scimmed through the docs, I instantly noticed how thoroughly planned and well-thought it was. The features Deepstream offers ended up making us able to drop REST HTTP completely. Normally when you build a back-end, you have all your application logic in one place (authentication, CRUD API, sending emails, etc) but the way RPC's and Providers in Deepstream are implemented, you want and can easily write your back-end as microservices. For example, you build a client that recieves RPC calls for hashing passwords, and if that client starts getting under heavy load, you can instantly spin up another instance of that client and connect to the Deepstream server, register for the RPC and Deepstream server will then distribute the RPC requests evenly among the two instances. Normally when you scale, you scale the entire back-end. Deepstream allow you to scale parts of your back-end really easily (e.g the part that sends emails, the part that hashes passwords or the part that crops & scales images). This allows for some really efficient fine-tuning of your back-end. Another awesome feature is Providers. Providers can do a lot of things, for example transform a standard third-party HTTP REST API (e.g Facebook) into an integrated record list in Deepstream that is synchronized in real-time. Another example is writing an efficient real-time provider that integrates with literally anything - filesystem, databases, logs, hardware utilization, nginx requests, third-party API:s, w/e. The capabilities of Providers are endless. Deepstream is insanely fast - I've worked with Meteor & other real-time solutions before and Deepstream simply outperforms them by a long shot. This is mainly thanks to the Cache Connectors that are available, for example the Redis Cache Connector. Writing Storage- or Cache Connectors is really simple and straight-forward and seamlessly integrates with any kind of storage you want. Heck you can even write a Storage Connector that stores its data in Firebase if you want to be really crazy (and idiotic). Or maybe in a GitHub repository? These are just stupid examples to illustrate how extendable Deepstream really is. I've just scratched the surface of Deepstream, so I really suggest that you go explore it yourself! I see a lot of questions regarding how Deepstream differs from Socket.io, SocketCluster, Meteor, etc. I'll go through each respectively a bit: Socket.io is nothing more than a WebSocket server. Both Deepstream and Socket.io uses Engine.io (can be configured in Deepstream) behind the scenes, but comparing the two would is not really optimal as Deepstream is Socket.io plus a ton of additional features and performance improvements. You can technically re-build Deepstream by using Socket.io if you want. Meteor is quite similar, but it has additional features (like being a full-stack framework) and their implementation of real-time sync is really bad. Meteor is by default locked-down to MongoDB and it's a pain to write support for any other database as they've also locked you down to the fibers coroutines and you need a client-side implementation of it as well for caching purposes. The number of connected clients the two supports is massively in Deepstream's favor, and Meteor doesn't even come close to being as distributable as Deepstream is. Believe it or not, but I used to work with Meteor until I found Deepstream. SocketCluster and Deepstream are very similar, and unfortunately I haven't tried SocketCluster apart from reading the docs but to me, Deepstream is more developer friendly and straightforward and looks to have a better implementation. Since SocketCluster is on the front-page of HN as well, just looking at the live example they have on their website shows that it's not able to keep up. The WebSocket connection is dropping a lot. For those of you who use front-end frameworks like Angular/Aurelia/React/Vue/etc you'll find it super simple to integrate Deepstream's client directly into the bindings of the framework. For example (Aurelia): `<input type="text" value.stream="pets.husky.favoriteToy">` would setup three-way bi-directional databinding to `pets/husky` on the property `favoriteToy`. (I'm currently in the process of doing exactly this for Aurelia)
- jessdawess 11y agoBeyond using a JSON model, what differentiates this so much from other key/value stores (e.g. Redis)?
- wolframhempel 11y agodeepstream is a realtime server for data-sync, pub-sub, request-response and WebRTC. That's a very different thing from Redis (although deepstream can use Redis for caching and message distribution)
- Matthias247 11y agoAlthough I could only look at the surface up to now, this looks really interesting! I always wondered why there are dozens of RPC-only and Pub/Sub-only protocols out there and nothing that is really suitable for record/property synchronization. Because this one important feature in my-domain I have implemented something quite similar to deepstream for my needs, but currently a little bit more limited (pure client-server model, there are no client-hosted RPC methods, no auth, etc.). I chose to make synchronization unidirectional, which avoids the need for merging and conflict resolution and works good for my use cases. There I would have a model where I say the client must call an RPC method on the server, that method would change a property and this would then get synced back to client. I guess that would also be possible with deepstream, but I woul need to configure it in order to let only a single client manipulate the property. Some tech questions: - Are there any ordering guarantees? E.g. between different RPC calls as well as between RPC calls and messages and property updates? Giving those guarantees can increase the difficulty in implementation a lot, but reduces complexity in API design. - Have you also thought on meteor like optimistic updates? I thought it can probably be implemented by putting some additional information in the results of RPC calls - some information about which properties where updates as a side effect of the call and in which version of these properties the updates are applied.
- vortico 11y ago297KB (114KB minified) seems a bit much for a client library, given its feature set.
- amelius 11y agoHow is security dealt with? I tried to find it in the documentation, but without luck so far.
- wolframhempel 11y ago- deepstream supports encryption via https / wss/ ssl - once a connection is established each client has to login. The server then decides whether to accept or reject the connection based on the authentication and connection data - from thereon, every incoming message is authenticated independently all this happens in a "permissionHandler". please find more information here http://deepstream.io/tutorials/authentication.html http://deepstream.io/tutorials/authentication.html http://deepstream.io/tutorials/permissioning.html http://deepstream.io/tutorials/permissioning.html
- morenoh149 11y agohow does this compare to express/koa/hapi/strongloop/socket.io?
- wolframhempel 11y ago- express, koa and hapi are http server frameworks, that's just a very different thing - strongloop offers pub/sub capabilities as well, but is also first and foremost a service for rest apis - socket.io is a transport layer with a low level concept of "rooms" to establish pub/sub behaviour. Deepstream and Socket.io actually both use engine.io for browser communication
- z3t4 11y agoAfter a streak of HN articles loading slow or crashing my old mobile browser, your site loaded instantly! I have a heuristic bias that whether a web-tech company's web site crash my mobile browser or not, reflects the quality of their technology.
- wolframhempel 11y agoHehe, thanks. I'll pass it on to AWS cloudfront
- stemuk 11y agoDo you have any suggestions how to get started with deepstream + polymer (webcomponent library by google www.polymer-project.org/1.0/ )? I have been doing some research on your website and I found toutorials for every major frontend framework except polymer, I would really appreciate a toutorial as a starting point for further projects, deepstream looks very promising to me!