5 ms·

A distributed, tag-based pub-sub service for modern web applications

67 points by sdomino 11y ago

zwily 11y ago
How is this distributed?
- tylerflint 11y ago
  When a node is brought online, it uses multicast discovery to learn about other nodes. You can see the gist here: https://github.com/nanopack/mist/blob/master/main.go#L68-L76 https://github.com/nanopack/mist/blob/master/main.go#L68-L76 Propagating events is really simple: When a client subscribes to a particular node, the node will replicate the subscription to all nodes in the cluster, and relay any messages from the 'proxied' subscriptions back to the client.
  - tylerflint 11y ago
    Also, if your network can't natively support multicast, you might want to check out red (https://github.com/nanopack/red https://github.com/nanopack/red) "A fast, in-kernel, ad-hoc point-to-point vxlan network."
tylerflint 11y ago
Also to clarify (and yes I also work at Pagoda Box on the Nanobox project), this service was initially intended to provide push capabilities for rails and other web frameworks that can't natively provide a push socket layer. It has successfully filled that niche and has expanded to fill other needs as well.
sanderson1 11y ago
Is this what what Pagoda Box uses for their in-dashboard live updates or is it an off-shoot of that project?
- deleted 11y ago
  [deleted]
sdomino 11y ago
As a point of clarification (I work at Pagoda Box/Nanobox). Pagoda Box uses an older suite of similar tools (https://github.com/stormpack-core https://github.com/stormpack-core) mostly written in Erlang. This newer project (written in Golang) is part of a group of tools (http://nanopack.io/ http://nanopack.io/) being written for https://nanobox.io/ https://nanobox.io/
- polskibus 11y ago
  I'm curious why did you drop Erlang in favor of Go?
  - tylerflint 11y ago
    The actual reason is much less exciting than you might have hoped and actually has nothing to do with erlang vs golang... The erlang version has been running successfully for about 2 years and we haven't had any issues whatsoever (excepting a wrestling match with mnesia early on). We are erlang/elixir advocates and have used the erlang vm successfully on highly critical multi-million-concurrency services for over 6 years. When we set out to build nanobox desktop (https://desktop.nanobox.io https://desktop.nanobox.io) our vision was to provide a single pre-compiled executable that could be run without any configuration. The design required a push layer and needed the same functionality that the erlang project was already providing. It wasn't feasible to package the erlang application into the nanobox binary, so we emulated the original project into a consumable golang package. As time went on we were porting more and more of the features into the golang port until all that was lacking was distribution and authentication. At that point we made the decision to consolidate our efforts into a single project, that could be a standalone service or composed within a golang binary. I regret to inform you that there isn't a mass exodus within our company to ditch erlang for golang. While that certainly would make for a fun thread, in this case it was simply a matter of fit and effort consolidation.
tylertreat 11y ago
Curious how this compares to gnatsd: https://github.com/nats-io/gnatsd https://github.com/nats-io/gnatsd
- lobster_johnson 11y ago
  Two things come to mind: gnats uses wildcard topics, very useful to create filtered subscriptions. And gnats is pretty good at dealing with backpressure; the stated aim of Mist is to silently drop messages if clients can't keep up. I would also say that gnats is mature and has been proven in production for several years, which might not be the case with this project.
rar_ram 11y ago
> Messages are not guaranteed to be delivered, if the client is running behind on processing messages, newer messages could be dropped. > Messages are not stored until they are delivered, if no client is available to receive the message, then it is dropped without being sent anywhere. Not sure in which use cases can I use this ? Doesn't this defeat the purpose of having a message queue.
- tylerflint 11y ago
  So to the first question, there's not a transactional layer to guarantee the client handled the message. Once a message is sent to the client, it is assumed the client has a mechanism for queuing messages if it can't keep up. As to the second, the caveat is basically saying that if a message is broadcast and nobody cares about the message (ie: there aren't any subscriptions) then mist isn't going to store the message anywhere, it just gets dropped.
  - rar_ram 11y ago
    Err, I get that! but I am still wondering on where would I use this ? If I dont need transactional layer or buffering, I could use event listener and dispatcher pattern. Anything more than that, a message queue like ZMQ works. Where does this fit in ?
    - tylerflint 11y ago
      Oh I see. Sorry for the confusion. The initial need was for a rails app to send data model updates to dashboard clients. Later, as our infrastructure became more micro-service-oriented, mist has been used as a place for the dashboard javascript client to subscribe to a centralized event queue. Each micro-service will publish messages to the mist cluster, and the dashboard can respond accordingly.
- sdomino 11y ago
  Mist isn't intended to be a message queue, its simply a message passer. Similar to the old question, "if a tree falls in the woods and no one is there to hear it does it make a sound?" Yes; similarly Mist will send a message whether or not anyone is listening, and will keep sending them whether or not the client is prepared to handle them.
ShaneWilton 11y ago
This looks like a great project! I'm curious about the following caveat from the readme, however: "Data flowing through mist is NOT touched in anyway. It is not verified in any way, but it MUST NOT contain a newline character as this will break the mist protocol." I haven't read the code yet, but my gut reaction is that this suggests it would be possible to inject commands using malicious user input. Given the caveat, this may be allowed for by your threat model, but it may be worth some effort to mitigate. EDIT: To clarify, this may not be the case at all, in which case I'd be curious to hear why this restriction is in place.
- tylerflint 11y ago
  This is referring to messaging being sent through mist. Mist doesn't care what the data looks like, and won't transform it. Currently the public-facing websocket client is not allowed to publish messages until this is mitigated, as you mentioned. Any feedback here would be appreciated.
  - ShaneWilton 11y ago
    Cool, thanks for the response! If I get a chance later, I'll look through the code and see if anything jumps out at me.
  - zrail 11y ago
    Netstrings are for this exact use case: http://cr.yp.to/proto/netstrings.txt http://cr.yp.to/proto/netstrings.txt
    - viraptor 11y ago
      Actually this proposal is quite bad compared to simple "<4 bytes size><size bytes contents>". If you use netstrings recursively like djb recommends, you will end up copying / reallocing the strings many times. Since your top-level length depends on the lower level length, you cannot send it without actually calculating the whole contents. Asn1 DER has the same issue because of the weird size encoding. RIFF is a bit better https://en.wikipedia.org/wiki/Resource_Interchange_File_Format https://en.wikipedia.org/wiki/Resource_Interchange_File_Form... with strict length,tag,contents format.
nodesocket 11y ago
A lot of open-source projects by Nanobox look awesome, but most don't have any documentation or demos :-(. I'm most interested in: [Hooky D](https://github.com/nanopack/hookyd) - Remote hook execution layer [Butter](https://github.com/nanopack/butter) - Git based deployment system [Shuttle](https://github.com/nanopack/shuttle) - Transparent TCP proxy for web
- sdomino 11y ago
  We're working feverishly to get documentation and demos available and would LOVE any help or contributions from the community ;).
- tylerflint 11y ago
  Yeah we hear ya. The nanopack cloud-initiative was just announced last friday (https://blog.nanobox.io/nanopack-a-new-vision-for-automated-infrastructure/ https://blog.nanobox.io/nanopack-a-new-vision-for-automated-...). We're getting there slowly but surely!
cbsmith 11y ago
It's like PubSubHubbub, except with JSON instead of XML, and bypassing that whole HTTP protocol that was never going to catch on anyway...
mappu 11y ago
I need one of these at $DAYJOB as of yesterday, so thanks! Any chance of basing it on `igm/sockjs-go` or similar instead of raw websockets?
- jkarneges 11y ago
  An alternative is to use a proxy. For example Pushpin (http://pushpin.org http://pushpin.org) can convert SockJS to native WebSockets. There might be others too.
foo42 11y ago
Struggling to see when I'd use this over, say, rabbitmq. Feels a bit like "not invented here" syndrome, though glad to be proven wrong. Also curious that this replaces previous tooling written in erlang, be interested to here what prompted the move to golang, I would think a message broker was right in erlang's wheelhouse
- lobster_johnson 11y ago
  Not the OP, but this looks like it's designed to be lossy, and prioritize availability over deliverability. If clients are unable to keep up with message volume, messages will be dropped. That makes is okay for things like realtime dashboards (which this is apparently designed for) and other rapid events where transactionality is not required. But it's obviously not suited for job-like scheduling like data processing or email delivery. Nor as an event bus for inter-service coordination ("on event X, do Y").
  - tylerflint 11y ago
    That's a pretty accurate assessment. Mist could be conceptualized as a self-hosted replacement for pusher (https://pusher.com/ https://pusher.com/). It's designed specifically to address building realtime web apps.
  - deleted 11y ago
    [deleted]
- tylerflint 11y ago
  Everything that mist does is "right in erlang's wheelhouse". Erlang is awesome and the actual reason we moved golang had nothing to do with erlang vs golang. I responded to that in detail at another spot on this thread (currently above, but who knows if that is still the case).

67 points by sdomino 11y ago

zwily 11y ago

How is this distributed?

tylerflint 11y ago
When a node is brought online, it uses multicast discovery to learn about other nodes. You can see the gist here: https://github.com/nanopack/mist/blob/master/main.go#L68-L76 https://github.com/nanopack/mist/blob/master/main.go#L68-L76 Propagating events is really simple: When a client subscribes to a particular node, the node will replicate the subscription to all nodes in the cluster, and relay any messages from the 'proxied' subscriptions back to the client.
- tylerflint 11y ago
  Also, if your network can't natively support multicast, you might want to check out red (https://github.com/nanopack/red https://github.com/nanopack/red) "A fast, in-kernel, ad-hoc point-to-point vxlan network."

tylerflint 11y ago

Also to clarify (and yes I also work at Pagoda Box on the Nanobox project), this service was initially intended to provide push capabilities for rails and other web frameworks that can't natively provide a push socket layer. It has successfully filled that niche and has expanded to fill other needs as well.

sanderson1 11y ago

Is this what what Pagoda Box uses for their in-dashboard live updates or is it an off-shoot of that project?

deleted 11y ago
[deleted]

sdomino 11y ago

As a point of clarification (I work at Pagoda Box/Nanobox). Pagoda Box uses an older suite of similar tools (https://github.com/stormpack-core https://github.com/stormpack-core) mostly written in Erlang. This newer project (written in Golang) is part of a group of tools (http://nanopack.io/ http://nanopack.io/) being written for https://nanobox.io/ https://nanobox.io/

polskibus 11y ago
I'm curious why did you drop Erlang in favor of Go?
- tylerflint 11y ago
  The actual reason is much less exciting than you might have hoped and actually has nothing to do with erlang vs golang... The erlang version has been running successfully for about 2 years and we haven't had any issues whatsoever (excepting a wrestling match with mnesia early on). We are erlang/elixir advocates and have used the erlang vm successfully on highly critical multi-million-concurrency services for over 6 years. When we set out to build nanobox desktop (https://desktop.nanobox.io https://desktop.nanobox.io) our vision was to provide a single pre-compiled executable that could be run without any configuration. The design required a push layer and needed the same functionality that the erlang project was already providing. It wasn't feasible to package the erlang application into the nanobox binary, so we emulated the original project into a consumable golang package. As time went on we were porting more and more of the features into the golang port until all that was lacking was distribution and authentication. At that point we made the decision to consolidate our efforts into a single project, that could be a standalone service or composed within a golang binary. I regret to inform you that there isn't a mass exodus within our company to ditch erlang for golang. While that certainly would make for a fun thread, in this case it was simply a matter of fit and effort consolidation.

tylertreat 11y ago

Curious how this compares to gnatsd: https://github.com/nats-io/gnatsd https://github.com/nats-io/gnatsd

lobster_johnson 11y ago
Two things come to mind: gnats uses wildcard topics, very useful to create filtered subscriptions. And gnats is pretty good at dealing with backpressure; the stated aim of Mist is to silently drop messages if clients can't keep up. I would also say that gnats is mature and has been proven in production for several years, which might not be the case with this project.

rar_ram 11y ago

> Messages are not guaranteed to be delivered, if the client is running behind on processing messages, newer messages could be dropped. > Messages are not stored until they are delivered, if no client is available to receive the message, then it is dropped without being sent anywhere. Not sure in which use cases can I use this ? Doesn't this defeat the purpose of having a message queue.

tylerflint 11y ago
So to the first question, there's not a transactional layer to guarantee the client handled the message. Once a message is sent to the client, it is assumed the client has a mechanism for queuing messages if it can't keep up. As to the second, the caveat is basically saying that if a message is broadcast and nobody cares about the message (ie: there aren't any subscriptions) then mist isn't going to store the message anywhere, it just gets dropped.
- rar_ram 11y ago
  Err, I get that! but I am still wondering on where would I use this ? If I dont need transactional layer or buffering, I could use event listener and dispatcher pattern. Anything more than that, a message queue like ZMQ works. Where does this fit in ?
  - tylerflint 11y ago
    Oh I see. Sorry for the confusion. The initial need was for a rails app to send data model updates to dashboard clients. Later, as our infrastructure became more micro-service-oriented, mist has been used as a place for the dashboard javascript client to subscribe to a centralized event queue. Each micro-service will publish messages to the mist cluster, and the dashboard can respond accordingly.
sdomino 11y ago
Mist isn't intended to be a message queue, its simply a message passer. Similar to the old question, "if a tree falls in the woods and no one is there to hear it does it make a sound?" Yes; similarly Mist will send a message whether or not anyone is listening, and will keep sending them whether or not the client is prepared to handle them.

ShaneWilton 11y ago

This looks like a great project! I'm curious about the following caveat from the readme, however: "Data flowing through mist is NOT touched in anyway. It is not verified in any way, but it MUST NOT contain a newline character as this will break the mist protocol." I haven't read the code yet, but my gut reaction is that this suggests it would be possible to inject commands using malicious user input. Given the caveat, this may be allowed for by your threat model, but it may be worth some effort to mitigate. EDIT: To clarify, this may not be the case at all, in which case I'd be curious to hear why this restriction is in place.

tylerflint 11y ago
This is referring to messaging being sent through mist. Mist doesn't care what the data looks like, and won't transform it. Currently the public-facing websocket client is not allowed to publish messages until this is mitigated, as you mentioned. Any feedback here would be appreciated.
- ShaneWilton 11y ago
  Cool, thanks for the response! If I get a chance later, I'll look through the code and see if anything jumps out at me.
- zrail 11y ago
  Netstrings are for this exact use case: http://cr.yp.to/proto/netstrings.txt http://cr.yp.to/proto/netstrings.txt
  - viraptor 11y ago
    Actually this proposal is quite bad compared to simple "<4 bytes size><size bytes contents>". If you use netstrings recursively like djb recommends, you will end up copying / reallocing the strings many times. Since your top-level length depends on the lower level length, you cannot send it without actually calculating the whole contents. Asn1 DER has the same issue because of the weird size encoding. RIFF is a bit better https://en.wikipedia.org/wiki/Resource_Interchange_File_Format https://en.wikipedia.org/wiki/Resource_Interchange_File_Form... with strict length,tag,contents format.

nodesocket 11y ago

A lot of open-source projects by Nanobox look awesome, but most don't have any documentation or demos :-(. I'm most interested in: [Hooky D](https://github.com/nanopack/hookyd) - Remote hook execution layer [Butter](https://github.com/nanopack/butter) - Git based deployment system [Shuttle](https://github.com/nanopack/shuttle) - Transparent TCP proxy for web

sdomino 11y ago
We're working feverishly to get documentation and demos available and would LOVE any help or contributions from the community ;).
tylerflint 11y ago
Yeah we hear ya. The nanopack cloud-initiative was just announced last friday (https://blog.nanobox.io/nanopack-a-new-vision-for-automated-infrastructure/ https://blog.nanobox.io/nanopack-a-new-vision-for-automated-...). We're getting there slowly but surely!

cbsmith 11y ago

It's like PubSubHubbub, except with JSON instead of XML, and bypassing that whole HTTP protocol that was never going to catch on anyway...

mappu 11y ago

I need one of these at $DAYJOB as of yesterday, so thanks! Any chance of basing it on `igm/sockjs-go` or similar instead of raw websockets?

jkarneges 11y ago
An alternative is to use a proxy. For example Pushpin (http://pushpin.org http://pushpin.org) can convert SockJS to native WebSockets. There might be others too.

foo42 11y ago

Struggling to see when I'd use this over, say, rabbitmq. Feels a bit like "not invented here" syndrome, though glad to be proven wrong. Also curious that this replaces previous tooling written in erlang, be interested to here what prompted the move to golang, I would think a message broker was right in erlang's wheelhouse

lobster_johnson 11y ago
Not the OP, but this looks like it's designed to be lossy, and prioritize availability over deliverability. If clients are unable to keep up with message volume, messages will be dropped. That makes is okay for things like realtime dashboards (which this is apparently designed for) and other rapid events where transactionality is not required. But it's obviously not suited for job-like scheduling like data processing or email delivery. Nor as an event bus for inter-service coordination ("on event X, do Y").
- tylerflint 11y ago
  That's a pretty accurate assessment. Mist could be conceptualized as a self-hosted replacement for pusher (https://pusher.com/ https://pusher.com/). It's designed specifically to address building realtime web apps.
- deleted 11y ago
  [deleted]
tylerflint 11y ago
Everything that mist does is "right in erlang's wheelhouse". Erlang is awesome and the actual reason we moved golang had nothing to do with erlang vs golang. I responded to that in detail at another spot on this thread (currently above, but who knows if that is still the case).