6 ms·
ØMQ: Mission Accomplished
- deleted 15y ago[deleted]
- h0h0 15y agoWhat HPC products use 0mq? Would it be applicable to large-scale optimization problems, e.g. in context of machine learning?
- rumcajz 15y agohttp://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html http://engineering.twitter.com/2011/08/storm-is-coming-more-...
- wladimir 15y agoLooks very interesting! I cannot find any details about its release though, even though the article talks about September 19. Will it be open source? proprietary?
- lobo_tuerto 15y agoWell, if "CERN is considering ØMQ as a control infrastructure for Large Hadron Collider." and they manage petabytes of data, also "There are trading firms using ØMQ as a secret weapon in low-latency arms race." I think it could be well suited for machine learning.
- jaspersfather 15y agoThat depends more on whether your algorithm is easily distributed. If it is, zeromq will be good for you. I am currently using it to transfer model parameters learned on machine #1 to machine #2 where real time prediction takes place. The predictions are then sent out via 0mq to a robot controller. The great thing of building systems with 0mq is that you just add another component without having to think too much about 'but then I will have to implement a protocol for it and HTTP will probably to slow and the request/reply pattern doesn't really fit there blabla'.
- rdtsc 15y agoIt can be used to stitch together low latency and high throughput distributed systems. Think of it as transport lego blocks, it lets you connect different parts of your system. They support a number of common messaging patterns : request-response, publish-subscribe, etc. But it stops there (and that is the beauty of it). It doesn't prescribe or make assumptions about the types of messages that are sent. Those are just byte blobs as far it it is concerned. You can choose to represent your messages as json, protobuf, xml, raw video frames. So can you use it for HPC? It is hard to tell, sure, it is like asking if hammer can be used to be fix a car. Yeah, it can, in some cases, but it is just a good versatile tool, that can be used for more than that.
- tel 15y agoI used it to build some simple topologies to parallelize collecting statistics over huge data sets. I never got around to doing dynamic programming using a 0mq topology, but I wrote up some fun plans for it.
- espeed 15y agoZeroMQ is amazing. The last few months I have been using it for all kinds of things -- from creating multiprocessing Python apps to replacing an HTTP REST backend with a high-performance ZeroMQ server.
- marquis 15y agoHow does this compare to XMPP? Are there any benefits to learning ZeroMQ for web notifications over using, say Strophe.js?
- rumcajz 15y ago0MQ allows to interconnect components directly. There's no need for a messaging server.
- paperwork 15y agoZeroMQ is very low level, compared to XMPP. ZeroMQ considers itself as part of the networking stack: tcp/udp/0mq (or perhaps just above tcp and udp). ZeroMQ and XMPP are not really competitors.
- rwmj 15y agoZeroMQ is more like an interesting sockets library. It's really not very much like AMQP or XMPP. The "Zero" in ZeroMQ means there is no broker (server) between the publisher and the subscribers. Instead what happens is that each process runs a separate thread which deals exclusively with ZeroMQ messaging. Another effect of having no broker is there is no disk persistence at all, so unless you take your own steps you will lose messages if any component crashes.
- atuladhar 15y agoAre there any hosting services out there that currently support using ZeroMQ?
- reuser 15y agoAny VPS host, or EC2, will let you run ZeroMQ in whatever way is appropriate to your problem. edit: and same for if you run your own server or rent a server, obviously
- beagle3 15y agoHowever, many services won't let you make full use of it; e.g. last year when I tried to get multicast between my EC2 nodes, it discovered it was unsupported. If you're only using ZeroMQ in tcp or inproc mode, it is nice, but in a pub/sub scenario you are wasting a lot of bandwidth and CPU time.
- oconnor0 15y agoCan you explain why you're wasting a lot of bandwidth/CPU time in pub/sub?
- hogu 15y agoif you're not using multicast in pub sub, you're establishing individual tcp connections between nodes, it's not any different than doing N point to point communications, whereas with multicast, the data is only copied when needed, your data will fan out if that makes sense with your network topology I think if you look at the picture it'll be clear http://en.wikipedia.org/wiki/Multicast http://en.wikipedia.org/wiki/Multicast
- swah 15y agoBut when you have to add the second ZeroMQ (say, two Linode VPSs), how do you secure communication between those nodes?
- deleted 15y ago[deleted]
- cmars 15y agoLast time I looked into zmq, it was not recommended for use in public internet services -- not resilient enough against a potential DoS attack or buggy client implementation. Has this changed?
- rumcajz 15y agoIt have got much better. The latest stable version should be pretty much resilient.
- malkia 15y agoI don't think it's supposed to be exposed in this way, rather it's very good for internal wiring between trusted systems.
- nolanw 15y agoIf I recall, the issue with a public 0mq socket was that anyone driving by sending something 0mq didn't like would cause an assertion to fail, so your process would crash. It wasn't so much a "supposed to be exposed" as "expose it and crash".
- malkia 15y agoYes you are right. I guess what I wanted to say, is that 0MQ relies on trusted code and network. It's okay for code to assert, as soon as there is someone to fix it, and it's probably expected. (Crash early, fix early).
- indygreg2 15y agoThrough versions 2.x, there were assert()s peppered throughout the code that could be triggered if malformed data was sent over the wire, making it unsuitable for internet exposure. There was also behavior in durable sockets where messages delivered to an identity that had disappeared would accumulate in memory, effectively manifesting as a memory leak. 3.x removes the asserts so you can't segfault remotely. 3.x also transparently drops messages on the floor destined for unroutable identities. This silent dropping sounds annoying, but you can work around it via a well-designed application protocol. Remember, 0MQ is effectively a transport layer, not a full-blown messaging system.
- zdw 15y agoAnyone know of a good and recent ØMQ/AMPQ/STOMP comparison, specifically with attention paid to the implementations available?
- rumcajz 15y ago0MQ is a product, AMQP and STOMP are protocols. In addition to that both AMQP and STOMP assume classic star topology with a messaging server in the middle, whereas 0MQ allows you to connect your applications directly and create arbitrary topologies.
- wink 15y agoNot saying you're wrong, but I wouldn't phrase it this way. I've only been using 0MQ as a library and AMQP in the form of RabbitMQ - so it can be quite the opposite.
- vph 15y agoHe's talking about 3.1, while publicly 3.0 is still in beta. Does anyone know when 3.1 will come out?
- rumcajz 15y agoCommunity vote taken last weeks was to continue 3.x series of releases by 3.1.x versions.
- bdr 15y agoWow, programming can really be romantic.
- ralphc 15y agoMy favorite - "I recall the hot summer in remote Bulgarian village, almost on the Turkish border, with car broken and no way to get home, where I devised the first version of ØMQ wire protocol. Thanks to the guy who fixed the car! Without him I would have been herding donkeys today. There would be no ØMQ."
- pedoh 15y agoI think 0MQ is brilliant. If this is your first time looking at 0MQ, I'd suggest the following: http://www.zeromq.org/intro:read-the-manual http://www.zeromq.org/intro:read-the-manual Watch the first video by Ian Barber. He's an excellent presenter. The guide ( http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all ) is very lengthy and comprehensive, but the "hello world" example and the "Divide and Conquer" example (fan out jobs, work, fan in results) will get you up and running and start showing you a sliver of the amazing power of 0MQ. They've even got a lot of their examples in many languages: C++ | C# | Clojure | CL | Erlang | F# | Haskell | Haxe | Java | Lua | Node.js | Perl | PHP | Python | Ruby | Scala | Ada | Basic | Go | Objective-C | ooc
- espeed 15y agoYes, these are good as well... ZeroMQ: Modern & Fast Networking Stack, by Ilya Grigorik http://www.igvita.com/2010/09/03/zeromq-modern-fast-networking-stack/ http://www.igvita.com/2010/09/03/zeromq-modern-fast-networki... ZeroMQ an Introduction, by Nicholas Piël http://nichol.as/zeromq-an-introduction http://nichol.as/zeromq-an-introduction Advanced Network Architectures with ZeroMQ, by Zed Shaw http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-advanced-network-architectures-with-zeromq-4896861 http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-adv...
- snewman 15y ago0MQ looks very interesting, but most of the information available seems to be "how to use it". I can't find detailed specifications of its behavior, nor a detailed description of the implementation that I could use to figure that out myself. (Short of wading through the source code.) So it's hard to understand exactly how load balancing works, exactly what happens to in-flight messages on a server crash, exactly what guarantees are provided for message ordering (e.g. in a complex multi-participant scenario), at-least-once vs. at-most-once message delivery, etc. Does anyone know of a good, detailed explanation of how 0MQ is implemented?
- jhawk28 15y agoThe man reference is a good place if you want an API centric view. http://api.zeromq.org/ http://api.zeromq.org/ The guide does show "how to use it", but chapter one does a good overview of the capabilities: http://zguide.zeromq.org/page:all#Chapter-One-Basic-Stuff http://zguide.zeromq.org/page:all#Chapter-One-Basic-Stuff
- moomin 15y agoYou're right, the exact protocol and many other important details are undocumented. As I understand it, the "receiver down" scenario has the rather nice property of neither guaranteeing that messages will be sent or won't be sent.
- jhawk28 15y agohttp://rfc.zeromq.org/spec:15 http://rfc.zeromq.org/spec:15 documents the protocol ZeroMQ uses. The previous version of the protocol (used by 2.1.x) can be found here: http://rfc.zeromq.org/spec:13 http://rfc.zeromq.org/spec:13
- grk 15y agoAs for in-flight messages and server crashes, some explanation is given in the zguide in the part about durable sockets. And you're right that some explanation of implementation is needed. I spent a few hours digging through the code today and it's not easy to read.
- smailq 15y agoCongratz! I've been using 0MQ for few years on distributed db, now it's hard not to architect distributed/threaded apps without 'message passing' design which 0mq is pushing for. Plus the mailing list is pretty helpful and friendly.
- sdafsdasda 15y agoWhy in the hell throw out work if he could have just branched it? H2 was a better product than Hypersonic, and likewise mongrel2 vs mongrel, but neither products were given up. I'm sure there are a multitude of examples why one shouldn't give up a new direction on a product. That much self-backpatting could be called a massage. The guy is congratulating himself publicly and voluminously because he chose to support his existing users. Whether or not what he did was smart, as I assume it was, it is not a good sign that he is so happy admit defeat.