5 ms·
Distributed Systems with ZeroMQ and gevent
- plainOldText 15y agoFor anyone interested in this subject, the presentation has clean, easy to understand drawings which make the reading so much pleasant. I read all 154 pages mainly because of that, and I must admit it gave me a cleaner picture of how ZeroMQ can be integrated in distributed systems. Thanks for sharing
- j2labs 15y agoPeople interested in these slides might like to check out the Brubeck framework. It can use gevent for concurrency and ZeroMQ to communicate with Mongrel2, it's webserver. http://brubeck.io http://brubeck.io
- espeed 15y agoSo Mongrel2 sits in front of Brubeck and proxies requests back? I've been playing with ZeroMQ quite a bit the last few weeks because I needed a fast way to pump data from a Python app into the Neo4j Batch Loader, which runs on the JVM. I wrote a lightweight ZeroMQ server in Jython called Lightsocket and a Python client that (with multiple clients) was pushing about 10,000 inserts per second into Neo4j on my dev system. Jython was the bottleneck, and I could have rewritten it in Java or Scala, but the data was already loaded. However, I went looking around to see if there was a ZeroMQ-based container or if anyone had modified Tomcat or Jetty to run as a ZeroMQ server, but I couldn't find much on it. Does anyone know of a ZeroMQ server/container for the JVM?
- j2labs 15y agoCorrect. Mongrel2 sends request messages to Brubeck which processes the request and responds with a complete HTTP response.
- getsat 15y agoReally great presentation and really great code examples! Thanks for uploading this.
- anotherjesse 15y agoI was just thinking of inviting jeff to give this talk at a openstack hacking event and I see slide 10 he references us :)
- redouane 15y agonice,i was just looking for something like this. thank you.
- sneak 15y agoHaven't read the preso yet but I just wanted to chime in and say that I love ZeroMQ - it's one of those simple libs that are so cool you want to run out and build something with it the moment you see a four line example. I've used it to build some cool stuff, and can't wait to use it more often.
- schlomie 15y ago> Ø Faster than TCP, for clustered products and supercomputing. Not quite following this statement, what makes it faster?
- xtacy 15y agoI think it means, it's faster than managing your own TCP connections for cluster applications. ZeroMQ's website itself makes no claim that they're faster than TCP: http://www.zeromq.org/area:faq http://www.zeromq.org/area:faq, under: The graph in the test results shows that ØMQ is slower than TCP/IP. What's the point then?
- schlomie 15y agoAh thanks, it must be something like that, though later on in the FAQ there they also say: > ØMQ gets you more throughput than TCP has using intelligent batching algorithms. yes, intelligent, sounds like a good idea. ;-) EDIT: ok here's a little more detail as to what they mean by that in the next FAQ entry: > ØMQ batches messages in opportunistic manner. Rather than waiting for a predefined number of messages and/or predefined time interval, it sends all the messages available at the moment in one go. Imagine the network interface card is busy sending data. Once it is ready to send more data it asks ØMQ for new messages. ØMQ sends all the messages available at the moment. Does it harm the latency of the first message in the batch? No. The message won't be sent earlier anyway because the networking card was busy. On the contrary, latency of subsequent messages will be improved because sending single batch to the card is faster then sending lot of small messages. On the other hand, if network card isn't busy, the message is sent straight away without waiting for following messages. Thus it'll have the best possible latency. EDIT: This is interesting too for super low latency, Myricom actually makes such a library (http://www.myricom.com/kb/index.php?title=DBL http://www.myricom.com/kb/index.php?title=DBL). > We are working on delivering ØMQ over alternative networking stacks, thus having advantage of features like kernel bypass, avoiding TCP/IP overhead, using high-performance networking hardware etc. That way we can get the latency as low as 10 microseconds.
- iam 15y agoIs there by any chance any use cases of why would someone use one concurrency model (e.g. ZeroMQ) over another (gevent)?
- rabidsnail 15y agohttp://svn.red-bean.com/bob/greenlet/trunk/greenlet.c http://svn.red-bean.com/bob/greenlet/trunk/greenlet.c There are real continuations for python!? Where has this been all my life?
- jacques_chester 15y agoThere's literature for this kind of architecture under the heading "Staged Event-Driven Architectures" or SEDA. http://en.wikipedia.org/wiki/Staged_event-driven_architecture http://en.wikipedia.org/wiki/Staged_event-driven_architectur...
- StavrosK 15y agoCan someone explain how this can be beneficial to more or less typical web apps? I don't see where the extra concurrency could be used... Alternatively, what sort of use cases is this good for?