4 ms·

I've played with ZeroMQ on some small projects, and I've been quite impressed by it. Basically, ZeroMQ was designed to send millions of small, mostly-asynchrono

by ekidd 16y ago

I've played with ZeroMQ on some small projects, and I've been quite impressed by it. Basically, ZeroMQ was designed to send millions of small, mostly-asynchronous messages at extremely high speeds. If you feed it messages very rapidly, it sends the first message immediately, and queues the rest in userspace until (IIRC) the network card is ready to transmit another packet. Then it sends all the queued messages as a batch. So you get both high throughput and low latency. There's some built-in support for common topologies (client/server, publish/subscribe, pipeline), and if you're willing to use the undocumented XREQ/XREP socket types, you can build more exotic topologies. Most of the value in ZeroMQ is the actual C++ implementation, so other languages generally just wrap the C++ code. The zeromq API is tiny. I haven't used it on a big production project yet, but I hope to do that soon.

andrewvc 16y ago
xrep/xreq really isn't well documented. I'm writing a learn by example series for ZeroMQ in Ruby, that already includes an XREP example. I'm planning on adding in XREQ this weekend. http://github.com/andrewvc/learn-ruby-zeromq http://github.com/andrewvc/learn-ruby-zeromq
- shotgun 16y ago
  Thanks, man. Very helpful indeed!
sshumaker 16y ago
Well, the idea that simply 'queueing the rest in userspace' is OK illustrates a pretty disconcerting lack of understanding in writing networking code. This is not an acceptable solution in the real world. I want to correct that assumption right now - it's one of those places that higher level networking libraries often fall down - because simply 'fire and forget' is a leaky abstraction. Let's say you're streaming a large file over TCP to a client. Generally, disk reads are far faster than you can send data over the internet. If you naively keep sending data, you'll quickly fill up the OS buffers - and begin to consume memory as it's queued on the client side. In an extreme case, if you're sending a multi-GB file, you'll probably end up with most of the file sitting in RAM (assuming you don't run of ram and start thrashing). And for what? You really only need to load additional data in from the file as fast as you can send it in the socket. In a real-life case, streaming multiple files to multiple clients, you'll quickly run out of RAM with even moderate size files. Some high level libraries provide an event to let you know that the OS buffers have been emptied out. For example, Node.js has a drain event. But this has to be handled explicitly, and isn't really highlighted in the docs. But if you write your code naively, you will definitely run into problems. While it's great that many of these high level networking libraries have tried to abstract away many of the complexities of network programming, they're also a trap for the unwary. Just remember that they aren't a silver bullet that let you handle sending data over the internet as a simple function call.
- ekidd 16y ago
  Have you checked out the ZeroMQ documentation on flow control and data rate limiting? I can't speak for anyone else's use case, but there's a nice selection of blocking primitives and data rate limits that have proven sufficient for the smaller programs I've written. It's hard for me to tell if your criticisms are based on real-world experience with the library, a review of the available documentation, or just my description of a single feature intended to reduce the cost traversing the userspace/kernel barrier. I'm definitely interested in real-world experience with ZeroMQ and any problems you've encountered. We may begin some larger-scale testing soon, and would love to know about any problems.
  - ehsanul 16y ago
    Instead of rate-limiting, there are two other options that solve the GP's concerns: the high water mark which controls the maximum queue length and the swap size, which saves messages to disk when the queue is full. http://api.zeromq.org/zmq_setsockopt.html http://api.zeromq.org/zmq_setsockopt.html
- nkurz 16y ago
  If you are streaming a large file over TCP, ZeroMQ, with its emphasis on small messages sent and received apparently atomically, would be a stunningly poor choice. It's also wholly unsuitable as a floor wax or a dessert topping. :) Does your criticism also apply to its intended use case? At a glance, it doesn't strike me as an unreasonable design choice, but I haven't thought about it deeply.
  - sandGorgon 16y ago
    what would be the best way of sending large files server-to-server ? this is a question that has been puzzling me. Is there like a library or a protocol that is tuned towards transferring multi-gb files ?
    - nkurz 16y ago
      Are you intending to have many of these transfers happening in parallel, or are you sending a single file and hoping to saturate your link? Do you have full control of both ends, or only the server? Are you streaming or is the whole file always transferred? Do you need encryption? For simultaneous large files with no special needs, my instinct would be to just choose any lightweight HTTP server that supports sendfile(). Basically, pass it off to the kernel, and let it handle it. But maybe there are more subtle keys to good large file performance: http://www.psc.edu/networking/projects/tcptune/ http://www.psc.edu/networking/projects/tcptune/ For cases where you are controlling both ends, you might find useful suggestions here: http://serverfault.com/questions/111813/what-is-the-best-way-to-transfer-a-single-large-file-over-a-high-speed-high-late http://serverfault.com/questions/111813/what-is-the-best-way... In particular, Tsunami and uftp look interesting.

by ekidd 16y ago

andrewvc 16y ago

xrep/xreq really isn't well documented. I'm writing a learn by example series for ZeroMQ in Ruby, that already includes an XREP example. I'm planning on adding in XREQ this weekend. http://github.com/andrewvc/learn-ruby-zeromq http://github.com/andrewvc/learn-ruby-zeromq

shotgun 16y ago
Thanks, man. Very helpful indeed!

sshumaker 16y ago

Well, the idea that simply 'queueing the rest in userspace' is OK illustrates a pretty disconcerting lack of understanding in writing networking code. This is not an acceptable solution in the real world. I want to correct that assumption right now - it's one of those places that higher level networking libraries often fall down - because simply 'fire and forget' is a leaky abstraction. Let's say you're streaming a large file over TCP to a client. Generally, disk reads are far faster than you can send data over the internet. If you naively keep sending data, you'll quickly fill up the OS buffers - and begin to consume memory as it's queued on the client side. In an extreme case, if you're sending a multi-GB file, you'll probably end up with most of the file sitting in RAM (assuming you don't run of ram and start thrashing). And for what? You really only need to load additional data in from the file as fast as you can send it in the socket. In a real-life case, streaming multiple files to multiple clients, you'll quickly run out of RAM with even moderate size files. Some high level libraries provide an event to let you know that the OS buffers have been emptied out. For example, Node.js has a drain event. But this has to be handled explicitly, and isn't really highlighted in the docs. But if you write your code naively, you will definitely run into problems. While it's great that many of these high level networking libraries have tried to abstract away many of the complexities of network programming, they're also a trap for the unwary. Just remember that they aren't a silver bullet that let you handle sending data over the internet as a simple function call.

ekidd 16y ago
Have you checked out the ZeroMQ documentation on flow control and data rate limiting? I can't speak for anyone else's use case, but there's a nice selection of blocking primitives and data rate limits that have proven sufficient for the smaller programs I've written. It's hard for me to tell if your criticisms are based on real-world experience with the library, a review of the available documentation, or just my description of a single feature intended to reduce the cost traversing the userspace/kernel barrier. I'm definitely interested in real-world experience with ZeroMQ and any problems you've encountered. We may begin some larger-scale testing soon, and would love to know about any problems.
- ehsanul 16y ago
  Instead of rate-limiting, there are two other options that solve the GP's concerns: the high water mark which controls the maximum queue length and the swap size, which saves messages to disk when the queue is full. http://api.zeromq.org/zmq_setsockopt.html http://api.zeromq.org/zmq_setsockopt.html
nkurz 16y ago
If you are streaming a large file over TCP, ZeroMQ, with its emphasis on small messages sent and received apparently atomically, would be a stunningly poor choice. It's also wholly unsuitable as a floor wax or a dessert topping. :) Does your criticism also apply to its intended use case? At a glance, it doesn't strike me as an unreasonable design choice, but I haven't thought about it deeply.
- sandGorgon 16y ago
  what would be the best way of sending large files server-to-server ? this is a question that has been puzzling me. Is there like a library or a protocol that is tuned towards transferring multi-gb files ?
  - nkurz 16y ago
    Are you intending to have many of these transfers happening in parallel, or are you sending a single file and hoping to saturate your link? Do you have full control of both ends, or only the server? Are you streaming or is the whole file always transferred? Do you need encryption? For simultaneous large files with no special needs, my instinct would be to just choose any lightweight HTTP server that supports sendfile(). Basically, pass it off to the kernel, and let it handle it. But maybe there are more subtle keys to good large file performance: http://www.psc.edu/networking/projects/tcptune/ http://www.psc.edu/networking/projects/tcptune/ For cases where you are controlling both ends, you might find useful suggestions here: http://serverfault.com/questions/111813/what-is-the-best-way-to-transfer-a-single-large-file-over-a-high-speed-high-late http://serverfault.com/questions/111813/what-is-the-best-way... In particular, Tsunami and uftp look interesting.