5 ms·

Zmqc: netcat for ØMQ

100 points by zacharyvoase 15y ago

zdw 15y ago
This looks like it would be a killer debugging tool. Tried to get licensing information on this, but the "unlicense.org" site appears to not resolve. Part of me wonders if this is a very sly tech joke.
- ComputerGuru 15y ago
  Not at all. Unlicense is just public domain: http://webcache.googleusercontent.com/search?sourceid=chrome&ie=UTF-8&q=cache%3Aunlicense.org http://webcache.googleusercontent.com/search?sourceid=chrome...
sk5t 15y ago
What about encodings - is this "left as an exercise for the console"? I love ZMQ but couldn't imagine using a netcat-like tool without ability to specify inbound or outbound text encoding, or serialization format / protobuf, etc.
- rumcajz 15y ago
  I guess you can contribute serialisisaion formats to the project.
  - technoweenie 15y ago
    ZMQ is specifically content agnostic. It looks like zmqc is too. As zacharyvoase pointed out, you'd just pipe the output of messages to other tools that can handle JSON/MsgPack/Protobufs/whatever.
- zacharyvoase 15y ago
  The UNIX way is, indeed, to leave this as an exercise to the console. You can use a tool like iconv to convert between encodings; the primary draw of zmqc is that everything is on stdin/stdout. For example: zmqc ... | iconv -f iso-8859-1 -t utf-8 If I'd added information on serialisation formats to the program, it would have become very bloated very quickly. JSON, BSON, XML, protobuf, bencode…where does one stop? And besides, since it's stdout, how does one losslessly represent that format without using the format itself?
  - sk5t 15y ago
    Aahhh, I didn't have my head on straight from writing so many non-console programs with ZMQ; you are certainly right about using iconv here.
switz 15y ago
Aside from the functionality, this is how every man page should look.
- mixmastamyk 15y ago
  50% examples, love it. Never understood why man authors are so afraid of them.
  - wglb 15y ago
    Agreed. Man page for rsync is another one done quite well.
chrislloyd 15y ago
There's a similar (but simpler IMHO) project like this already: https://github.com/quackingduck/mp https://github.com/quackingduck/mp. Either tool is great for learning and debugging ØMQ.
_exec 15y ago
I'm having a hard time understanding what the usefulness of 'messaging' is. Could someone point them out or point me to a few real world applications? I've checked the wiki and I'm still puzzled.
- yeureka 15y ago
  I have never used zmq, but while working in finance all the servers I worked on used a comercial messaging package to communicate with each other. It simplifies communication between processes running on the same or different machines by abstracting away network topologies. See it as a simplified UDP.
  - _exec 15y ago
    I'm going to give it a try on one of my client-server side projects to see it in action and learn more. Thanks!
- rwmj 15y ago
  A good way to think about it is "email for applications". Email is a convenient asynchronous way for two people to talk, and (with a bit of formalization) it can be a good way for two or more processes to talk. "Email" here usually means a formal XML message with a standard header and app-defined body. For example, a process can do part of a job, and pass it along to a mailbox. Another process can (perhaps later, or on a different machine) pick up work from this mailbox and finish it. This can hide the network topology (although in practice, that's not really so true) and also makes it easier to manage because you can take down or restart systems without losing work. Messaging systems like Apache / Red Hat's qpid are getting very advanced, for example offering guaranteed latency, guarantees against loss of messages, and multiple ways to set up communications. You can set it up so one process "publishes" data while other processes "subscribe" to that feed of data. You can have 1-N, N-1 and N-M mailboxes. You can have edge-triggered or level-triggered data in some message systems. In the majority of messaging systems, you have a process known as the "broker" which runs on all/most machines, and basically acts as the mailbox, persistent storage and router. It's kind of like sendmail on steroids. Zero MQ is unusual because there is no broker, and it acts more like an advanced sockets library. There are reasons why this is both good and bad compared to ordinary broker systems.
bambambazooka 15y ago
Just wrote a Arch Linux package: https://aur.archlinux.org/packages.php?ID=58283 https://aur.archlinux.org/packages.php?ID=58283
ZephyrP 15y ago
Calumny! I've found salvation in a git repo! I can't believe I didn't think of writing this earlier with all the tcpdump I've done on ØMQ today. A small shrine shall be erected in your honor, Mr. Zacharyvoase

100 points by zacharyvoase 15y ago

zdw 15y ago

This looks like it would be a killer debugging tool. Tried to get licensing information on this, but the "unlicense.org" site appears to not resolve. Part of me wonders if this is a very sly tech joke.

ComputerGuru 15y ago
Not at all. Unlicense is just public domain: http://webcache.googleusercontent.com/search?sourceid=chrome&ie=UTF-8&q=cache%3Aunlicense.org http://webcache.googleusercontent.com/search?sourceid=chrome...

sk5t 15y ago

What about encodings - is this "left as an exercise for the console"? I love ZMQ but couldn't imagine using a netcat-like tool without ability to specify inbound or outbound text encoding, or serialization format / protobuf, etc.

rumcajz 15y ago
I guess you can contribute serialisisaion formats to the project.
- technoweenie 15y ago
  ZMQ is specifically content agnostic. It looks like zmqc is too. As zacharyvoase pointed out, you'd just pipe the output of messages to other tools that can handle JSON/MsgPack/Protobufs/whatever.
zacharyvoase 15y ago
The UNIX way is, indeed, to leave this as an exercise to the console. You can use a tool like iconv to convert between encodings; the primary draw of zmqc is that everything is on stdin/stdout. For example: zmqc ... | iconv -f iso-8859-1 -t utf-8 If I'd added information on serialisation formats to the program, it would have become very bloated very quickly. JSON, BSON, XML, protobuf, bencode…where does one stop? And besides, since it's stdout, how does one losslessly represent that format without using the format itself?
- sk5t 15y ago
  Aahhh, I didn't have my head on straight from writing so many non-console programs with ZMQ; you are certainly right about using iconv here.

switz 15y ago

Aside from the functionality, this is how every man page should look.

mixmastamyk 15y ago
50% examples, love it. Never understood why man authors are so afraid of them.
- wglb 15y ago
  Agreed. Man page for rsync is another one done quite well.

chrislloyd 15y ago

There's a similar (but simpler IMHO) project like this already: https://github.com/quackingduck/mp https://github.com/quackingduck/mp. Either tool is great for learning and debugging ØMQ.

_exec 15y ago

I'm having a hard time understanding what the usefulness of 'messaging' is. Could someone point them out or point me to a few real world applications? I've checked the wiki and I'm still puzzled.

yeureka 15y ago
I have never used zmq, but while working in finance all the servers I worked on used a comercial messaging package to communicate with each other. It simplifies communication between processes running on the same or different machines by abstracting away network topologies. See it as a simplified UDP.
- _exec 15y ago
  I'm going to give it a try on one of my client-server side projects to see it in action and learn more. Thanks!
rwmj 15y ago
A good way to think about it is "email for applications". Email is a convenient asynchronous way for two people to talk, and (with a bit of formalization) it can be a good way for two or more processes to talk. "Email" here usually means a formal XML message with a standard header and app-defined body. For example, a process can do part of a job, and pass it along to a mailbox. Another process can (perhaps later, or on a different machine) pick up work from this mailbox and finish it. This can hide the network topology (although in practice, that's not really so true) and also makes it easier to manage because you can take down or restart systems without losing work. Messaging systems like Apache / Red Hat's qpid are getting very advanced, for example offering guaranteed latency, guarantees against loss of messages, and multiple ways to set up communications. You can set it up so one process "publishes" data while other processes "subscribe" to that feed of data. You can have 1-N, N-1 and N-M mailboxes. You can have edge-triggered or level-triggered data in some message systems. In the majority of messaging systems, you have a process known as the "broker" which runs on all/most machines, and basically acts as the mailbox, persistent storage and router. It's kind of like sendmail on steroids. Zero MQ is unusual because there is no broker, and it acts more like an advanced sockets library. There are reasons why this is both good and bad compared to ordinary broker systems.

bambambazooka 15y ago

Just wrote a Arch Linux package: https://aur.archlinux.org/packages.php?ID=58283 https://aur.archlinux.org/packages.php?ID=58283

ZephyrP 15y ago

Calumny! I've found salvation in a git repo! I can't believe I didn't think of writing this earlier with all the tcpdump I've done on ØMQ today. A small shrine shall be erected in your honor, Mr. Zacharyvoase