3 ms·

Apologies, but what does this provide that Kafka doesn't already?

by jsjohnst 10y ago

pram 10y ago
I don't get it either, your example is indistinguishable from a normal Kafka producer/consumer ... ?
- mhausenblas 10y ago
  Define 'normal' :) Joking aside: since I've used Kafka for the reference implementation it naturally looks very similar, yes but it's really about the semantics. For example, does your 'normal' Kafka consumer start from the beginning or use latest index, etc.
  - pram 10y ago
    The 'normal' Kafka consumer starts from the last offset
Tepix 10y ago
It uses the file paradigm just like local named pipes. That makes this tech accessible to pretty much every UNIX command.
- mbrock 10y ago
  The example doesn't show any files, just stdout/stdin based interaction. You could make actual named pipes and connect them to the "dnpipes" tool, I guess...
- Animats 10y ago
  Not really. Each message has to be an atomic item, not a stream of bytes, since there can be multiple readers consuming a single queue. The interface specification has problems. "A pull does not remove a message from a dnpipes, it merely delivers its content to the consumer." So if the same consumer pulls the same dnpipe again, does it get the same message? Do messages ever get removed from dnpipes without a reset? Unclear. Does pull block, support async completions, or just return an error when no data is available? Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
  - mhausenblas 10y ago
    > ... if the same consumer pulls the same dnpipe again, does it get the same message No. Each message is delivered at most once. But good point, need to make that clearer! > Do messages ever get removed from dnpipes without a reset. No. Again, something I need to clarify as it seems. > Does pull block, support async completions, or just return an error when no data is available? It blocks. > Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset. I don't follow. Reset empties the underlying queue. In general, since the consumers start to consume not from the beginning of time (as in Kafka's `--from-beginning`) but wherever they happen to be (that is, in Kafka terminology from `latest` index) this shouldn't be a problem. I tried to model as close as possible and as it makes sense after the semantics of (local) named pipes. I might have fudged up here but I'm not 100% clear on where :)
    - Animats 10y ago
      > Do messages ever get removed from dnpipes without a reset. No. Again, something I need to clarify as it seems. So what happens after the system has been running for a while? Why doesn't the dnpipe system fill up with old messages that will never be read again? If new subscribers don't see them, and old subscribers have read them, why are they not removed? It would seem that once every subscriber who subscribed before a message was sent has received that message, the message is dead and can be removed. Why keep the history? Did you really mean that? Also, what happens if one of many subscribers stops making PULL requests? Maybe it's blocked on something, or hung. Do the queues start to build up? Does PUSH eventually block? > Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset. I don't follow. Reset empties the underlying queue. On most queuing systems, when a publisher wants to shut down, they close the channel's sending end or send and EOF message. When all subscribers have read up to the EOF, they close their receiving end. The last messages get processed, and then the subscribers stop. If the only shutdown mechanism is a reset, that can lose messages not yet received. That's OK if the intent is just "kill everything and terminate", but not if you need a clean shutdown. (ROS, the Robot Operating System, has a publish/subscribe system something like this. It's a soft real time system, so old data is discarded and you never want to block a publisher. They chose to lose messages if a subscriber isn't reading often enough. That's appropriate to a robotics use case. It probably wouldn't be for a containerized web backend. See [1].) [1] https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_Delivery_Issues https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_patt...
deleted 10y ago
[deleted]

Apologies, but what does this provide that Kafka doesn't already?

pram 10y ago

I don't get it either, your example is indistinguishable from a normal Kafka producer/consumer ... ?

mhausenblas 10y ago
Define 'normal' :) Joking aside: since I've used Kafka for the reference implementation it naturally looks very similar, yes but it's really about the semantics. For example, does your 'normal' Kafka consumer start from the beginning or use latest index, etc.
- pram 10y ago
  The 'normal' Kafka consumer starts from the last offset

Tepix 10y ago

It uses the file paradigm just like local named pipes. That makes this tech accessible to pretty much every UNIX command.

mbrock 10y ago
The example doesn't show any files, just stdout/stdin based interaction. You could make actual named pipes and connect them to the "dnpipes" tool, I guess...
Animats 10y ago
Not really. Each message has to be an atomic item, not a stream of bytes, since there can be multiple readers consuming a single queue. The interface specification has problems. "A pull does not remove a message from a dnpipes, it merely delivers its content to the consumer." So if the same consumer pulls the same dnpipe again, does it get the same message? Do messages ever get removed from dnpipes without a reset? Unclear. Does pull block, support async completions, or just return an error when no data is available? Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
- mhausenblas 10y ago
  > ... if the same consumer pulls the same dnpipe again, does it get the same message No. Each message is delivered at most once. But good point, need to make that clearer! > Do messages ever get removed from dnpipes without a reset. No. Again, something I need to clarify as it seems. > Does pull block, support async completions, or just return an error when no data is available? It blocks. > Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset. I don't follow. Reset empties the underlying queue. In general, since the consumers start to consume not from the beginning of time (as in Kafka's `--from-beginning`) but wherever they happen to be (that is, in Kafka terminology from `latest` index) this shouldn't be a problem. I tried to model as close as possible and as it makes sense after the semantics of (local) named pipes. I might have fudged up here but I'm not 100% clear on where :)
  - Animats 10y ago
    > Do messages ever get removed from dnpipes without a reset. No. Again, something I need to clarify as it seems. So what happens after the system has been running for a while? Why doesn't the dnpipe system fill up with old messages that will never be read again? If new subscribers don't see them, and old subscribers have read them, why are they not removed? It would seem that once every subscriber who subscribed before a message was sent has received that message, the message is dead and can be removed. Why keep the history? Did you really mean that? Also, what happens if one of many subscribers stops making PULL requests? Maybe it's blocked on something, or hung. Do the queues start to build up? Does PUSH eventually block? > Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset. I don't follow. Reset empties the underlying queue. On most queuing systems, when a publisher wants to shut down, they close the channel's sending end or send and EOF message. When all subscribers have read up to the EOF, they close their receiving end. The last messages get processed, and then the subscribers stop. If the only shutdown mechanism is a reset, that can lose messages not yet received. That's OK if the intent is just "kill everything and terminate", but not if you need a clean shutdown. (ROS, the Robot Operating System, has a publish/subscribe system something like this. It's a soft real time system, so old data is discarded and you never want to block a publisher. They chose to lose messages if a subscriber isn't reading often enough. That's appropriate to a robotics use case. It probably wouldn't be for a containerized web backend. See [1].) [1] https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern#Message_Delivery_Issues https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_patt...

deleted 10y ago

[deleted]