3 ms·
This looks like a naive implementation of adding ordering keys on one side, and waiting until all messages arrive on the other in order, without improving the u
by matt_oriordan 6y ago
This looks like a naive implementation of adding ordering keys on one side, and waiting until all messages arrive on the other in order, without improving the underlying delivery transport to support ordering.
From what I can tell, the publisher is responsible for now maintaining arbitrary ordering IDs and passing this in the publish method, see https://github.com/googleapis/nodejs-pubsub/blob/master/samples/publishOrderedMessage.js#L65-L82 https://github.com/googleapis/nodejs-pubsub/blob/master/samp....
Given the documentation now clearly states "Publishing messages with ordering keys might increase latency", I think that what Google is doing is effectively doing is putting lipstick on their service that does not support ordering and instead offering a message reconstruction capability on the subscriber end.
This approach, whilst it may work a lot of the time, is pretty flawed given one failed published message could cause the entire stream to stall indefinitely. And given Google Pub/Sub does not support idempotency, and does not support exactly-once delivery, they recommend "In general, accommodating more-than-once delivery requires your subscriber to be idempotent when processing messages."
Google, I am afraid whilst it's nice to welcome you to the exactly-once semantic party, I think you may need to go back to the drawing board and bake ordering and idempotency into the transport layer, in the same way Kafka and Ably do so that true ordering is supported.
Disclaimed: I am the co-founder of ably.com, a far edge enterprise messaging solution, with exactly-once semantics and real ordering :)
- hongalex 6y ago> the publisher is responsible for now maintaining arbitrary ordering IDs and passing this in the publish method Ah, thanks for pointing this out. The "ordering ID" bit is actually a remnant of how to publish messages before we had officially ordering key support (and only shows up in Node, but not our other samples https://cloud.google.com/pubsub/docs/publisher#using_ordering_keys https://cloud.google.com/pubsub/docs/publisher#using_orderin...). We'll be fixing this sample shortly. Disclaimer: I work on Cloud Pub/Sub. This explanation is my own.
- matt_oriordan 6y agoAh, OK, glad I could help :) Out of interest then, what is the reasoning behind the warning about additional latency. Is Google Pub/Sub doing some kind of reconstruction on the receiving end to piece together an unordered stream, or has Google Pub/Sub changed how the underlying streaming primitives work to natively support ordering.