3 ms·

Cloud Pub/Sub offers global topics/subscriptions that scale automatically and are completely managed for you (no dealing with your own cluster). With the orderi

by pradn 6y ago

Cloud Pub/Sub offers global topics/subscriptions that scale automatically and are completely managed for you (no dealing with your own cluster). With the ordering keys feature, there's no need to specify the number or ID of the ordering keys ahead of time (as one may need to do for traditional message queue partitions). In general, there's no provisioning ahead of time, so topics/subscriptions will scale as your usage grows. These are a few of the properties Cloud Pub/Sub offers compared to traditional self-hosted message queues. Disclaimer: I work on the Cloud Pub/Sub team, but this explanation is my own.

camgunz 6y ago
Hey thanks for your answer. I'm less concerned about scale or not managing my own cluster, and more about the mechanics of how this is implemented. Is it something like: - User configures an ordering key - Application sends a message w/ ordering key "4" - Pub/Sub gets message w/ ordering key "4" - Pub/Sub buffers "4" until it gets a message > "4" - Application sends a message w/ ordering key "13" - Pub/Sub sends "4" - Pub/Sub buffers "13" until it gets a message > "13" I just thought that the lack of order was in inherent tension with performance because you need a buffer to order things. Am I wrong about that (probably)?
- pradn 6y ago
  It's more like 1) Application sends a message w/ ordering key "abc" (let's call it msg1) 2) Pub/Sub gets msg1. 3) Pub/Sub acknowledges that msg1 was received. Now, you can be secure we have stored this message durably. 4) Pub/Sub sends msg1 to the subscriber ASAP if there's one available. 5) A subscriber client assigned to ordering key "abc" gets msg1. 6) The subscriber client processes msg1 and sends an ack to Pub/Sub. 7) Pub/Sub receives the ack and removes the message from storage. (It's possible to retain acked messages, but let's not talk about that for simplicity.) Another message for the same ordering key "abc" arriving at Pub/Sub any time after step 4 will be delivered to the subscriber only after msg1. To summarize, the timestamp of a successful publish response is the timestamp by which we order messages sent to the client. Pub/Sub may not send messages immediately if the subscriber is too slow to ack the message (perhaps each message takes a while to process) or if the subscriber is down or if there's too many messages in the backlog for the ordering key. In these cases, you can be assured that they're safely stored on disk and will be resent later (after some backoff period.) So, to answer your question directly, we only stall the sending of ordered messages to subscribers if there's a backlog for that ordering key or if the subscriber not fast enough.
  - camgunz 6y ago
    Thank you so much for this answer, really appreciate it.
    - pradn 6y ago
      You're welcome!
- manigandham 6y ago
  A better name would be partioning or shard key. It's not physically ordering the items based on the key, just hashing them into a partition. The physical time they are received by the service is the order they're stored and returned as.

by pradn 6y ago

camgunz 6y ago

Hey thanks for your answer. I'm less concerned about scale or not managing my own cluster, and more about the mechanics of how this is implemented. Is it something like: - User configures an ordering key - Application sends a message w/ ordering key "4" - Pub/Sub gets message w/ ordering key "4" - Pub/Sub buffers "4" until it gets a message > "4" - Application sends a message w/ ordering key "13" - Pub/Sub sends "4" - Pub/Sub buffers "13" until it gets a message > "13" I just thought that the lack of order was in inherent tension with performance because you need a buffer to order things. Am I wrong about that (probably)?

pradn 6y ago
It's more like 1) Application sends a message w/ ordering key "abc" (let's call it msg1) 2) Pub/Sub gets msg1. 3) Pub/Sub acknowledges that msg1 was received. Now, you can be secure we have stored this message durably. 4) Pub/Sub sends msg1 to the subscriber ASAP if there's one available. 5) A subscriber client assigned to ordering key "abc" gets msg1. 6) The subscriber client processes msg1 and sends an ack to Pub/Sub. 7) Pub/Sub receives the ack and removes the message from storage. (It's possible to retain acked messages, but let's not talk about that for simplicity.) Another message for the same ordering key "abc" arriving at Pub/Sub any time after step 4 will be delivered to the subscriber only after msg1. To summarize, the timestamp of a successful publish response is the timestamp by which we order messages sent to the client. Pub/Sub may not send messages immediately if the subscriber is too slow to ack the message (perhaps each message takes a while to process) or if the subscriber is down or if there's too many messages in the backlog for the ordering key. In these cases, you can be assured that they're safely stored on disk and will be resent later (after some backoff period.) So, to answer your question directly, we only stall the sending of ordered messages to subscribers if there's a backlog for that ordering key or if the subscriber not fast enough.
- camgunz 6y ago
  Thank you so much for this answer, really appreciate it.
  - pradn 6y ago
    You're welcome!
manigandham 6y ago
A better name would be partioning or shard key. It's not physically ordering the items based on the key, just hashing them into a partition. The physical time they are received by the service is the order they're stored and returned as.