4 ms·
Is push/pop really not available as a Cloud Service?
We are looking for a message queuing system that ensures the sequential processing of messages based on their row numbers.
This seems like a foundational part of computing - but we seem unable to find it.
Here's what we mean.
PUSH/POP: Adding and removing items from a list is common in code (push/pop). These methods typically ensure every item is processed once-and-only-once ("exactly once processing") and in-order (ordinality).
Finding this as a cloud service has seemed elusive.
GOOGLE: We were told by Google reps that Pub/Sub can guarantee items from a list are emitted in order and once-and-only-once but cannot guarantee that the items you added to the list were added in the order you expected. (The internet can be slow - or drop an item - and the order can be lost.)
APACHE: Kafka, as we were also told, cannot guarantee exactly once processing with ordinality.
The result is that both seemed to be partially implemented push/pop capability.
Does any such Cloud Message Queuing service exist? Or is this simply something we will have to write ourselves?
This seems so basic and fundamental to queue processing that we are surprised if it doesn't exist.
----
Example Background
Let’s say I have five messages labeled with row numbers one through five, but they may arrive at the message queue out of order. For example, a message with row number 3 might arrive before the one with row number 2. The message queue should forward the messages to the service in ascending order based on their row numbers. Additionally, if there is a delay or error causing a row number to be missing, we want the queue to pause processing until the missing message arrives before sending the subsequent messages to the service. Is there really no suitable message queuing system for this requirement?
- doormatt 3y agoAre you not looking for Message ordering/FIFO Queues with SQS? https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-message-order.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
- HealthLab 3y agoThanks for the reply ... Perhaps ... there is one thing we did not see ... Is there assurance in SQS to ensure that what is first processed is actually what is first? (i.e. process these in the order of the row number - not the order you receive them.) Meaning - typically with outside systems we have a list of messages ... ("we need these 1000 messages processed in order") ... ... with push/pop - we could typically ensure the first message is the first message ... ... For SQS - how do we ever ensure the first message (and every subsequent message) is the actual order we desire? (Not the order the internet arbitrarily delivered them)
- doormatt 3y agohttps://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-message-order.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS... >The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it. >In addition, FIFO queues support message groups that allow multiple ordered message groups within a single queue. There is no quota to the number of message groups within a FIFO queue.
- programd 3y ago> Additionally, if there is a delay or error causing a row number to be missing, we want the queue to pause processing until the missing message arrives before sending the subsequent messages to the service There's your problem. On the real Internet after some surprisingly finite amount of time the probability of some message never arriving is ~1.0. The number one falsehood programmers believe about networking is that the network is reliable. Even Google and Amazon, as good as they are, can't guarantee perfect network connectivity. In the real world you'll get hit with hardware failures, human failures (e.g. misconfiguration, bugs), random failures (flooding, cosmic rays), malicious failures (hackers), and so on. With this requirement you're guaranteed to have your system grind to a halt eventually. Can you engineer around it? Well yes to a point, redundency and all that, but it's expensive and will still fail eventually. So both Google and Kafka gave you the correct answer.
- HealthLab 3y agoThanks for those thoughts. Our approach is that the sending system sends a numerical order for the messages. We took this approach in healthcare and it helped for when reliability of queues needed to be 100%. In that way we always knew if any message is missing in the queue. Given how often developers need to process a stream of messages - and assure the order of processing is reliable - it seems like such a surprise that queues would not have that native functionality for reliability.
- ac2u 3y ago>We took this approach in healthcare and it helped for when reliability of queues needed to be 100%. You might need to revisit what the concept of reliability means here. Not that it's possible, but your queuing and network system could be infallible but it'll not help if a producer of messages crashes and causes a missing message to begin with. >Given how often developers need to process a stream of messages - and assure the order of processing is reliable I mean what do you do if there's not a message there? Wait? How long? What if it never comes, does your queue just grow and grow in the meantime? Are you expecting to have these messages processed within the hour generally? What if it takes a month to get your missing message and then your system processes them in order a month later, is that ok? As a previous poster said, you can engineer around some things, but all you can do here is pull X messages from the queue/log, check if they have missing messages, if everything is present, process them in order, if it's not, wait and try again Y minutes later. But you still need to decide what constraints to relax, how many times you retry before you give up etc. There's no magic wand here that will provide a free lunch in the form of a product, the decisions have to be made. Also, a good way to reverse this problem is to ask yourself why you need the messages processed in order. If the reason is that certain data is computed when the messages arrive in and the data will only be correct if processed in order, then your answer might be to make the data derived computations from a table where you store the messages after consuming. That way you're not adding massive complexity and headache to your queueing infrastructure. (Kafka/Kinesis is another option here instead of a table, but most people don't need the throughput guarantees).