5 ms·

What, exactly, do you think a purpose-built queuing system does under the hood? If it’s sane, it’s doing exactly the same transactional row locking an RDBMS is

What, exactly, do you think a purpose-built queuing system does under the hood? If it’s sane, it’s doing exactly the same transactional row locking an RDBMS is highly optimized to do. I have apps with RDBMS queues doing thousands of work items per second with 16 workers. Very few applications need more than this. Introducing a whole new subsystem and API for queueing is a bad engineering decision in most cases. Fan-out is generally an anti-pattern, but if you need it, then explore other options Note that I am talking about Kafka-style linear queues in an RDBMS, not pub-sub. Triggers or app code do very fast transactional inserts into queue as part of the initial write; consumers fast-poll with “update top 1 ... where status = <unprocessed>” and back off polling exponentially when the queue is empty. This is just a few lines of code trivial to do correctly and cannot deadlock in READ COMMITTED or SERIALIZABLE modes.

scarface74 8y ago
How is “fan out” an anti pattern? You have an event or message that needs to be processed by multiple systems. This is completely normal. So introducing a purpose built queueing system - that people have been doing for decades instead of using a database is technically bad? A queueing system at most has to respond to a request, set a flag for “processing” and it’s done and since by definition, a queue only has to read the top most item, it is more efficient. Besides, how do you handle a message that is sitting in a “processing” state because the process that originally read it crashed and didn’t change the status? Triggers or app code do very fast transactional inserts into queue as part of the initial write; consumers fast-poll with “update top 1 ... where status = <unprocessed>” and back off polling exponentially when the queue is empty. This is just a few lines of code trivial to do correctly and cannot deadlock in READ COMMITTED or SERIALIZABLE modes. Or instead of reinventing the wheel with your own bespoke database as a queue that has all of the maintenance overhead you could just use a queueing system that is already optimized for that use case. Note that I am talking about Kafka-style linear queues in an RDBMS, not pub-sub. In general parlance, most people would call Kafka style processing stream processing. Even then, why write and maintain a pseudo streaming process that you have to maintain instead of just using Kafka where you can let it handle all of the fault tolerance, partitioning etc? It’s about like a past company that I worked for where the “architect” had his own homegrown ORM, encryption scheme, and configuration system instead of just using an off the shelf solution because he thought his system was its own special snowflake.
- tatersolid 8y ago
  Fan-out is an anti-pattern because it introduces the complexity of concurrency to an asynchronous process where timing isn’t critical. And If you think a “stream” and a “queue” are somehow different I don’t know what to say. 95% of applications only need a queue for “do this thing asynchronously so the user doesn’t have to wait.” This is where a DB table queue is the best solution. Introducing RabbitMQ or any other service in such a common case is a terrible idea. Every real-world message passing implementation I’ve encountered ends up with the complexity of a bespoke state database for each consumer to handle re-ordering and crashes, as well as poorly written code and more state in a database to handle message replay. Not all messages can be idempotent, and consumers never end up stateless in the real world. All of these solutions lost events in production under various conditions. My point is: think about if you really need the complexity of managing another service in production, when all you really need is “do this thing as soon as you can”.
  - scarface74 8y ago
    Fan-out is an anti-pattern because it introduces the complexity of concurrency to an asynchronous process where timing isn’t critical. And If you think a “stream” and a “queue” are somehow different I don’t know what to say. So now the common definitions of things is wrong. So how do you propose that multiple systems that all care about a single event get notified? You realize that processing queues and using a fan-out pattern has been done for decades? 95% of applications only need a queue for “do this thing asynchronously so the user doesn’t have to wait.” This is where a DB table queue is the best solution. Introducing RabbitMQ or any other service in such a common case is a terrible idea. Because based on your anecdotal experience you can say with confidence that “95%” of people are doing it wrong.... Every real-world message passing implementation I’ve encountered ends up with the complexity of a bespoke state database for each consumer to handle re-ordering and crashes, as well as poorly written code and more state in a database to handle message replay. Not all messages can be idempotent, and consumers never end up stateless in the real world. Well, maybe “in your real world”, but people have been managing queues with idempotency, statelessness, and out of order execution for decades. As far as handling crashes, there is nothing to do. Once the message is in process for a certain amount of time (“in flight”) and the consumer hasn’t acknowledged successful processing, the queueing system automatically puts the message back in the queue. After a certain number of retries it goes into a dead letter queue. All of these solutions lost events in production under various conditions. Don’t blame a poor implementation on the technology. I preach to people all of the time unless you are working at Google or even Twitter scale, you’re not a special snowflake that needs to reinvent the wheel and try to re-solve solved problems. My point is: think about if you really need the complexity of managing another service in production, when all you really need is “do this thing as soon as you can”. “Managing” RabbitMQ is not rocket science. But these days, I don’t deal with managing infrastructure. That’s what cloud providers are for.

scarface74 8y ago

How is “fan out” an anti pattern? You have an event or message that needs to be processed by multiple systems. This is completely normal. So introducing a purpose built queueing system - that people have been doing for decades instead of using a database is technically bad? A queueing system at most has to respond to a request, set a flag for “processing” and it’s done and since by definition, a queue only has to read the top most item, it is more efficient. Besides, how do you handle a message that is sitting in a “processing” state because the process that originally read it crashed and didn’t change the status? Triggers or app code do very fast transactional inserts into queue as part of the initial write; consumers fast-poll with “update top 1 ... where status = <unprocessed>” and back off polling exponentially when the queue is empty. This is just a few lines of code trivial to do correctly and cannot deadlock in READ COMMITTED or SERIALIZABLE modes. Or instead of reinventing the wheel with your own bespoke database as a queue that has all of the maintenance overhead you could just use a queueing system that is already optimized for that use case. Note that I am talking about Kafka-style linear queues in an RDBMS, not pub-sub. In general parlance, most people would call Kafka style processing stream processing. Even then, why write and maintain a pseudo streaming process that you have to maintain instead of just using Kafka where you can let it handle all of the fault tolerance, partitioning etc? It’s about like a past company that I worked for where the “architect” had his own homegrown ORM, encryption scheme, and configuration system instead of just using an off the shelf solution because he thought his system was its own special snowflake.

tatersolid 8y ago
Fan-out is an anti-pattern because it introduces the complexity of concurrency to an asynchronous process where timing isn’t critical. And If you think a “stream” and a “queue” are somehow different I don’t know what to say. 95% of applications only need a queue for “do this thing asynchronously so the user doesn’t have to wait.” This is where a DB table queue is the best solution. Introducing RabbitMQ or any other service in such a common case is a terrible idea. Every real-world message passing implementation I’ve encountered ends up with the complexity of a bespoke state database for each consumer to handle re-ordering and crashes, as well as poorly written code and more state in a database to handle message replay. Not all messages can be idempotent, and consumers never end up stateless in the real world. All of these solutions lost events in production under various conditions. My point is: think about if you really need the complexity of managing another service in production, when all you really need is “do this thing as soon as you can”.
- scarface74 8y ago
  Fan-out is an anti-pattern because it introduces the complexity of concurrency to an asynchronous process where timing isn’t critical. And If you think a “stream” and a “queue” are somehow different I don’t know what to say. So now the common definitions of things is wrong. So how do you propose that multiple systems that all care about a single event get notified? You realize that processing queues and using a fan-out pattern has been done for decades? 95% of applications only need a queue for “do this thing asynchronously so the user doesn’t have to wait.” This is where a DB table queue is the best solution. Introducing RabbitMQ or any other service in such a common case is a terrible idea. Because based on your anecdotal experience you can say with confidence that “95%” of people are doing it wrong.... Every real-world message passing implementation I’ve encountered ends up with the complexity of a bespoke state database for each consumer to handle re-ordering and crashes, as well as poorly written code and more state in a database to handle message replay. Not all messages can be idempotent, and consumers never end up stateless in the real world. Well, maybe “in your real world”, but people have been managing queues with idempotency, statelessness, and out of order execution for decades. As far as handling crashes, there is nothing to do. Once the message is in process for a certain amount of time (“in flight”) and the consumer hasn’t acknowledged successful processing, the queueing system automatically puts the message back in the queue. After a certain number of retries it goes into a dead letter queue. All of these solutions lost events in production under various conditions. Don’t blame a poor implementation on the technology. I preach to people all of the time unless you are working at Google or even Twitter scale, you’re not a special snowflake that needs to reinvent the wheel and try to re-solve solved problems. My point is: think about if you really need the complexity of managing another service in production, when all you really need is “do this thing as soon as you can”. “Managing” RabbitMQ is not rocket science. But these days, I don’t deal with managing infrastructure. That’s what cloud providers are for.