3 ms·

We support both push and pull models (it seems both RabbitMQ and Kafka support both). Our implementation is actually more like Kafka (for example, it supports r

We support both push and pull models (it seems both RabbitMQ and Kafka support both). Our implementation is actually more like Kafka (for example, it supports replay), but implementing RabbitMQ's routing isn't very complicated either. However, I'm concerned about the Kafka protocol, as it seems to have been revised many times historically, which could lead to client compatibility issues. My users typically want to decouple processing logic (not for large-scale data scenarios like logging, more like RabbitMQ use cases), so I'm quite conflicted.

sc68cal 2y ago
So, as another commenter asked, why would someone use your solution instead of Kafka or RabbitMQ directly? It's not clear to me what usecase you are trying to fulfill. Like, at least for me, I reach for RabbitMQ when I am doing RPC or want reliable delivery of messages between systems. I have a lot more experience with RabbitMQ for that usecase. With Kafka, my experience with it is mostly the classic log processing usecase where we are processing syslog messages with Kafka and then Vector to process the syslog messages and transform them. Like, you COULD use RabbitMQ to do log processing and COULD use Kafka to do RPC and reliable message delivery, but each seems to have a more comfortable fit doing the jobs I described and you don't spend a lot of time forcing them to go in ways that thet don't easily go. So again, what is your system trying to accomplish, and what protocol matches what you are trying to do more? I've seen a couple of your comments and it seems like you are trying to be both?
- aljun_invictus 2y ago
  We have currently implemented both push and pull models, along with disk storage, and we provide services internally through gRPC. Our goal now is to offer these services externally. However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. My motivation is quite simple: to leverage existing ecosystems while providing a better implementation for users and reducing migration costs (similar to how Redpanda partially implements the Kafka protocol). Currently, our implementation is in C++ and Rust, but the protocol layer can be in other languages. We have an advantage in throughput and have also implemented features like priority queues that Kafka doesn't have. The AMQP protocol is quite complete (and allows for some customization), but I am not very familiar with the details of the Kafka protocol. I've heard it changes frequently, and I'm concerned that this might affect the implementation.
  - sc68cal 2y ago
    > However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. So, I have a little bit of experience with this, because of some work using Celery which abstracted out the message queue protocol, as well as Redis' pub/sub and AMQP directly when I did some work on Ansible EDA. Honestly the code for consuming from these message systems was an incredibly small percentage of the code, due to the plethora of existing libraries for those protocols. Like, they were almost one-liners to block and consume from the queue. The majority of the code was handling the actual message contents and acting upon them. So, I don't think your statement is accurate. This may be something to consider, because if you are doing the AMQP or Kafka protocol just to be compatible, it may not actually be worth it, and instead it may be worth just providing a protocol that actually maps to what your system does.
    - aljun_invictus 2y ago
      You're right, there are many trade-offs involved. I'm considering this as well, but it's best to use a widely adopted ecosystem for external use.

sc68cal 2y ago

So, as another commenter asked, why would someone use your solution instead of Kafka or RabbitMQ directly? It's not clear to me what usecase you are trying to fulfill. Like, at least for me, I reach for RabbitMQ when I am doing RPC or want reliable delivery of messages between systems. I have a lot more experience with RabbitMQ for that usecase. With Kafka, my experience with it is mostly the classic log processing usecase where we are processing syslog messages with Kafka and then Vector to process the syslog messages and transform them. Like, you COULD use RabbitMQ to do log processing and COULD use Kafka to do RPC and reliable message delivery, but each seems to have a more comfortable fit doing the jobs I described and you don't spend a lot of time forcing them to go in ways that thet don't easily go. So again, what is your system trying to accomplish, and what protocol matches what you are trying to do more? I've seen a couple of your comments and it seems like you are trying to be both?

aljun_invictus 2y ago
We have currently implemented both push and pull models, along with disk storage, and we provide services internally through gRPC. Our goal now is to offer these services externally. However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. My motivation is quite simple: to leverage existing ecosystems while providing a better implementation for users and reducing migration costs (similar to how Redpanda partially implements the Kafka protocol). Currently, our implementation is in C++ and Rust, but the protocol layer can be in other languages. We have an advantage in throughput and have also implemented features like priority queues that Kafka doesn't have. The AMQP protocol is quite complete (and allows for some customization), but I am not very familiar with the details of the Kafka protocol. I've heard it changes frequently, and I'm concerned that this might affect the implementation.
- sc68cal 2y ago
  > However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. So, I have a little bit of experience with this, because of some work using Celery which abstracted out the message queue protocol, as well as Redis' pub/sub and AMQP directly when I did some work on Ansible EDA. Honestly the code for consuming from these message systems was an incredibly small percentage of the code, due to the plethora of existing libraries for those protocols. Like, they were almost one-liners to block and consume from the queue. The majority of the code was handling the actual message contents and acting upon them. So, I don't think your statement is accurate. This may be something to consider, because if you are doing the AMQP or Kafka protocol just to be compatible, it may not actually be worth it, and instead it may be worth just providing a protocol that actually maps to what your system does.
  - aljun_invictus 2y ago
    You're right, there are many trade-offs involved. I'm considering this as well, but it's best to use a widely adopted ecosystem for external use.