4 ms·

We have currently implemented both push and pull models, along with disk storage, and we provide services internally through gRPC. Our goal now is to offer thes

We have currently implemented both push and pull models, along with disk storage, and we provide services internally through gRPC. Our goal now is to offer these services externally. However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. My motivation is quite simple: to leverage existing ecosystems while providing a better implementation for users and reducing migration costs (similar to how Redpanda partially implements the Kafka protocol). Currently, our implementation is in C++ and Rust, but the protocol layer can be in other languages. We have an advantage in throughput and have also implemented features like priority queues that Kafka doesn't have. The AMQP protocol is quite complete (and allows for some customization), but I am not very familiar with the details of the Kafka protocol. I've heard it changes frequently, and I'm concerned that this might affect the implementation.

sc68cal 2y ago
> However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. So, I have a little bit of experience with this, because of some work using Celery which abstracted out the message queue protocol, as well as Redis' pub/sub and AMQP directly when I did some work on Ansible EDA. Honestly the code for consuming from these message systems was an incredibly small percentage of the code, due to the plethora of existing libraries for those protocols. Like, they were almost one-liners to block and consume from the queue. The majority of the code was handling the actual message contents and acting upon them. So, I don't think your statement is accurate. This may be something to consider, because if you are doing the AMQP or Kafka protocol just to be compatible, it may not actually be worth it, and instead it may be worth just providing a protocol that actually maps to what your system does.
- aljun_invictus 2y ago
  You're right, there are many trade-offs involved. I'm considering this as well, but it's best to use a widely adopted ecosystem for external use.

sc68cal 2y ago

> However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. So, I have a little bit of experience with this, because of some work using Celery which abstracted out the message queue protocol, as well as Redis' pub/sub and AMQP directly when I did some work on Ansible EDA. Honestly the code for consuming from these message systems was an incredibly small percentage of the code, due to the plethora of existing libraries for those protocols. Like, they were almost one-liners to block and consume from the queue. The majority of the code was handling the actual message contents and acting upon them. So, I don't think your statement is accurate. This may be something to consider, because if you are doing the AMQP or Kafka protocol just to be compatible, it may not actually be worth it, and instead it may be worth just providing a protocol that actually maps to what your system does.

aljun_invictus 2y ago
You're right, there are many trade-offs involved. I'm considering this as well, but it's best to use a widely adopted ecosystem for external use.