3 ms·

API vs MQ : if there is an API you don't need MQ, or will you make a front MQ for every API you're calling ? This article is inconsiderate of the added complex

API vs MQ : if there is an API you don't need MQ, or will you make a front MQ for every API you're calling ? This article is inconsiderate of the added complexity of MQs. How is the payment processor given as example decoupled from the main process from let's say an e-commerce website using a MQ ? If the payment processor isn't responding, the payment process on the main site is broken and a client can't purchase. If many clients try again to pay, the MQ will be flooded and attain its max limit. How do you deal with added complexity and extra error handling of the MQ ? Same question for integrating a new payment processor: with a different API or SDK to integrate, how would that not be the case with a MQ as stated in the article? A MQ is simply a buffer when you need to scale up service consumption. But services with APIs don't need it, they do rate limiting and if they're down you should better set the status on your program and stop communicating until it's up again. MQs are not automatic and this article is selling false promises and examples.

yuppiepuppie 6y ago
> if there is an API you don't need MQ Im not sure what you are talking about here. The two concepts are quite different and are meant to be used for different processes. > selling false promises AFAIK, the article wasnt selling anything. I think it was a nice intro and overview to message queues.
vorpalhex 6y ago
Using an MQ with a payment processor is very common - payment processors do have transient outages and we still want to allow customers to transact. We may also use multiple payment processors in this case and fallback to a more expensive one if we see our preferred one go offline for a while. MQ's are distinct from APIs. APIs require both sides to be free to talk, while an MQ allows one side to send when it's able and the other to read when it's able which can happen out of sync. MQs do have limits but this is typically several orders of magnitude higher than an API - a few million queued messages is pretty trivial on any decent MQ cluster. An MQ isn't a buffer alone, it's also routing, retries, error handling, deadlettering, etc. This is the difference between your crappy in-memory blind buffer and something like RabbitMQ. Yes MQs still require maintenance/attention/etc, they're not magic.
KaiserPro 6y ago
> if there is an API you don't need MQ, or will you make a front MQ for every API you're calling ? I don't think this is quite right. An API might use a message queue as a transport mechanism. The MQ only gives you a way to communicate, it doesn't define how you structure your requests. in your payment example, if you replace MQ with REST, it gives you the same answer. Some message queues give you rich diagnostics for when messages fail. Others like NATS only give you the guarantee that the server will be up. MQs come in many flavours. some act like python's Queue, but allow you to use many machines. Some are as you say, a buffer. However I like to use them for connecting n transient instances of a service to an API front end. The routing is handled for you, and you can isolate services from each other.
sudhirj 6y ago
I learnt a ton form this talk by Tim Bray about how Amazon does event driven architecture. The talk isn't directly about queues, but they play a massive part in the narrative, including notes about idempotency, duplication, FIFO (or lack of it), and message design. The article adapts a lot of these ideas from a pure queue point of view. https://www.youtube.com/watch?v=h46IquqjF3E https://www.youtube.com/watch?v=h46IquqjF3E

yuppiepuppie 6y ago

> if there is an API you don't need MQ Im not sure what you are talking about here. The two concepts are quite different and are meant to be used for different processes. > selling false promises AFAIK, the article wasnt selling anything. I think it was a nice intro and overview to message queues.

vorpalhex 6y ago

Using an MQ with a payment processor is very common - payment processors do have transient outages and we still want to allow customers to transact. We may also use multiple payment processors in this case and fallback to a more expensive one if we see our preferred one go offline for a while. MQ's are distinct from APIs. APIs require both sides to be free to talk, while an MQ allows one side to send when it's able and the other to read when it's able which can happen out of sync. MQs do have limits but this is typically several orders of magnitude higher than an API - a few million queued messages is pretty trivial on any decent MQ cluster. An MQ isn't a buffer alone, it's also routing, retries, error handling, deadlettering, etc. This is the difference between your crappy in-memory blind buffer and something like RabbitMQ. Yes MQs still require maintenance/attention/etc, they're not magic.

KaiserPro 6y ago

> if there is an API you don't need MQ, or will you make a front MQ for every API you're calling ? I don't think this is quite right. An API might use a message queue as a transport mechanism. The MQ only gives you a way to communicate, it doesn't define how you structure your requests. in your payment example, if you replace MQ with REST, it gives you the same answer. Some message queues give you rich diagnostics for when messages fail. Others like NATS only give you the guarantee that the server will be up. MQs come in many flavours. some act like python's Queue, but allow you to use many machines. Some are as you say, a buffer. However I like to use them for connecting n transient instances of a service to an API front end. The routing is handled for you, and you can isolate services from each other.

sudhirj 6y ago

I learnt a ton form this talk by Tim Bray about how Amazon does event driven architecture. The talk isn't directly about queues, but they play a massive part in the narrative, including notes about idempotency, duplication, FIFO (or lack of it), and message design. The article adapts a lot of these ideas from a pure queue point of view. https://www.youtube.com/watch?v=h46IquqjF3E https://www.youtube.com/watch?v=h46IquqjF3E