5 ms·

Uses For A Message Queue

117 points by paddyforan 14y ago

praptak 14y ago
Not listed: ability to route through very restrictive network setups ("galvanically isolated" networks.) MQs, being latency insensitive, can even go over protocols like e-mail.
- carimura 14y ago
  Nice one. #11.
KeyBoardG 14y ago
Great stuff. We used message queues when rewriting major components from an ancient VFP Client heavy application to newer C# code running on an application server. It enabled both asynchronous work as well as ordered operations.
Spearchucker 14y ago
In the freezing cold at Clapham waiting for a train, so not much time here to dig deeper. However, the elasticity part is interesting, and I'd like very much to know what that really means in context. Other than that I've worked with messages queues a lot over the years, 'specially in the late '90s/early 2000's. Cutting to the chase - I have a problem with message queues, in that they tend to add a lot of bloat (in terms of both deployment size and integration cost) and in return don't solve a very difficult problem. First and foremost, "message queue" is really just store and forward. This is hugely relevant in today's mobile world. Being able to edit a row in a database while on the London Underground, and knowing that it will just sync up the next time I connect to a network is a big thing. That said, other than elasticity I'd like very much to know how this is different to MQ Series or MSMQ. Things like in-order delivery and guaranteed delivery are not a big deal to achieve. Guaranteed delivery is super easy. When uploading, you call and the server with your payload, and inspect the server response. If the response is an acknowledge/success you've guaranteed delivery. Downloading a message requires two calls - the first to get the payload, the second to confirm back to the server that you've received it. Again, guaranteed delivery, done! If either the upload or download fails then just keep trying until it works. It doesn't matter if the server gets the data 20 times, it only needs to record one receipt. "Only once" delivery is the same - the server just discards subsequent uploads from clients that didn't get the appropriate response, for whatever reason. In-order delivery is solved by adding a sequence number to the batch/payload packets. You can actually send them in any order you like -- the server simply re-assembles the messages in the required order when the whole batch has been delivered. Where messages are spaced apart in time this is even less of a requirement. What's really interesting to me, and what isn't mentioned at all, is whether they do push from the server. Everything there can be done using client pull. MSMQ and MQ Series both fall down with push, because you can't see through a NAT'd network. Finally, store and forward, by definition, provides resilience and buffering. The same concept of store before forward means that any half-decent implementation will handle spikes (hence my question on "elasticity"). Async comms are always a good idea, because they help you scale. Not sure how others do comms, but I haven't used a synchronous call since .NET first came out in 2002. I think real innovations around store and forward/message queues come from performance (relevant especially on mobile networks, where I prefer to avoid chatty formats like XML or JSoN), and security (encrypting a bitstream without requiring SSL). Because of this I wrote a library that I simply drop into every one of my mobile app projects. It took me a week to build and stabilise (I add that to demonstrate that this is not a difficult engineering problem - and I'm no rocket scientist by any stretch of the imagination). The other innovation that's a fair bit harder to solve, is when a single item is edited by two parties while one is offline, and then merging the updates when both versions sync up. [Edit] I've had a look now, and elasticity means little more here than the caching nature of message queues (the store part of store and forward). As an architect I see yet another middleware vendor, albeit with hipster terms like spike and elasticity. As a programmer I imagine they solved this problem for themselves the same way I did - and if they can make some money from their efforts then they've done more with it than I have, and I think that's awesome for them!
- paddyforan 14y ago
  I'm a little in love with you for the phrase "hipster terms". I am not ignoring your technical points--I think we may have to agree to disagree, here. You certainly have a valid opinion, and are entitled to it. We think a little differently, but variety is the spice, etc. etc. I just saw your edit and couldn't contain my glee at finally earning the moniker "hipster", so I had to comment.
- treeder 14y ago
  What is meant by elasticity is that you can handle varying load on demand. You don't need to provision/scale for spiky/unexpected loads in advance, your infrastructure will expand and contract on demand. This is a very difficult problem to solve. Also, this is on a much different scale than a small queue on a mobile app like you are comparing it to. A queue on a mobile app doesn't really need elasticity or scale. But when you're dealing with millions/billions of messages, the complexity is on a whole different level. And providing the ability to scale/grow with zero effort is extremely compelling. Now if you had millions of users of your mobile app and they all sent messages for processing, an elastic, scalable message queue would be a godsend.
  - Spearchucker 14y ago
    You're not describing anything here that isn't just an inherent attribute of message queuing. It just sends when it can, and keeps trying until the local persistence cache is cleared. What am I missing?
    - RobAley 14y ago
      Nothing, that's what the article is about, "uses for a message queue". The inherent attributes of a queue are what make a queue useful.
- jacques_chester 14y ago
  > In-order delivery is solved by adding a sequence number to the batch/payload packets. And just as I let TCP do this for me at a lower layer, I'd personally be quite happy to farm this out to a queue library or service. That a problem has been solved in a well-known pattern, and that I could solve it with my own code, doesn't mean that I ought to spend time and effort doing so. > security (encrypting a bitstream without requiring SSL). Because of this I wrote a library that I simply drop into every one of my mobile app projects. I am wary of not using SSL. I mean it's a heavy protocol, but that's what you need to rebuff the many, many clever attacks against homegrown protocols. Recently I filed a patent for a protocol involving cryptographic elements and it took me 11 goes-around with a Well-Known Expert to get a protocol that wasn't basically retarded beyond all comprehension. This after spending a year reading about crypto and security for my honours project. I tried, but really, it's one of those topics where soaking for a few years pays and being out of the loop for a month can cost. Anyhow. I guess my question is, why not use SSL? Performance/battery life?
jjm 14y ago
Side note, I think the picture are of White Geese and not ducks.
- paddyforan 14y ago
  I had the same thought. However. http://www.flickr.com/photos/david-hilgart/4142396713/ http://www.flickr.com/photos/david-hilgart/4142396713/ If the photographer calls them ducks, who am I to argue?
xugle 14y ago
It almost sounds like the writer just discovered the concept of a queue. Just a plain old queue. It's as if a 100 years of queueing theory development just illuminated his world in a blinding flash of light and he was compelled to come up with brilliant new terms such as "spikability" to make sense of it all.
bobinator30 14y ago
old news http://blog.pasker.net/2008/06/16/you-might-need-messaging-if/ http://blog.pasker.net/2008/06/16/you-might-need-messaging-i...

117 points by paddyforan 14y ago

Not listed: ability to route through very restrictive network setups ("galvanically isolated" networks.) MQs, being latency insensitive, can even go over protocols like e-mail.

carimura 14y ago
Nice one. #11.

KeyBoardG 14y ago

Great stuff. We used message queues when rewriting major components from an ancient VFP Client heavy application to newer C# code running on an application server. It enabled both asynchronous work as well as ordered operations.

Spearchucker 14y ago

In the freezing cold at Clapham waiting for a train, so not much time here to dig deeper. However, the elasticity part is interesting, and I'd like very much to know what that really means in context. Other than that I've worked with messages queues a lot over the years, 'specially in the late '90s/early 2000's. Cutting to the chase - I have a problem with message queues, in that they tend to add a lot of bloat (in terms of both deployment size and integration cost) and in return don't solve a very difficult problem. First and foremost, "message queue" is really just store and forward. This is hugely relevant in today's mobile world. Being able to edit a row in a database while on the London Underground, and knowing that it will just sync up the next time I connect to a network is a big thing. That said, other than elasticity I'd like very much to know how this is different to MQ Series or MSMQ. Things like in-order delivery and guaranteed delivery are not a big deal to achieve. Guaranteed delivery is super easy. When uploading, you call and the server with your payload, and inspect the server response. If the response is an acknowledge/success you've guaranteed delivery. Downloading a message requires two calls - the first to get the payload, the second to confirm back to the server that you've received it. Again, guaranteed delivery, done! If either the upload or download fails then just keep trying until it works. It doesn't matter if the server gets the data 20 times, it only needs to record one receipt. "Only once" delivery is the same - the server just discards subsequent uploads from clients that didn't get the appropriate response, for whatever reason. In-order delivery is solved by adding a sequence number to the batch/payload packets. You can actually send them in any order you like -- the server simply re-assembles the messages in the required order when the whole batch has been delivered. Where messages are spaced apart in time this is even less of a requirement. What's really interesting to me, and what isn't mentioned at all, is whether they do push from the server. Everything there can be done using client pull. MSMQ and MQ Series both fall down with push, because you can't see through a NAT'd network. Finally, store and forward, by definition, provides resilience and buffering. The same concept of store before forward means that any half-decent implementation will handle spikes (hence my question on "elasticity"). Async comms are always a good idea, because they help you scale. Not sure how others do comms, but I haven't used a synchronous call since .NET first came out in 2002. I think real innovations around store and forward/message queues come from performance (relevant especially on mobile networks, where I prefer to avoid chatty formats like XML or JSoN), and security (encrypting a bitstream without requiring SSL). Because of this I wrote a library that I simply drop into every one of my mobile app projects. It took me a week to build and stabilise (I add that to demonstrate that this is not a difficult engineering problem - and I'm no rocket scientist by any stretch of the imagination). The other innovation that's a fair bit harder to solve, is when a single item is edited by two parties while one is offline, and then merging the updates when both versions sync up. [Edit] I've had a look now, and elasticity means little more here than the caching nature of message queues (the store part of store and forward). As an architect I see yet another middleware vendor, albeit with hipster terms like spike and elasticity. As a programmer I imagine they solved this problem for themselves the same way I did - and if they can make some money from their efforts then they've done more with it than I have, and I think that's awesome for them!

paddyforan 14y ago
I'm a little in love with you for the phrase "hipster terms". I am not ignoring your technical points--I think we may have to agree to disagree, here. You certainly have a valid opinion, and are entitled to it. We think a little differently, but variety is the spice, etc. etc. I just saw your edit and couldn't contain my glee at finally earning the moniker "hipster", so I had to comment.
treeder 14y ago
What is meant by elasticity is that you can handle varying load on demand. You don't need to provision/scale for spiky/unexpected loads in advance, your infrastructure will expand and contract on demand. This is a very difficult problem to solve. Also, this is on a much different scale than a small queue on a mobile app like you are comparing it to. A queue on a mobile app doesn't really need elasticity or scale. But when you're dealing with millions/billions of messages, the complexity is on a whole different level. And providing the ability to scale/grow with zero effort is extremely compelling. Now if you had millions of users of your mobile app and they all sent messages for processing, an elastic, scalable message queue would be a godsend.
- Spearchucker 14y ago
  You're not describing anything here that isn't just an inherent attribute of message queuing. It just sends when it can, and keeps trying until the local persistence cache is cleared. What am I missing?
  - RobAley 14y ago
    Nothing, that's what the article is about, "uses for a message queue". The inherent attributes of a queue are what make a queue useful.
jacques_chester 14y ago
> In-order delivery is solved by adding a sequence number to the batch/payload packets. And just as I let TCP do this for me at a lower layer, I'd personally be quite happy to farm this out to a queue library or service. That a problem has been solved in a well-known pattern, and that I could solve it with my own code, doesn't mean that I ought to spend time and effort doing so. > security (encrypting a bitstream without requiring SSL). Because of this I wrote a library that I simply drop into every one of my mobile app projects. I am wary of not using SSL. I mean it's a heavy protocol, but that's what you need to rebuff the many, many clever attacks against homegrown protocols. Recently I filed a patent for a protocol involving cryptographic elements and it took me 11 goes-around with a Well-Known Expert to get a protocol that wasn't basically retarded beyond all comprehension. This after spending a year reading about crypto and security for my honours project. I tried, but really, it's one of those topics where soaking for a few years pays and being out of the loop for a month can cost. Anyhow. I guess my question is, why not use SSL? Performance/battery life?

jjm 14y ago

Side note, I think the picture are of White Geese and not ducks.

paddyforan 14y ago
I had the same thought. However. http://www.flickr.com/photos/david-hilgart/4142396713/ http://www.flickr.com/photos/david-hilgart/4142396713/ If the photographer calls them ducks, who am I to argue?

xugle 14y ago

It almost sounds like the writer just discovered the concept of a queue. Just a plain old queue. It's as if a 100 years of queueing theory development just illuminated his world in a blinding flash of light and he was compelled to come up with brilliant new terms such as "spikability" to make sense of it all.

bobinator30 14y ago

old news http://blog.pasker.net/2008/06/16/you-might-need-messaging-if/ http://blog.pasker.net/2008/06/16/you-might-need-messaging-i...