13 ms·

ActiveMQ: Not ready for prime time

96 points by sorenbs 15y ago

mgkimsal 15y ago
So, what might you recommend instead? I'm using rabbitmq on a couple of projects, and have been a bit underwhelmed by the lack of management tools that come with it. I figured something as basic as getting a list of what's in a queue and being able to then remove that item from a queue would not require me to write custom code, but I seem to have been wrong.
- timclark 15y ago
  I believe the RabbitMQ management plugin is what you need, provides HTML user interface and HTML API. The BQL plugin, which is now sadly unsupported by the Rabbits, used to be a nice command line interface, it is a shame it is no longer supported.
  - AmrMostafa 15y ago
    The RabbitMQ management plugin does provide an excellent command line interface. You download it along with its docs right from the management UI (/cli). Btw, I'm using RabbitMQ, and I love it. My needs do not include high load or high availability though so I can't speak for that. So what's nice about it? AMQP (you automatically get lots of tools, docs, "expectations", etc), very friendly and active mailing lists, small & clean code base, small footprint, simple, active (more features are always being added). What I don't like about RabbitMQ? While it's FLOSS inside out, its development isn't exactly a "community" work. For example, I can't report an issue, attach a patch, and receive a reviewer/committer feedback about it and possibly get it in. In fact, I can't even report an issue into their issue tracking system -- I just have the mailing list. That said, I believe they said they are going to fix that part "soon".
    - rabbitmq 15y ago
      We have always accepted server patches via the mailing list: https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq... But it's true that the community has been much more involved in clients like Pika, for example. And yes we are planning to open up the bug tracker. A piece of advice for anyone doing an open source project - start with an open tracker, because opening up a previously closed tracker is a royal pain in the butt.
      - dpw 15y ago
        And while we don't use github internally, people can and do submit issues and pull requests there: https://github.com/rabbitmq https://github.com/rabbitmq (N.B. we need contributors to sign a contributor agreement.) David (rabbiteer)
  - rabbitmq 15y ago
    Tim, what do you think is cool about BQL that might be needed in the future?
th0ma5 15y ago
a lot of what is listed here are all standard caveats of things one needs to consider in just about any heavy enterprise-java environment. i think you could sum this up that he simply wasn't working in a situation that warranted the division-of-labor and configuration flexibility that the product offers.
lucraft 15y ago
One thing I remember in particular from ActiveMQ was that it would often get itself into the state of having negative queue lengths.
- cmkrnl 15y ago
  The prefetching must be quite aggressive :)
- rajdavies 15y ago
  That hasn't been the case for a long while
adamgravitis 15y ago
The thing I've found about message queues is that it's pretty unimportant what platform they're based on. I used to be tempted to go with ActiveMQ "because it runs on the JVM", but that turns out to be pretty irrelevant. No matter what your queue provider, you're most likely just using TCP sockets anyway, so don't think you need to be using Erlang just to take advantage of RabbitMQ. Yes, this means the MQ can basically be thought of as a black box. But that's also kind of the point.
- marshray 15y ago
  What I've seen with RabbitMQ is that the memory usage can grow and grow. If it runs out of either committed RAM or address space, it crashes. I suspect this may be a feature of the open source Erlang VM that's improved in the pay-for-support version. Even though RabbitMQ pays the performance cost of bouncing everything off the disk, it's not quite recoverable from that type of crash. I'm sure my information is old, perhaps these things are improved in newer versions. Still, the behavior of the Erlang VM itself was relevant. As a C++ programmer, I wouldn't feel comfortable abstracting away memory management for this type of server which is both so memory intensive and requires high reliability. But of course many people swear by highly reliable JVM based stuff, so perhaps I'm wrong about it. :-)
  - deleted 15y ago
    [deleted]
  - FooBarWidget 15y ago
    I've used RabbitMQ 2.4 in production with queue lengths of several hundred thousand. Each queue item can be as large as 100 KB. RabbitMQ's memory usage never went beyond ~150 MB.
  - rbranson 15y ago
    Actually Erlang/OTP provides exactly the type of abstraction layer that's extremely useful for RabbitMQ's use case. It's specifically designed for consistent latency, predictable memory usage, and resilience under failure conditions. You know, basically the things that make an MQ worth using. It was true that previously RabbitMQ could run out of memory and be unable to take more messages, but that changed in 2.0 and it will now swap messages to disk. Of course, then you pay the consequences of slow disks and can always run out of disk space, but nothing is invincible.
PaulHoule 15y ago
sorta reminds me of the time I built something in 2001 which was a reactive system that was sort of a web crawler but also a bit more... seems demented looking back, but I used qmail as a message queue system for it.
- th0ma5 15y ago
  i've considered xmpp as a queue, and hear some do use it!
  - rexreed 15y ago
    Yes - I've been seriously considering XMPP for this purpose. I haven't seen any serious reason not to consider it other than some people's dislike of IM protocols.
- calloc 15y ago
  qmail's speed is something that it's rivals still don't have.
ibejoeb 15y ago
> It's not correctly configured out of the box. You mean there's a correct configuration? Also, Tomcat, Jetty, PostgreSQL, and Ngnix don't "install correctly" out of the box as described here. Tomcat clusters set themselves up? Postgres? Hell, I think default shared buffers allocation is 32 MB... > Anything that streams bytes I'm not sure what the author is trying to do. Doesn't sound like queuing to me... The documentation is terrible, so I'll give that up. It is unfortunate that there is so much experimentation involved in setting it up. ActiveMQ is immensely versatile and configurable, and it's sort of necessarily complex to get it perfect for your task. It's not a turn-key software; it's a systems architecture component.
- ja27 15y ago
  Well, if every piece of enterprise software worked right out of the box, consultants would be out of business. I have to admit that I'm not a big user of ActiveMQ, but I've used a closed-source equivalent from a slightly bigger vendor (IBM) for many years. It also is not configured correctly out of the box for most enterprises. When new features or platforms come out they are often buggy. The documentation though is usually decent until you really want to dig into the internals and understand it.
  - cmkrnl 15y ago
    Ah, dear old WebSphere MQ. Generally awful, bad high-availability story unless you invest in heavy-handed clustering at the OS level. Great for enterprise integration if there's a mainframe around, I suppose. Documentation: quantity over quality. They do have a lot of documentation to show for the cost of the product. Written in C++, so more memory-efficient than messaging middleware written in Java. Excuse my ranting -- I'm a middleware system administrator and WMQ is the necessary evil at my workplace.
- cmkrnl 15y ago
  Yes, streaming doesn't sound like queueing, but both JMS (and many vendors implement JMS) and AMQP support some form of streaming. Of course, in combination with high availability, YMMV.
- 15y ago
nonrecursive 15y ago
Does anyone have recommendations for resources on messaging/queueing basics? I'm look for something on the basic patterns (pubsub, broadcast... ?) and basic considerations to be aware of.
- tlrobinson 15y ago
  As far as considerations to be aware of, Second Life posted their message queue evaluation notes: http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Not...
  - nonrecursive 15y ago
    This looks helpful, too. Thank you!
- andrewcooke 15y ago
  http://www.eaipatterns.com/ http://www.eaipatterns.com/ perhaps?
  - nonrecursive 15y ago
    That looks helpful- thanks!
  - vailripper 15y ago
    This is what I used to get up to speed on integration patterns. The book that the site is based on is very very valuable as well.
- jhawk28 15y ago
  The getting started guide for RabbitMQ is well done: http://www.rabbitmq.com/getstarted.html http://www.rabbitmq.com/getstarted.html The ZeroMQ guide goes into more indepth: http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all
  - 15y ago
veyron 15y ago
How does activemq compare to zeromq?
- andrewcooke 15y ago
  activemq should offer higher level guarantees about reliability. zeromq is simpler. you can argue that activemq has things you will probably end up implementing yourself on top of zeromq, or that zeromq has less to get wrong... another way of saying the same thing, which illustrates the cultural differences: - activemq is intended to be used in "the enterprise". it tries to implements a logical ideal, which is a reliable infrastructure that services can use without being coupled to each other - without worrying about whether messages were received, or exactly who they go to. to reduce complexity it uses a central broker (so you send messages to a central "hub"). - zeromq is intended (imho) for programmers that want to wire things together. it's less concerned with abstractions and more with providing something simple clear, simple and flexible that can be understood and used well. to reduce latency it uses direct connections between peers. from that viewpoint, you can see that the two are both orthogonal and yet similar... (disclaimer: i haven't used either, but i used work on an ESB so have a vague grasp of what's going on. please someone correct me if this is wrong - i might as well learn as i lose karma ;) ps rabbitmq is somewhere in the middle and was (i think) originally more performance-motivated (i believe it's used in finance for example - when speed might be critical).
  - veyron 15y ago
    "i might as well learn" <-- that's why I read HN -- learn something new every day :)
  - aaronblohowiak 15y ago
    "it uses direct connections between peers" -- this is kind of true.. the guide suggests that ultimately you will want a broker (what it calls "devices") to intermediate producer-consumer interactions.
  - rabbitmq 15y ago
    If you want to know what motivated the creation of RabbitMQ please read chapter 1 of Jason and Alvaro's book, "RabbitMQ in Action" - http://www.manning.com/videla/Videla_MEAPCH01.pdf http://www.manning.com/videla/Videla_MEAPCH01.pdf RabbitMQ's main motivation has been to make it easier to join systems together, scale your applications and manage complex environments. That is what messaging is for. Back in 2006, we felt there was a need for a good, stable and scalable open source licensed product that could compete with the incumbents. Notice that I did not mention performance. RabbitMQ has good performance and it is used quite a lot in finance, but the majority of users are what you might categorise as "anyone using MySQL or Postgres". Re activemq vs zeromq, I recommend reading "broker vs brokerless" on our blog. Hope this helps.
lkesteloot 15y ago
I used ActiveMQ in a large production environment for two years. It's by a large margin the worst piece of software I've used professionally. You wouldn't believe the number of very serious problems we had with it (including some mentioned here, like negative queue lengths, but mostly broker crashes, missing or wrong documentation, outright broken features, serious threading issues, and poor performance). If anyone's not convinced, just peek at the source code. It's frightening. We ended up writing a new queuing system from scratch in a month, and in its first week it was already more stable, performant, and bug-free than ActiveMQ.
- jsvaughan 15y ago
  Hi - this is pretty damning. We're currently building an application that uses ActiveMQ but I could switch to a different broker if I thought it was justified. Do you remember which version you were using?
  - lkesteloot 15y ago
    I believe it was 5.0 and 5.1.
    - ShabbyDoo 15y ago
      I experienced similar issues with these versions.
- mindcrime 15y ago
  Why'd you write one from scratch instead of just using JBossMQ or HornetQ or something? ActiveMQ is far from being the only open-source message queuing product, even if you only consider JMS compliant ones.
  - lkesteloot 15y ago
    This was two years ago. JBossMQ was being rewritten and the new version was too immature. I don't remember HornetQ. We investigated RabbitMQ, which seemed good but still a bit young to bet our company on. We also needed some interesting features that none of these provided. In the end writing our own was the best decision we made my whole time at that company -- it paid dividends the rest of my time there. I just found a thread I started about ActiveMQ and scaling: http://activemq.2283324.n4.nabble.com/6000-ActiveMQ-clients-td2361579.html http://activemq.2283324.n4.nabble.com/6000-ActiveMQ-clients-... Note the last message there. 36,000 threads when using NIO! And these answers were from the people commercially supporting ActiveMQ (FuseSource). I never did find out (on that thread or elsewhere) whether ActiveMQ has ever been used in a situation that large. I suspect it hasn't.
kitplummer 15y ago
Yikes. If you're smart, you'll do your diligence and not rely on one person's experience with ActiveMQ. Remember, it is an Apache project - so if something aint there, feel free to fill it in (re: documentation). I can't believe the ActiveMQ/FuseSource guys weren't willing to work it out with David either. Perhaps there's a bit of open source/commercial software wrangling behind this story. Won't be the first/last time.
- rajdavies 15y ago
  I'm not sure who David Pollak talked to at FuseSource - nobody on the ActiveMQ team there know him - and the execs are scratching their heads too.
Maro 15y ago
I'm not too familiar with the message queue use-case, but perhaps this is something you could do with ScalienDB, which supports replication and failover. Disclaimer: I wrote it.
- aaronblohowiak 15y ago
  A message queue is like a database except people can open a socket and say "notify me when there is a new value for this key" -- and you can chose to let multiple people get in line for such updates and then wether you want to let everyone know or just the first person or so-on. you can implement all of this on any db with polling and locks, but that is not an efficient way to do things.
rbranson 15y ago
It's absolutely incredible that it can take so much effort just to make it run. ActiveMQ seems shiny and interesting on the outside, but the prolific number of war stories took it off my list. In brilliant contrast, RabbitMQ is it-just-works software. It's a piece of infrastructure that works so well it practically just fades into the background. I couldn't imagine having to worry about a service in which the chief goal is to help alleviate reliability concerns.
- Uchikoma 15y ago
  The Reddit guys think differently: "[...] (rabbitmq) died, which added about an hour to the downtime. It dies like this pretty often at 2am or at other especially bad times. Usually it doesn't cause any data-loss, just sleep-loss (its queues are persisted and the apps just build up their own queues until it comes back up), but in this case it decided to crash in a way that corrupted its database of persisted queues beyond repair. rabbitmq accounts for the only unrecoverable data-loss incurred, which was about 400 votes. [...] Coincidentally, rabbitmq crashed twice more that day and a few more times into the weekend. [...] Things have improved thus far, but replacing rabbitmq is at the top end of our extremely long list of things to do."
  - rbranson 15y ago
    "Crashed"... I'm glad they're using such specific terms. I give them a lot of slack because they run that shop with a skeleton crew, but they sure do run into a lot of issues with perfectly good software, have Twitter levels of performance & availability, and make some very odd technical decisions.
  - rabbitmq 15y ago
    So For The Record. Reddit are atill using RabbitMQ and as far as I can tell they are happy with it. If anyone has any doubts or questions about this, please email us. info@rabbitmq.com
ShabbyDoo 15y ago
I see lots of questions here about which queuing implementations one ought to consider. Some things to think about: 1. Do you need transactionality? What if your consumer experiences an error after it takes a message off the queue but before it finishes up processing it? Is it ok for that message to disappear forever? The common use case to consider is a consumer which takes a message from a queue, does some processing, and updates a database. To avoid distributed transactions, people often write these consumers so that they check first to see if a message has already been processed. This way, one simply ensures that his DB transaction commits before committing to the consumption of the message. 2. What level of message durability is required? Is it ok for all enqueued messages to disappear when a power cord is inadvertently pulled? I used ActiveMQ to populate a user behavior data warehouse. As users did stuff on a website, the application servers would enqueue observations (went to the product detail page, etc.) and these observations would eventually wind-up in a big Oracle DB. A delay of a day or so didn't matter, but we didn't want to lose more than a fraction of a percent of our observations. So, the queue was in effect a temporary system of record, and we had to allow for reboots, power outages, etc. 3. High Availability. Many modern queueing implementations can be deployed in redundant, scalable "meshes", ActiveMQ included. I haven't kept up on the feature sets of RabbitMQ, JBoss's offering, etc. or I would comment more here. There are obviously a bunch of other considerations, but these are often the ones which are ignored when comparing ZeroMQ to ActiveMQ, etc. There's no right or wrong answer. "Lighter" implementations are often appropriate for messages where, if the shit hits the fan with your environment, the messages are useless anyway. IIRC, eBay builds pages by firing off a bunch of async requests for page parts, waits a maximum of N milliseconds, and then renders the page based on which pieces made it back on time. There's no value in persistence or transactionality, but the mesh sure better scale up.
nerd_in_rage 15y ago
lol. At a previous job, I used ActiveMQ. It was definitely a flaky piece of junk.
wslh 15y ago
There is a lot of buzz about open source queue systems. I have tried for example the RabbitMQ with very bad results. I'll rather explain... one interesting concept about queues is contention and this is where RabbitMQ and others are behind. The strange thing is that contention is an old concept from the old mainframes. For example the last time I checked you can't block a produced in RabbitMQ based on the number of messages on the queue.
- rabbitmq 15y ago
  If you have any questions or problems involving RabbitMQ please email us (info@rabbitmq.com) or post to the mailing list. "one interesting concept about queues is contention and this is where RabbitMQ and others are behind" Can you explain what this means? What kind of contention are you talking about. You say "RabbitMQ and others" - which others? Who implements this feature and what does it look like? "you can't block a produced in RabbitMQ based on the number of messages on the queue" Yes you can. Well - it depends on your use case. RabbitMQ enables you to determine queue length, and supports flow control. But, perhaps you had something else in mind?
  - wslh 15y ago
    Please let me know where I am wrong: - RabbitMQ has a memory based flow control based on the total memory you have in the machine. I know I can develop something over RabbitMQ to accomplish what I want (limits based on number of messages) but I prefer to have the support within RabbitMQ. - WebLogic JMS can block producers based on number of messages as a flow control method.
    - rabbitmq 15y ago
      I don't know what WebLogic can do for flow control exactly. Please email rabbitmq-discuss to get a definitive answer and discussion on what you can do with Rabbit, which supports various mechanisms.
anttipoi 15y ago
I've now been involved in two projects where I would have needed a queue with following characteristics: - simple pub-sub, queue-like: m producers, n consumers - decent performance (order of hundreds of requests per second) - high availability for message persistence (I don't want to loose messages) - no strict FIFO needed, just some kind of lesser fairness (newer items should not block older ones from passing through) ActiveMQ with active-backup setup over shared disk mount is the current choice, and the start-up is really slow if the queue has a lot of data. RabbitMQ does not persist messages in HA fashion, so I've ignored it so far. Maybe HornetQ needs some attention. I see a lot of flexibility and feature-richness in the queue landscape and it perplexes me that getting this simple combination of basics right is so difficult.
- rbranson 15y ago
  RabbitMQ itself doesn't have HA built-in, but warm spare HA is rather simple and well-documented, built on-top of Pacemaker and DRBD. It works quite well.
  - rabbitmq 15y ago
    Thank-you. The docs for HA are here: http://www.rabbitmq.com/pacemaker.html http://www.rabbitmq.com/pacemaker.html This also works with Veritas if you use that instead of DRDB. Please note that we are currently QAing a new HA model which is active/active. Watch this space!
rajdavies 15y ago
BTW - David Pollak didn't talk to any of the ActiveMQ team at FuseSource - else we would have helped him out - he didn't speak to any of the execs either. Its a shame he doesn't allow comments on his blog.
ghostganz 15y ago
I had a lot of trouble with the network-of-brokers feature in ActiveMQ a few years ago, which in the end forced us to switch to another solution. In a later, simpler, project I had issues with ActiveMQ locking up after a certain number of messages, which was solved with an upgrade. On a whole, I think the quality of ActiveMQ is lower than other open source projects with similar brand recognition, e.g. other popular Apache projects.
nvarsj 15y ago
As a counter point, I have used ActiveMQ for a pub/sub architecture and it worked great (millions of messages a day, which was benchmarked as only about 10% of potential on a single broker). We did treat it as an API rather than a deployable component though. We wrapped the broker in our own service architecture (you can instantiate a broker like any POJO) and disabled persistency. Queues were used only very lightly. So maybe we dodged a bullet there... :-).
tvaillancourt 15y ago
We use ActiveMQ in production. I couldn't agree more that it is not ready for prime time. We've had a pretty bad experience overall. Man I wish we used Amazon SQS... We have had incidents where it's Master/Slave replication continues on failure when configured to shutdown the master and slave brokers. It doesn't achieve the consistency it lists as a feature, and provides zero visibility of progress. I feel many of the features the project lists are incomplete, and are listed as a grasp at straws to have the bigger list. Due to the poor quality of the replication, we had to invest a tonne of time to implement replication at the block device level with DRBD+GFS2, complicating the system and (benchmarks pending) likely decreasing performance, all to achieve a 'feature' ActiveMQ boasts. I would also challenge a project to have worse, more incomplete documentation. At the same time, I thank the ActiveMQ team for their contributions and ask the community to not turn it's back on this potentially decent solution. The ideas are solid. It isn't production ready, I feel, but that just means it needs some love. Cheers, Tim Vaillancourt

96 points by sorenbs 15y ago

So, what might you recommend instead? I'm using rabbitmq on a couple of projects, and have been a bit underwhelmed by the lack of management tools that come with it. I figured something as basic as getting a list of what's in a queue and being able to then remove that item from a queue would not require me to write custom code, but I seem to have been wrong.

timclark 15y ago
I believe the RabbitMQ management plugin is what you need, provides HTML user interface and HTML API. The BQL plugin, which is now sadly unsupported by the Rabbits, used to be a nice command line interface, it is a shame it is no longer supported.
- AmrMostafa 15y ago
  The RabbitMQ management plugin does provide an excellent command line interface. You download it along with its docs right from the management UI (/cli). Btw, I'm using RabbitMQ, and I love it. My needs do not include high load or high availability though so I can't speak for that. So what's nice about it? AMQP (you automatically get lots of tools, docs, "expectations", etc), very friendly and active mailing lists, small & clean code base, small footprint, simple, active (more features are always being added). What I don't like about RabbitMQ? While it's FLOSS inside out, its development isn't exactly a "community" work. For example, I can't report an issue, attach a patch, and receive a reviewer/committer feedback about it and possibly get it in. In fact, I can't even report an issue into their issue tracking system -- I just have the mailing list. That said, I believe they said they are going to fix that part "soon".
  - rabbitmq 15y ago
    We have always accepted server patches via the mailing list: https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq... But it's true that the community has been much more involved in clients like Pika, for example. And yes we are planning to open up the bug tracker. A piece of advice for anyone doing an open source project - start with an open tracker, because opening up a previously closed tracker is a royal pain in the butt.
    - dpw 15y ago
      And while we don't use github internally, people can and do submit issues and pull requests there: https://github.com/rabbitmq https://github.com/rabbitmq (N.B. we need contributors to sign a contributor agreement.) David (rabbiteer)
- rabbitmq 15y ago
  Tim, what do you think is cool about BQL that might be needed in the future?

th0ma5 15y ago

a lot of what is listed here are all standard caveats of things one needs to consider in just about any heavy enterprise-java environment. i think you could sum this up that he simply wasn't working in a situation that warranted the division-of-labor and configuration flexibility that the product offers.

lucraft 15y ago

One thing I remember in particular from ActiveMQ was that it would often get itself into the state of having negative queue lengths.

cmkrnl 15y ago
The prefetching must be quite aggressive :)
rajdavies 15y ago
That hasn't been the case for a long while

adamgravitis 15y ago

The thing I've found about message queues is that it's pretty unimportant what platform they're based on. I used to be tempted to go with ActiveMQ "because it runs on the JVM", but that turns out to be pretty irrelevant. No matter what your queue provider, you're most likely just using TCP sockets anyway, so don't think you need to be using Erlang just to take advantage of RabbitMQ. Yes, this means the MQ can basically be thought of as a black box. But that's also kind of the point.

marshray 15y ago
What I've seen with RabbitMQ is that the memory usage can grow and grow. If it runs out of either committed RAM or address space, it crashes. I suspect this may be a feature of the open source Erlang VM that's improved in the pay-for-support version. Even though RabbitMQ pays the performance cost of bouncing everything off the disk, it's not quite recoverable from that type of crash. I'm sure my information is old, perhaps these things are improved in newer versions. Still, the behavior of the Erlang VM itself was relevant. As a C++ programmer, I wouldn't feel comfortable abstracting away memory management for this type of server which is both so memory intensive and requires high reliability. But of course many people swear by highly reliable JVM based stuff, so perhaps I'm wrong about it. :-)
- deleted 15y ago
  [deleted]
- FooBarWidget 15y ago
  I've used RabbitMQ 2.4 in production with queue lengths of several hundred thousand. Each queue item can be as large as 100 KB. RabbitMQ's memory usage never went beyond ~150 MB.
- rbranson 15y ago
  Actually Erlang/OTP provides exactly the type of abstraction layer that's extremely useful for RabbitMQ's use case. It's specifically designed for consistent latency, predictable memory usage, and resilience under failure conditions. You know, basically the things that make an MQ worth using. It was true that previously RabbitMQ could run out of memory and be unable to take more messages, but that changed in 2.0 and it will now swap messages to disk. Of course, then you pay the consequences of slow disks and can always run out of disk space, but nothing is invincible.

PaulHoule 15y ago

sorta reminds me of the time I built something in 2001 which was a reactive system that was sort of a web crawler but also a bit more... seems demented looking back, but I used qmail as a message queue system for it.

th0ma5 15y ago
i've considered xmpp as a queue, and hear some do use it!
- rexreed 15y ago
  Yes - I've been seriously considering XMPP for this purpose. I haven't seen any serious reason not to consider it other than some people's dislike of IM protocols.
calloc 15y ago
qmail's speed is something that it's rivals still don't have.

ibejoeb 15y ago

> It's not correctly configured out of the box. You mean there's a correct configuration? Also, Tomcat, Jetty, PostgreSQL, and Ngnix don't "install correctly" out of the box as described here. Tomcat clusters set themselves up? Postgres? Hell, I think default shared buffers allocation is 32 MB... > Anything that streams bytes I'm not sure what the author is trying to do. Doesn't sound like queuing to me... The documentation is terrible, so I'll give that up. It is unfortunate that there is so much experimentation involved in setting it up. ActiveMQ is immensely versatile and configurable, and it's sort of necessarily complex to get it perfect for your task. It's not a turn-key software; it's a systems architecture component.

ja27 15y ago
Well, if every piece of enterprise software worked right out of the box, consultants would be out of business. I have to admit that I'm not a big user of ActiveMQ, but I've used a closed-source equivalent from a slightly bigger vendor (IBM) for many years. It also is not configured correctly out of the box for most enterprises. When new features or platforms come out they are often buggy. The documentation though is usually decent until you really want to dig into the internals and understand it.
- cmkrnl 15y ago
  Ah, dear old WebSphere MQ. Generally awful, bad high-availability story unless you invest in heavy-handed clustering at the OS level. Great for enterprise integration if there's a mainframe around, I suppose. Documentation: quantity over quality. They do have a lot of documentation to show for the cost of the product. Written in C++, so more memory-efficient than messaging middleware written in Java. Excuse my ranting -- I'm a middleware system administrator and WMQ is the necessary evil at my workplace.
cmkrnl 15y ago
Yes, streaming doesn't sound like queueing, but both JMS (and many vendors implement JMS) and AMQP support some form of streaming. Of course, in combination with high availability, YMMV.
15y ago

nonrecursive 15y ago

Does anyone have recommendations for resources on messaging/queueing basics? I'm look for something on the basic patterns (pubsub, broadcast... ?) and basic considerations to be aware of.

tlrobinson 15y ago
As far as considerations to be aware of, Second Life posted their message queue evaluation notes: http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Not...
- nonrecursive 15y ago
  This looks helpful, too. Thank you!
andrewcooke 15y ago
http://www.eaipatterns.com/ http://www.eaipatterns.com/ perhaps?
- nonrecursive 15y ago
  That looks helpful- thanks!
- vailripper 15y ago
  This is what I used to get up to speed on integration patterns. The book that the site is based on is very very valuable as well.
jhawk28 15y ago
The getting started guide for RabbitMQ is well done: http://www.rabbitmq.com/getstarted.html http://www.rabbitmq.com/getstarted.html The ZeroMQ guide goes into more indepth: http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all
- 15y ago

veyron 15y ago

How does activemq compare to zeromq?

andrewcooke 15y ago
activemq should offer higher level guarantees about reliability. zeromq is simpler. you can argue that activemq has things you will probably end up implementing yourself on top of zeromq, or that zeromq has less to get wrong... another way of saying the same thing, which illustrates the cultural differences: - activemq is intended to be used in "the enterprise". it tries to implements a logical ideal, which is a reliable infrastructure that services can use without being coupled to each other - without worrying about whether messages were received, or exactly who they go to. to reduce complexity it uses a central broker (so you send messages to a central "hub"). - zeromq is intended (imho) for programmers that want to wire things together. it's less concerned with abstractions and more with providing something simple clear, simple and flexible that can be understood and used well. to reduce latency it uses direct connections between peers. from that viewpoint, you can see that the two are both orthogonal and yet similar... (disclaimer: i haven't used either, but i used work on an ESB so have a vague grasp of what's going on. please someone correct me if this is wrong - i might as well learn as i lose karma ;) ps rabbitmq is somewhere in the middle and was (i think) originally more performance-motivated (i believe it's used in finance for example - when speed might be critical).
- veyron 15y ago
  "i might as well learn" <-- that's why I read HN -- learn something new every day :)
- aaronblohowiak 15y ago
  "it uses direct connections between peers" -- this is kind of true.. the guide suggests that ultimately you will want a broker (what it calls "devices") to intermediate producer-consumer interactions.
- rabbitmq 15y ago
  If you want to know what motivated the creation of RabbitMQ please read chapter 1 of Jason and Alvaro's book, "RabbitMQ in Action" - http://www.manning.com/videla/Videla_MEAPCH01.pdf http://www.manning.com/videla/Videla_MEAPCH01.pdf RabbitMQ's main motivation has been to make it easier to join systems together, scale your applications and manage complex environments. That is what messaging is for. Back in 2006, we felt there was a need for a good, stable and scalable open source licensed product that could compete with the incumbents. Notice that I did not mention performance. RabbitMQ has good performance and it is used quite a lot in finance, but the majority of users are what you might categorise as "anyone using MySQL or Postgres". Re activemq vs zeromq, I recommend reading "broker vs brokerless" on our blog. Hope this helps.

lkesteloot 15y ago

I used ActiveMQ in a large production environment for two years. It's by a large margin the worst piece of software I've used professionally. You wouldn't believe the number of very serious problems we had with it (including some mentioned here, like negative queue lengths, but mostly broker crashes, missing or wrong documentation, outright broken features, serious threading issues, and poor performance). If anyone's not convinced, just peek at the source code. It's frightening. We ended up writing a new queuing system from scratch in a month, and in its first week it was already more stable, performant, and bug-free than ActiveMQ.

jsvaughan 15y ago
Hi - this is pretty damning. We're currently building an application that uses ActiveMQ but I could switch to a different broker if I thought it was justified. Do you remember which version you were using?
- lkesteloot 15y ago
  I believe it was 5.0 and 5.1.
  - ShabbyDoo 15y ago
    I experienced similar issues with these versions.
mindcrime 15y ago
Why'd you write one from scratch instead of just using JBossMQ or HornetQ or something? ActiveMQ is far from being the only open-source message queuing product, even if you only consider JMS compliant ones.
- lkesteloot 15y ago
  This was two years ago. JBossMQ was being rewritten and the new version was too immature. I don't remember HornetQ. We investigated RabbitMQ, which seemed good but still a bit young to bet our company on. We also needed some interesting features that none of these provided. In the end writing our own was the best decision we made my whole time at that company -- it paid dividends the rest of my time there. I just found a thread I started about ActiveMQ and scaling: http://activemq.2283324.n4.nabble.com/6000-ActiveMQ-clients-td2361579.html http://activemq.2283324.n4.nabble.com/6000-ActiveMQ-clients-... Note the last message there. 36,000 threads when using NIO! And these answers were from the people commercially supporting ActiveMQ (FuseSource). I never did find out (on that thread or elsewhere) whether ActiveMQ has ever been used in a situation that large. I suspect it hasn't.

kitplummer 15y ago

Yikes. If you're smart, you'll do your diligence and not rely on one person's experience with ActiveMQ. Remember, it is an Apache project - so if something aint there, feel free to fill it in (re: documentation). I can't believe the ActiveMQ/FuseSource guys weren't willing to work it out with David either. Perhaps there's a bit of open source/commercial software wrangling behind this story. Won't be the first/last time.

rajdavies 15y ago
I'm not sure who David Pollak talked to at FuseSource - nobody on the ActiveMQ team there know him - and the execs are scratching their heads too.

Maro 15y ago

I'm not too familiar with the message queue use-case, but perhaps this is something you could do with ScalienDB, which supports replication and failover. Disclaimer: I wrote it.

aaronblohowiak 15y ago
A message queue is like a database except people can open a socket and say "notify me when there is a new value for this key" -- and you can chose to let multiple people get in line for such updates and then wether you want to let everyone know or just the first person or so-on. you can implement all of this on any db with polling and locks, but that is not an efficient way to do things.

rbranson 15y ago

It's absolutely incredible that it can take so much effort just to make it run. ActiveMQ seems shiny and interesting on the outside, but the prolific number of war stories took it off my list. In brilliant contrast, RabbitMQ is it-just-works software. It's a piece of infrastructure that works so well it practically just fades into the background. I couldn't imagine having to worry about a service in which the chief goal is to help alleviate reliability concerns.

Uchikoma 15y ago
The Reddit guys think differently: "[...] (rabbitmq) died, which added about an hour to the downtime. It dies like this pretty often at 2am or at other especially bad times. Usually it doesn't cause any data-loss, just sleep-loss (its queues are persisted and the apps just build up their own queues until it comes back up), but in this case it decided to crash in a way that corrupted its database of persisted queues beyond repair. rabbitmq accounts for the only unrecoverable data-loss incurred, which was about 400 votes. [...] Coincidentally, rabbitmq crashed twice more that day and a few more times into the weekend. [...] Things have improved thus far, but replacing rabbitmq is at the top end of our extremely long list of things to do."
- rbranson 15y ago
  "Crashed"... I'm glad they're using such specific terms. I give them a lot of slack because they run that shop with a skeleton crew, but they sure do run into a lot of issues with perfectly good software, have Twitter levels of performance & availability, and make some very odd technical decisions.
- rabbitmq 15y ago
  So For The Record. Reddit are atill using RabbitMQ and as far as I can tell they are happy with it. If anyone has any doubts or questions about this, please email us. info@rabbitmq.com

ShabbyDoo 15y ago

I see lots of questions here about which queuing implementations one ought to consider. Some things to think about: 1. Do you need transactionality? What if your consumer experiences an error after it takes a message off the queue but before it finishes up processing it? Is it ok for that message to disappear forever? The common use case to consider is a consumer which takes a message from a queue, does some processing, and updates a database. To avoid distributed transactions, people often write these consumers so that they check first to see if a message has already been processed. This way, one simply ensures that his DB transaction commits before committing to the consumption of the message. 2. What level of message durability is required? Is it ok for all enqueued messages to disappear when a power cord is inadvertently pulled? I used ActiveMQ to populate a user behavior data warehouse. As users did stuff on a website, the application servers would enqueue observations (went to the product detail page, etc.) and these observations would eventually wind-up in a big Oracle DB. A delay of a day or so didn't matter, but we didn't want to lose more than a fraction of a percent of our observations. So, the queue was in effect a temporary system of record, and we had to allow for reboots, power outages, etc. 3. High Availability. Many modern queueing implementations can be deployed in redundant, scalable "meshes", ActiveMQ included. I haven't kept up on the feature sets of RabbitMQ, JBoss's offering, etc. or I would comment more here. There are obviously a bunch of other considerations, but these are often the ones which are ignored when comparing ZeroMQ to ActiveMQ, etc. There's no right or wrong answer. "Lighter" implementations are often appropriate for messages where, if the shit hits the fan with your environment, the messages are useless anyway. IIRC, eBay builds pages by firing off a bunch of async requests for page parts, waits a maximum of N milliseconds, and then renders the page based on which pieces made it back on time. There's no value in persistence or transactionality, but the mesh sure better scale up.

nerd_in_rage 15y ago

lol. At a previous job, I used ActiveMQ. It was definitely a flaky piece of junk.

wslh 15y ago

There is a lot of buzz about open source queue systems. I have tried for example the RabbitMQ with very bad results. I'll rather explain... one interesting concept about queues is contention and this is where RabbitMQ and others are behind. The strange thing is that contention is an old concept from the old mainframes. For example the last time I checked you can't block a produced in RabbitMQ based on the number of messages on the queue.

rabbitmq 15y ago
If you have any questions or problems involving RabbitMQ please email us (info@rabbitmq.com) or post to the mailing list. "one interesting concept about queues is contention and this is where RabbitMQ and others are behind" Can you explain what this means? What kind of contention are you talking about. You say "RabbitMQ and others" - which others? Who implements this feature and what does it look like? "you can't block a produced in RabbitMQ based on the number of messages on the queue" Yes you can. Well - it depends on your use case. RabbitMQ enables you to determine queue length, and supports flow control. But, perhaps you had something else in mind?
- wslh 15y ago
  Please let me know where I am wrong: - RabbitMQ has a memory based flow control based on the total memory you have in the machine. I know I can develop something over RabbitMQ to accomplish what I want (limits based on number of messages) but I prefer to have the support within RabbitMQ. - WebLogic JMS can block producers based on number of messages as a flow control method.
  - rabbitmq 15y ago
    I don't know what WebLogic can do for flow control exactly. Please email rabbitmq-discuss to get a definitive answer and discussion on what you can do with Rabbit, which supports various mechanisms.

anttipoi 15y ago

I've now been involved in two projects where I would have needed a queue with following characteristics: - simple pub-sub, queue-like: m producers, n consumers - decent performance (order of hundreds of requests per second) - high availability for message persistence (I don't want to loose messages) - no strict FIFO needed, just some kind of lesser fairness (newer items should not block older ones from passing through) ActiveMQ with active-backup setup over shared disk mount is the current choice, and the start-up is really slow if the queue has a lot of data. RabbitMQ does not persist messages in HA fashion, so I've ignored it so far. Maybe HornetQ needs some attention. I see a lot of flexibility and feature-richness in the queue landscape and it perplexes me that getting this simple combination of basics right is so difficult.

rbranson 15y ago
RabbitMQ itself doesn't have HA built-in, but warm spare HA is rather simple and well-documented, built on-top of Pacemaker and DRBD. It works quite well.
- rabbitmq 15y ago
  Thank-you. The docs for HA are here: http://www.rabbitmq.com/pacemaker.html http://www.rabbitmq.com/pacemaker.html This also works with Veritas if you use that instead of DRDB. Please note that we are currently QAing a new HA model which is active/active. Watch this space!

rajdavies 15y ago

BTW - David Pollak didn't talk to any of the ActiveMQ team at FuseSource - else we would have helped him out - he didn't speak to any of the execs either. Its a shame he doesn't allow comments on his blog.

ghostganz 15y ago

I had a lot of trouble with the network-of-brokers feature in ActiveMQ a few years ago, which in the end forced us to switch to another solution. In a later, simpler, project I had issues with ActiveMQ locking up after a certain number of messages, which was solved with an upgrade. On a whole, I think the quality of ActiveMQ is lower than other open source projects with similar brand recognition, e.g. other popular Apache projects.

nvarsj 15y ago

As a counter point, I have used ActiveMQ for a pub/sub architecture and it worked great (millions of messages a day, which was benchmarked as only about 10% of potential on a single broker). We did treat it as an API rather than a deployable component though. We wrapped the broker in our own service architecture (you can instantiate a broker like any POJO) and disabled persistency. Queues were used only very lightly. So maybe we dodged a bullet there... :-).

tvaillancourt 15y ago

We use ActiveMQ in production. I couldn't agree more that it is not ready for prime time. We've had a pretty bad experience overall. Man I wish we used Amazon SQS... We have had incidents where it's Master/Slave replication continues on failure when configured to shutdown the master and slave brokers. It doesn't achieve the consistency it lists as a feature, and provides zero visibility of progress. I feel many of the features the project lists are incomplete, and are listed as a grasp at straws to have the bigger list. Due to the poor quality of the replication, we had to invest a tonne of time to implement replication at the block device level with DRBD+GFS2, complicating the system and (benchmarks pending) likely decreasing performance, all to achieve a 'feature' ActiveMQ boasts. I would also challenge a project to have worse, more incomplete documentation. At the same time, I thank the ActiveMQ team for their contributions and ask the community to not turn it's back on this potentially decent solution. The ideas are solid. It isn't production ready, I feel, but that just means it needs some love. Cheers, Tim Vaillancourt