4 ms·
Great point about Twitter never explaining why it didn't just use one of the many awesome open source message queues already in existence.
by grandalf 18y ago
Great point about Twitter never explaining why it didn't just use one of the many awesome open source message queues already in existence.
- simonw 18y agoHe suggests the following message queues: http://www.rabbitmq.com/ http://www.rabbitmq.com/ - first version 8th February 2007, but the first version not to have "alpha" or "beta" status was 1.5.1 released 21st January 2009 http://memcachedb.org/memcacheq/ http://memcachedb.org/memcacheq/ - version 0.1.1 released 26th November 2008 http://www.ejabberd.im/ http://www.ejabberd.im/ - not really a message queue http://xph.us/software/beanstalkd/ http://xph.us/software/beanstalkd/ - first public release 11th December 2007, hit 1.0 28th May 2008 http://activemq.apache.org/ http://activemq.apache.org/ - not sure when it was first released but the mailing list goes back to December 2005 The first public release of Starling (Twitter's first custom message queue) was 10th January 2008. Presumably they had it running internally for a while before they released it. From this, we can see that when they built their own pretty much the only realistic open source option was ActiveMQ, which can hardly be described as a light-weight solution (not to mention it still doesn't have a stellar reputation under high loads). When the alternatives aren't rock solid yet, rolling your own (where at least you understand all of the code and how it works) seems like a perfectly practical alternative.
- johnbender 18y agoYes but what's the rationale for doing it now in scala?
- simonw 18y agoAt a guess, a few reasons. Firstly, they had everything else written against a message queue with particular behaviour - so better to upgrade that queue than switch to a completely new one and have to rewrite everything that interacts with it. Secondly, after running a custom message queue for well over a year they know EXACTLY what they need from one, so writing their own still makes sense. Thirdly, if they're going to start moving other core bits of Twitter infrastructure to Scala it makes sense to try it out with an important piece of the puzzle that they thoroughly understand first. And finally, Twitter's core competency is delivering messages. As such, it's really not so extreme to use their own software to do that - they work at a high enough scale that they need to be experts in whatever solution they are using. As Douglas Crockford once said, "The good thing about reinventing the wheel is that you can get a round one.". The fact that Twitter's reliability over the past 6-12 months has been a huge improvement, despite the enormous growth the service has seen (it's mentioned in the mainstream media all the time) would suggest that their decision to roll their own message queue paid off.
- tptacek 18y agoIt's also worth noting that the one queue Arcieri proposed might have been interface-compatibile --- MemcacheQ --- is a single-developer side project written in C. Arcieri is "completely confused" by the fact that that Twitter didn't adopt this as the core of their service.
- NealR 18y agoThis is a straw-man argument. MemcacheQ is a straightforward mash-up of two very stable software stacks.. memcached and BerkeleyDB. The lines of code to accomplish it are trivial. So what.
- evgen 18y agoWhile it is possible that Starling had been running internally before its release, this does not excuse overlooking rabbitmq. A software package that had proven itself in real-world scaling and been designed by people with real experience in the problem domain (c.f. the financial services world) is going to be much better at "alpha" or "beta" quality than Starling is going to be even after the twitter devs hammer at it for a couple of years. The twitter devs were starting from scratch, writing something that other people out there actually had some experience with, and decided to not take an existing solution and fix/adapt it to their needs.
- simonw 18y agoRabbitMQ was less than a year old, and significantly more fully featured than what Twitter needed. Fixing bugs in that would be a whole lot harder than fixing bugs in 1500 lines of code they wrote in a language they knew.
- tptacek 18y agoI take issue with the idea that the financial services world has real experience in Twitter's problem domain. My experience with the financial services world is significant technically, but casual in a career sense. That said: I think hi-fi devs make lots of stupid decisions in the name of performance. In the few cases where their actual outcomes match up to their posturing, it's because their code is obsessively cobbled around one specific use case they've been working on since 1989. Have you ever read an order management system, or looked at Tibco Rendezvous on the wire? Most of the hi-fi companies adopting MQ are built around straight AMQ, and bare-metal performance was out the window long before they bolted their crappy WebSphere app onto it. What these companies are looking for is predictability, not performance, and their problem sets are much simpler and most stable than Twitter's.
- intranation 18y agoI believe they call that the "not invented here" phenomenon.
- samt 18y agoHaving evaluated oss message queues for high volume web services I can see exactly how twitter might make this decision. - activemq: high resource usage, not super stable at high loads - rabbitmq: messages must ALL fit into memory. Your message queue should be a "rescue me" button - if all your dbs go down, just queue crap to disk. You do not want to worry about memory overflow in that situation. - memcacheq: super stable, low resource consumption but fixed message size (padding for smaller messages). We ended up using memcacheq but it's hardly what you would call full featured, so we'll have to do prioritization, etc in our code.
- tptacek 18y agoAwesome comment, thank you.
- mechanical_fish 18y agoI can't read the original article right now. So I must poke around in the darkness when trying to respond. Sorry if I'm being unfair. But it sounds like we're once again enmeshed in the never-ending process of second-guessing Twitter. Let me offer a pertinent quote, from the Hugo-winning novel The Vor Game: [You did] a right thing. Perhaps not the best of all possible right things. Three days from now you may think of a cleverer tactic, but you were the man on the ground at the time. I try not to second-guess my field commanders. Twitter solved their problem. It is unlikely that they did so as elegantly as possible, and it's quite possible that their logic at the time will be of no use to the rest of us at all, because it was based on incomplete information about a rapidly-changing software universe.
- jeremyw 18y agoWithout second guessing them, it remains a fun curiosity that at their size and failure rate they continue to reject vendor expertise in this space. If Starling and Kestrel reinvent little, what can we learn about this counter intuitive behavior, especially re management and investors?