7 ms·

Ask YC: Why the excitement over XMPP on the backend?

• jfarmer 18y ago
  Nope.
• pelle 18y ago
  Almost none of the current generation of large scale web applications including twitter are doing anything remotely new. The investment banking world has as you say had to deal with huge feeds as well as mashups from multiple sources. http://www.zeromq.org/ http://www.zeromq.org/ for example is coming out of the financial world and looks like it's going to be the nginx of amqp systems. It looks like it's way faster and "scalable" than creating something over xmpp.
  • newton 18y ago
    Also check out this Erlang/OTP AMQP server: http://www.rabbitmq.com http://www.rabbitmq.com AMQP is very interesting. Check it out if you are building a distributed system or if you have a requirement for reliable asynchronous messaging. When I last looked into it about a year ago, the implementations (all two of them that I found - apache qpid and rabbitmq) were still a little rough but it looks like a lot has happened. It's wierd how few people in the web world are aware of asynchronous messaging.
• bjclark 18y ago
  "which is a stateful chat protocol" You're missing that it's not stateful chat protocol, it's stateful xml protocol (that has, so far, been use mostly for chat). People are excited that it's fast, stable (some of the implementations anyway), and they don't have to reinvent the wheel.
  • puppetsock 18y ago
    "stateful xml protocol" makes as much sense as "stateful ASCII protocol" or "stateful bits protocol".
    • neilc 18y ago
      "Stateful XML protocol" makes complete sense, it just isn't very informative. It describes a protocol that is both stateful and XML-based. Those are independent attributes of a protocol; obviously, you can have both stateful non-XML-based protocols and XML-based stateless protocols, but I don't see why the OP's statement is nonsensical.
    • bjclark 18y ago
      Totally agreed. That doesn't negate anything I just said. It's stateful. It's XML. It's a stateful XML protocol (and it happens to also have presence, which I left out, but does make some people's knickers wet).
• eugenejen 18y ago
  No. You didn't. But the context is different. XMPP as MQ mechanism may work better in Internet scale distributed system. Just like HTTP works better as a client server mechanism on Internet scale distributed system than other protocols before it. I worked in a startup called PubSub from 2003 to 2006. We in fact used XMPP in 2004 internally to relay our error message log of our cluster of machines in real time to our office's archive server. It was an experiment but it seems pretty useful. We called it as fuselet before in our company.
  • jawngee 18y ago
    We use XMPP for that now. Any major exceptions, errors or site activity across our farm is pumped to a monitoring app via XMPP. I can also control the cluster as well.
  • briansmith 18y ago
    "XMPP as MQ mechanism may work better in Internet scale distributed system." Why do you think so? What advantages does XMPP have over other MQ technologies at "internet scale"?
    • ezmobius 18y ago
      XMPP pierces NAT for one reason. This means you can have XMPP clients behind many more firewalls then you can with amqp. Also xmpp libraries are available in almost every language and platform and traditional MQ's do not have presence or federation(ie horizontal scalability). It's not about using XMPP as a message queue per say. It's about using xmpp as a p2p messaging system for machine to machine communication as well as integration with chat.
    • eugenejen 18y ago
      Is there any existing MQ system that is decentralized? I said "may works better" because it is a conjecture limited by my ignorant mind. Internet is essentially decentralized. So if any MQ system can be modified to worked in decentralized environment. Then they may work better than XMPP. I believe the performance of throughput of those system may be higher than XMPP. On the other hand. If I only have an internal cluster, I will choose to use existing MQ system instead of XMPP. I may want higher throughput, and I don't care authentication of endpoints because they are all trusted. Why do I need all the extra stuff that XMPP demands? Regarding the case I mentioned in PubSub: It was just an idea. In fact, I was the one who mentioned the idea then. And our CTO and VP of Eng thought it was an interesting idea because in this case, we just wanted to see how does it work out and did a test implementation. And it was pretty flexible for us to use it. Of course XMPP is not the first MQ system on Internet. SMTP is the first one, right? In SMTP, we don't know anything of endpoints. We assume senders and recipients exist and are taken care of by SMTP servers. I think in XMPP, because the original application is chatting, then it adds an new protocol to notify "presence" of endpoints. Edit: I think when I said "internet scale", I actually meant "a system of billion nodes that are loosely coupled together"
• wmf 18y ago
  The MQ people don't seem to be explaining their value to the Web 2.0 kids. When I read the ZeroMQ site the other day I kept thinking "but what is it used for?"
  • newton 18y ago
    It probably doesn't help that books on the subject have desperately unhip titles like "Enterprise Integration Patterns". The introduction to that book is actually a great essay on when and how to use asynchronous messaging, and it is online here: http://www.enterpriseintegrationpatterns.com/Introduction.html http://www.enterpriseintegrationpatterns.com/Introduction.ht...
• bct 18y ago
  Why not? It's extensible, it builds on tech people are already familiar with, and it's fast enough for most purposes. It's also got lots of good libraries for many languages, and neat tricks like bridges between it and the web browser.
• neilk 18y ago
  You're missing that they don't intend it as an RPC or REST interaction. It's to "stream" updates out of services, that can have multiple listeners. This is already happening, but in isolated cases. Behind the scenes, Six Apart streams all the blog posts on its services directly to Google Blog search. There are quite a few well-known services that get more API traffic (search engines, etc, pinging for updates) than actual human traffic. So something like this is ultimately necessary, whether it's XMPP or something similar. I think they are aware of message queues, didn't they allude to them in the talk? Anyway they billed their talk as two guys who build web services making a tentative foray into the message-queue world, so if you think you can help, send feedback.
  • puppetsock 18y ago
    The OP understands that XMPP provides pub/sub message queues. He's asking what XMPP provides over the other MQs.
    • melo 18y ago
      Federation. I read a bit over ActiveMQ, RabbitMQ, and the recent and lovely ZeroMQ. As far as I could find, none of them provide federation on a Internet scale. Specially Open Federation using TLS. I'm a XMPP developer for 6 or 7 years now, and if someone thinks that XMPP is going to replace MQ systems, they are delusional. But for Web integration, XMPP beats the crap out of current MQ systems because of Federation. Sure, you can get 2.5M messages/sec on ZeroMQ. So what? I've seen personally a XCP XMPP server doing 1M messages per second also. But for internet scale stuff, even slower servers like OpenFire would work just fine for the current loads they are seeing. It also helps that for scripting languages commonly used (like Perl, Ruby, Python), XMPP bindings are commonly available, while AMQP are still rare. Please please prove me wrong :). My recommendation for startups? Use a MQ system internally from day one to develop your solution, but use XMPP to talk to the outside world. RabbitMQ (and I'm suggesting it based only on what I read, I don't use it myself, I use another one) is the best placed MQ system right now, because it offers the bridge between local MQ system and XMPP world. Best regards,
• briansmith 18y ago
  Message queuing is old, boring, and well understood. The same kinds of people that are trying to convince us that REST is a huge improvement over WS-<star> (despite the fact that once-RESTful protocols are becoming increasingly similar to WS-<star>) are trying to convince us that XMPP is to message queuing as REST is to WS-<star>. More cynically, XMPP vendors generally give everything they have away in hopes that somebody with money will buy support for it. Since customers of message queuing products are people with money, it is only natural for people to hype XMPP as the "worse is better", cheaper alternative to "enterprisy" MQ products in hopes of having some money fall into their pockets. Less cynically, the people hyping XMPP understand it very well from their personal experiences with chat systems and "chatbot" like applications including Twitter. It is easy to say "hey, if a computer can IM me, I bet a computer could IM another computer." It is a little tricky to abstract that into "hey, that is the same thing as that old, boring message queing stuff," and anybody that gets that far realizes it's not that exciting to blog about.
  • bct 18y ago
    > once-RESTful protocols are becoming increasingly similar to WS-* How so?
    • briansmith 18y ago
      It is pretty common for a RESTful protocol to end up with batch-fetching and batch-updating operations. Once you add those operations, you've already violated the "uniform interface" constraint and removed all the benefits (primarily caching, authentication, authorization, and parallelism) that derive from the uniform interface constraint. Once you have batch-fetch and batch-update, those two operations can generally do everything that you usually did via individual PUT/POST/DELETE operations. So, why do you need the separate PUT/POST/DELETE operations at all? It is just more surface area for you to maintain. Now, maybe you want to combine your batch-update and batch-fetch operations into a single "batch-update-and-fetch" POST operation. Well, that is exactly what WS-* does. On the flip side, sometimes instead of batching you want to fetch or update only parts of individual resources via some extension to partial GET requests and or a PATCH method. There is currently no uniform interface for those so, just like in the case of batching, you lose all the uniform-interface benefits. Now, let's say you want to combine your partial-resource operations with batch operations over multiple resources to reduce the number of requests you receive and/or the reduce the aggregate size of requests, so you add the ability to do patching and partial-fetches to your batch-update-and-fetch mechanism. Now take a step back. Doesn't your batch-update-and-fetch-with-patch-and-partial-fetch look a lot like SOAP with WS-ResourceTransfer? It will probably look a lot like SOAP; not even the newish SOAP 1.2 that actually is somewhat HTTP cache friendly, but old school RPC-only SOAP + WS-ResourceTransfer. Look at Google's GData and OpenSocial's RESTful API for examples. And, that is totally ignoring anything to do with transactions.
• aconbere 18y ago
  I'm not sure this is an either or proposition. I haven't seen many message queue protocols with tools for identity, user authentication, federation, etc. I'm also pretty sure that xmpp doesn't even begin to replace message queuing on the backend, but when you look at what folks like EngineYard are doing with it (basically distributing messages), it seems like the value for them was more along the lines of 1) easy ui integration, 2) well understood problems and tools by developers, 3) well understood servers by admins. More on the proposal of mixing the two, the rabbitMQ folks have done some interesting work integrating with xmpp. (See their xmpp transport (http://www.lshift.net/blog/2008/07/01/rabbitmq-xmpp-gateway-released http://www.lshift.net/blog/2008/07/01/rabbitmq-xmpp-gateway-...), and rabbiter (http://github.com/tonyg/rabbiter/tree/master http://github.com/tonyg/rabbiter/tree/master). So there we have an interesting mixture of the two technologies, you get the insanely fast message routing and throughput of RabbitMQ while leveraging the tools that xmpp brings with it (users, federation, etc.)
  • aaronblohowiak 18y ago
    The merging of xmpp and rabbit mq is fascinating -- i've bookmarked this and will follow-up on it. I'm reading through the documentation now, but from what I can see this is great tech. (mod parent up!)
• deleted 18y ago
  [deleted]
• aaronblohowiak 18y ago
  you have a categorical error when you compare ActiveMQ to AMQP; they exist at different layers of the MQ stack (AMQP is a transport protocol.) http://activemq.apache.org/how-does-activemq-compare-to-amqp.html http://activemq.apache.org/how-does-activemq-compare-to-amqp...
  • gstar 18y ago
    Noted, but my intention wasn't to list products rather to name some technologies that are available and open for MQ.
• joshu 18y ago
  Nobody's proposing using it for RPC. They're suggesting it as a replacement for polling.