11 ms·

From Kafka to ZeroMQ for real-time log aggregation

197 points by janczukt 10y ago

jvoorhis 10y ago
2015
htn 10y ago
FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.
- janczukt 10y ago
  We need an on-premise and cloud story, so cloud only solutions did not cut it for us.
  - PieterH 10y ago
    The article is a little old. How has the system run since you deployed it? Do you have any interesting figures?
    - janczukt 10y ago
      It continues to run beautifully. Since we rolled it out back in 2015 we had zero issues with real time logging. I have particularly fond memories of the first week after rollout, it felt like vacation. I finally could get some sleep.
  - ZenoArrow 10y ago
    I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do). To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?
    - hashmp 10y ago
      Use 5 zookeepers, on a separate set of servers. Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well for auto DNS registration and auto discovery. Use the yahoo Kafka-manager app to manage the cluster and to see what is going on. Don't use the Kafka default of storing data in /tmp/. Your OS will periodically clean it.
      - Gigablah 10y ago
        > Don't use the Kafka default of storing data in /tmp/. That seems like MySQL level of bad defaults.
      - ZenoArrow 10y ago
        Five zookeepers? Seems like a lot. Why five? Is it hard to keep them active? Thanks for the tips.
        darkr 10y ago
        Zab [the distributed consensus algorithm that powers ZK] shares some similarities with Paxos, and requires a quorum of nodes to be online. If you want highly available ZK, your choices are 3, 5, 7... nodes, for which you can have 1, 2, or 3 nodes offline at any one time. If you have one node fully down on a 3 node cluster, and there is even a tiny network blip or partition (as often happens in cloud environments) then you are down.
      - cbsmith 10y ago
        Five kind of kills performance compared to three, and doesn't map well into AWS, where you generally have 3 or 4 AZ's. I tend to go with three but make sure you've got fully automated responses towards failures.
- mpd 10y ago
  Eh, Kinesis has some pretty significant trade-offs to know about if you are comparing it with Kafka (e.g. data retention time and write latency).
- abritishguy 10y ago
  Kinesis is very poor
siscia 10y ago
Did you consider MQTT? Sound to me a more natural choice.
bdowling 10y ago
But why ZeroMQ and not nanomsg?
- PieterH 10y ago
  See http://hintjens.com/blog:112 http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.
  - raarts 10y ago
    From the blog: "Crazy Idea: Clone nanomsg, move to zeromq organization, relicense as MPL, support ZMTP, only new socket types and expose CZMQ API." Did that ever happen? I still like the idea behind nanomsg.
    - PieterH 10y ago
      It never happened because (IMO, I'm guessing), no-one actually needs nanomsg. I've never really understood the hostility some people had towards e.g. the ZeroMQ protocols, since these RFCs are as plastic and open to contributions as any part of the project. Yet people like to react against things, and this was a large part of nano's reason for existence. In reality, people who need stable working code just take ZeroMQ and that's it. Which is all kind of a shame since it would have been so nice to see a new C engine in the community. We have new engines in C#, Erlang, Java, yet the old core project is still that somewhat clunky C++ engine.
  - kal31dic 10y ago
    With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user. There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it being before. The mailing list is quiet it is true. Somebody just released a Rust version, and version 1.0.0 of nanomsg was indeed released. You can see commit history here: https://github.com/nanomsg/nanomsg/commits/master/src https://github.com/nanomsg/nanomsg/commits/master/src
Nimimi 10y ago
You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.
- oblio 10y ago
  Do you mean this? https://dcos.io/get-started/ https://dcos.io/get-started/ aka DC/OS? From what I can see it doesn't really support database deployments except for ArangoDB and Cassandra.
  - Nimimi 10y ago
    Ok, database was the wrong term. Maybe "things that big data companies use" or something.
  - ryanmaclean 10y ago
    Riak and MySQL are in Universe, for example: https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/M/mysql/0 https://github.com/mesosphere/universe/tree/version-3.x/repo...
- bogomipz 10y ago
  >"DC/IO is quickly becoming the go-to solution for database deployments." It is? Can you provide any evidence supporting this claim? Mesos is mostly used to deploy stateless services.
- mej10 10y ago
  I wish there was more information on how much work it takes to maintain a DC/OS setup. All the marketing makes it out to be the easiest thing in the world.
weitzj 10y ago
Did you look at nsq.io or NATS?
- tapirl 10y ago
  Do you have any experience on NATS? It (with NATS steam) looks great.
- tjholowaychuk 10y ago
  +1 for NSQ, it's not a magic bullet in terms of scalability but you can get quite far. When I was at Segment we were pushing an easy 2-3B messages per day through it, if not more with message "amplification" internally.
TheHydroImpulse 10y ago
FYI, Kafka doesn't need to fetch from disk every time as it caches the logs pretty aggressively, as long as you have enough memory. Running Zk and Kafka on the same nodes is likely not the best thing.
- im_down_w_otp 10y ago
  Why? I would think that, as long as there wasn't massive I/O contention between the two, that co-locating Kafka and Zookeeper on the same machines would mitigate a whole massive class of weird edge cases by removing one of the failure modes; the network boundary between the two critical components. Though for my part I still don't understand why Zookeeper wasn't built as a library to add distributed strongly consistent coordination to software that needs/benefits from it rather than being an external service that needs to be connected to, and thus introduces a gnarly mess of new failure modes that make Zookeeper client behavior extremely critical and often fragile. Something that's more like a "libpaxos/libraft" (e.g. serf for Go-lang or riak_ensemble for Erlang) seems a lot more valuable. /shrug
  - TheHydroImpulse 10y ago
    But co-locating them won't actually remove a class of errors because Zk is not HA. The Kafka brokers need to communicate with the leader in the Zk cluster. If we have K1,Z1 -- K2,Z2 -- K3,Z3 -- and one node goes down, you've now taken down both a broker and a Zk node. Remember, the brokers don't care about connecting to any Zk node, they want the leader. So you aren't gaining any more fault tolerant by co-locating them. If there's a network partition between the leader Zk node and other nodes, the local Kafka broker won't actually be able to do much because the Zk cluster will elect a new leader, on another node, so again, you aren't gaining anything. Moreover, you're now tying the scalability of Kafka with Zk. Zk doesn't scale linearly, so there's only so many nodes you may have in a cluster. Kafka, on the other hand, scales linearly. So if you're colocating them and you have to bump up Kafka, do you still start up Zk for those nodes (but they don't actually join the cluster)? You're now special casing and adding more edge cases.
jpgvm 10y ago
Probably should have been running ZK and Kafka queues separate to CoreOS/container shenanigans. If deployed using the Netflix co-processes both are very durable.
wcdolphin 10y ago
Did you ever try running 5 ZK's in the ensemble? 3 is the absolute minimum to survive a single machine failure. If you are having trouble with availability, it seems natural to increase your safety factor there. I was surprised by the contrasting sense of importance of delivery guarantees in the article. At the start, losing a message was akin to the death of a child. At the end, shrug. Now every single machine failure (or even ømq process restart) failure will lose you log messages stored in memory :(. Glad to hear you found a solution that worked for you though! Would love to hear about difficulties you had with the new system, in particular adding brokers.
- _qc3o 10y ago
  They said availability was "death of a child", not dropping log messages. The trade-off they've made here in terms of being available with some potential loss of visibility is the right one. The system overall is clearly simpler and simpler systems have simpler failure modes and so it is easier to add mitigation components on top that can recover from those failure modes to guarantee higher uptime. I've never heard anyone say managing a production Kafka cluster was easy or simple. Well, anyone who has had to actually maintain such clusters hasn't said it anyway.
  - kod 10y ago
    I've managed a production kafka cluster at my current gig for over two years. It has been easy and trouble free with the exception of one incident, which was ultimately our fault.
  - fauigerzigerk 10y ago
    >They said availability was "death of a child", not dropping log messages. True, but it appears to me that availability problems and dropped log messages often have the same root cause - network issues. So whenever they do have availability issues (and dying babies) they won't be able to investigate properly because log messages are being lost as well. That's obviously a very general observation. It may well be that in their architecture availability issues are mostly caused by something unrelated to networking (e.g. the database).
bachback 10y ago
With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - much more powerful and the sane default. It seems at some point people confused internal networking solutions with the Internet.
- PieterH 10y ago
  If you try to use ZeroMQ to replace TCP with the same semantics, then no, it won't work.
  - bachback 10y ago
    How should ZeroMQ replace TCP? Internet runs on TCP/IP (last time I checked). In private networks many problems of public networks don't appear. I'm aware that organisations use it for their internal systems. As soon as one bridges to the outside world, one is going to hit with a problem.
    - PieterH 10y ago
      ZeroMQ is a messaging layer that runs over numerous network protocols (TCP, UDP, PGM, TIPC,...), as well as inter-process and inter-thread. It provides abstractions (like pub-sub) that work over all these protocols, and it shows the developer the abstraction using a socket-like API. In practice you build services which are internally composed and scaled using threads and processes, and which talk to each other over plain (unencrypted) protocols, e.g. TCP between boxes, IPC between processes on the same box. And when you need to talk to the outside world you build bridges that can speak any protocol you like, such as HTTP, or ZeroMQ's encrypted TCP protocol (CurveZMQ). From the developer's point of view, it's mostly transparent. Obviously you need security infrastructure, e.g. key/certificate management. You can build entire ZeroMQ applications using only encrypted TCP and they will run on public infrastructure. You can develop and test the same apps on a laptop. You might run the same code using IPC on a laptop, and CurveZMQ over the Internet. So though ZeroMQ definitely uses TCP as one of its transport protocols, it's not a replacement for it. Which means that doing things like asking for the IP address of a peer make no sense. What's the IP address of a thread, or another process on the same box? If you need identity information, you should not use the transport protocol for that.
thomaslee 10y ago
I used to be on a team responsible for a single small-ish Kafka cluster (between 6-12 nodes) doing non-trivial throughput on bare metal. Without commenting on whether ZeroMQ is the right alternative: I can understand being scared off. Our hand was forced such that we had to go the other way and understand what was going on in Kafka. The kicker is that Kafka can be rock solid in terms of handling massive throughput and reliability when the wheels are well greased, but there are a lot of largely undocumented lessons to learn along the way RE: configuration and certain surprising behavior that can arise at scale (such as https://issues.apache.org/jira/browse/KAFKA-2063 https://issues.apache.org/jira/browse/KAFKA-2063, which our team ran into maybe a year ago & is only being fixed now). Symptoms of these issues can cause additional knock-on effects with respect to things like leader election (we wound up with a "zombie leader" in our cluster that caused all sorts of bizarre problems) and graceful shutdowns. Add to that the fact the software is still very much under active development (sporadic partition replica drops after an upgrade from 0.8.1 to 0.8.2; we had to apply some small but crucial patches from Uber's fork) & that it needs a certain level of operational maturity to monitor it all ... it's easy to get nervous about what the next "surprise" will be. Having said all that, I'd use Kafka again in a heartbeat for those high volume use cases where reliability matters. Not sure I'd advise others without similar operational experience to do the same for anything mission critical, though -- unless you like stress. That stress is why Confluent is in business. :)
- BrandonBradley 10y ago
  I can attest to 'getting nervous about what the next surprise will be' with Kafka. And I'm only dealing with a single node right now. Kafka and Confluent Platform are very much still works in progress. I had to patch Kafka Connect HDFS connector because a fix I needed was left out of the last release. Be prepared to do something similar with any of Kafka's components.
buster 10y ago
To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. How can you claim to run a serious business and say "babies will die" when you can't be sure to be able to find problems? Because, when will you lose logs? Not in normal operation, but when weird things happen. When networking has a hiccup. When Load on the system is too high, so most likely when many people are using your service. Exactly when shit hits the fan. And you just made the decision that it's ok to drop log messages in such cases? That's not good. I think you should either dive into Kafka/Zookeeper and fix your problems or switch to another logging solution. You should probably just drop that non-sense "streaming and real-time logs" requirement and live with a log delay of a few seconds and build something really stable instead of building something inherently unstable. Honestly, just collecting syslogs on the core vm and sending them to a central server would have been the better solution. Better then looking into fancy real-time, streaming logs on a sunday night because the system is having a breakdown and you can't even be sure that you are not missing essential logs.
- macns 10y ago
  [..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that. It seems that when shit hits the fan for this reason ZMQ really is a more reliable choice because it's more flexible.
  - buster 10y ago
    No. When you can't rely on your collected ZMQ logs and need to "access historical logs" by some other means, why use the ZMQ logs at all? You usually don't know that something was not logged. Also, as he describes in the article, historical logs are scoped out and it is "likely" they they will develop something for those logs in the future. Again it looks like the plan is to use ZMQ and a subscriber to put those logs into logstash. That doesn't solve the problem i mentioned at all. ZMQ may still drop the logs! So, as far as i understand they don't have a plan for reliable logging. Even if they would, they'd have one reliable solution and an unreliable solution. The unreliable ZMQ based approach is probably neat and leads to fancy realtime dashboard stuff, but since it's not a reliable source of information it's not a good solution for operating a system where "babies will die".
k__ 10y ago
I'm a total message queue noob. What are the usecases for them? I used MQTT but only as a message bus.
- macns 10y ago
  You should have a look at http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all It's a great read and describes most scenarios well and easy to understand.
- zo1 10y ago
  From my point of view, the main things behind message queues (Not zMQ specifically) is guaranteed delivery, persistence, multiple-message atomicity, message passing/forwarding, and sometimes guaranteed message ordering. Other than that, all it does is facilitate communication between different actors. Nothing magical/weird about it, just depends on whether or not you've got a nail to hammer with your MQ-hammer.
deleted 10y ago
[deleted]
StreamBright 10y ago
The author correctly points out that he is comparing apples to oranges. Kafka gives you features that certain systems cannot live without, like on disk persistence (saved my life couple of times) and topics. Filtering messages on the client side like ZeroMQ does it not an option in many cases, just think about security. I think Kafka has a long way to go before it can be used as a general message queue (many features are not there yet like visibility timeout for example) but if you can manage Zookeeper and have means to work with it (somebody understands it and knows its quirks) it can provide a reliable platform for distributing a large number of messages with low latency and high throughput, just like it does at LinkedIN.
agentgt 10y ago
I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k messages/s when embarrassingly had debug turned on in production and RabbitMQ did not fail but Elastic Search and sadly Flume did as well (I tried to get rid of flume with a custom Rust AMQP to Elastic Search client but at the time had some bugs with the libraries.. Maybe I will recheck out Mozilla Heka someday). There is this sort of beating of the developer chest with a lot of tech companies.. that hey listen we are ultra important and we are dealing with ridiculously traffic and we need ultra high performance. Please tell/show me these numbers.... Or maybe stop logging crap you don't need to log. Or maybe I'm wrong and we should log absolutely everything and Auth0 made the right choice given their needs (lets assume they have millions of messages a second), I still think I could make a sharded RabbitMQ go pretty far. This goes with other technology as well. You don't need to pick hot glamorous NoSQL when Postgresql or MySQL and a tiny bit of engineering will get the job done just fine particularly when mature solutions give you such many things free out of the box (RabbitMQ gives you a ton of stuff like a cool admin UI and routing that you would have to build in ZeroMQ).
- brightball 10y ago
  Agree with everything you said, just curious is the GT in your name for Georgia Tech?
  - agentgt 10y ago
    Yep... I was the first agent@cc.gatech.edu circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
    - brightball 10y ago
      No apologies needed, lol. I went to Clemson. Just saw it and was curious.
efangs 10y ago
Anyone use collectd + rrd for this purpose? Still trying to understand at what level it's worth to move to something else.
markpapadakis 10y ago
Maybe TANK ( https://github.com/phaistos-networks/TANK https://github.com/phaistos-networks/TANK ) would have been a good alternative on there. No features parity with Kafka, but setting it up is a matter of running one binary and creating a few topics, and it is faster than Kafka for produce/consume operations. (disclosure: I am involved in its development).
asasidh 10y ago
So you used Kafka for something that should have been handled by a MQTT or ZeroMQ in the first place ?
- cbsmith 10y ago
  MQTT is just a protocol, so not sure how that helps. 0MQ doesn't sound like it is the right solution either, but yeah... often you pick the wrong tool and learn something in the process.
wanderr 10y ago
I came up with a very different solution for real time access to logs: tail them to slack. It's not an aggregation solution and doesn't work well if you have chatty logs with nothing to filter on, but if you just want to be notified when things are happening in the logs it's pretty nice and doesn't need any infrastructure. http://wanderr.com/jay/tail-error-logs-to-slack-for-fun-and-profit/2016/05/12/ http://wanderr.com/jay/tail-error-logs-to-slack-for-fun-and-...
- wanderr 10y ago
  why the downvote? the article says "Real-time access to server-side logs is what makes backend development palatable in the era of cloud computing. As a developer you want to be able to get real-time feedback from your server side code deployed to the actual execution environment in the cloud, especially during active development or staging." and this is another solution that provides that.
_halgari 10y ago
ZMQ's default behavior (and in some cases only behavior) of dropping new messages when buffers are full, made it a no-go for my client. We ended up switching away from ZMQ to a more traditional durable queue and ended up saving a ton of code complexity and got a lot of reliability in the process. Having now researched it I can't think of a reason I'd ever use ZMQ again. I'll either use a durable queue when I care about message delivery, or something much more traditional when I don't.
deleted 10y ago
[deleted]
manigandham 10y ago
Why dont all these companies ever just use real enterprise software? There are about a dozen message systems out there that will handle much more than Kafka with minimal or no operational overhead while supporting everything they need.

197 points by janczukt 10y ago

2015

htn 10y ago

FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.

janczukt 10y ago
We need an on-premise and cloud story, so cloud only solutions did not cut it for us.
- PieterH 10y ago
  The article is a little old. How has the system run since you deployed it? Do you have any interesting figures?
  - janczukt 10y ago
    It continues to run beautifully. Since we rolled it out back in 2015 we had zero issues with real time logging. I have particularly fond memories of the first week after rollout, it felt like vacation. I finally could get some sleep.
- ZenoArrow 10y ago
  I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do). To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?
  - hashmp 10y ago
    Use 5 zookeepers, on a separate set of servers. Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well for auto DNS registration and auto discovery. Use the yahoo Kafka-manager app to manage the cluster and to see what is going on. Don't use the Kafka default of storing data in /tmp/. Your OS will periodically clean it.
    - Gigablah 10y ago
      > Don't use the Kafka default of storing data in /tmp/. That seems like MySQL level of bad defaults.
    - ZenoArrow 10y ago
      Five zookeepers? Seems like a lot. Why five? Is it hard to keep them active? Thanks for the tips.
      - darkr 10y ago
        Zab [the distributed consensus algorithm that powers ZK] shares some similarities with Paxos, and requires a quorum of nodes to be online. If you want highly available ZK, your choices are 3, 5, 7... nodes, for which you can have 1, 2, or 3 nodes offline at any one time. If you have one node fully down on a 3 node cluster, and there is even a tiny network blip or partition (as often happens in cloud environments) then you are down.
    - cbsmith 10y ago
      Five kind of kills performance compared to three, and doesn't map well into AWS, where you generally have 3 or 4 AZ's. I tend to go with three but make sure you've got fully automated responses towards failures.
mpd 10y ago
Eh, Kinesis has some pretty significant trade-offs to know about if you are comparing it with Kafka (e.g. data retention time and write latency).
abritishguy 10y ago
Kinesis is very poor

siscia 10y ago

Did you consider MQTT? Sound to me a more natural choice.

bdowling 10y ago

But why ZeroMQ and not nanomsg?

PieterH 10y ago
See http://hintjens.com/blog:112 http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.
- raarts 10y ago
  From the blog: "Crazy Idea: Clone nanomsg, move to zeromq organization, relicense as MPL, support ZMTP, only new socket types and expose CZMQ API." Did that ever happen? I still like the idea behind nanomsg.
  - PieterH 10y ago
    It never happened because (IMO, I'm guessing), no-one actually needs nanomsg. I've never really understood the hostility some people had towards e.g. the ZeroMQ protocols, since these RFCs are as plastic and open to contributions as any part of the project. Yet people like to react against things, and this was a large part of nano's reason for existence. In reality, people who need stable working code just take ZeroMQ and that's it. Which is all kind of a shame since it would have been so nice to see a new C engine in the community. We have new engines in C#, Erlang, Java, yet the old core project is still that somewhat clunky C++ engine.
- kal31dic 10y ago
  With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user. There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it being before. The mailing list is quiet it is true. Somebody just released a Rust version, and version 1.0.0 of nanomsg was indeed released. You can see commit history here: https://github.com/nanomsg/nanomsg/commits/master/src https://github.com/nanomsg/nanomsg/commits/master/src

Nimimi 10y ago

You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.

oblio 10y ago
Do you mean this? https://dcos.io/get-started/ https://dcos.io/get-started/ aka DC/OS? From what I can see it doesn't really support database deployments except for ArangoDB and Cassandra.
- Nimimi 10y ago
  Ok, database was the wrong term. Maybe "things that big data companies use" or something.
- ryanmaclean 10y ago
  Riak and MySQL are in Universe, for example: https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/M/mysql/0 https://github.com/mesosphere/universe/tree/version-3.x/repo...
bogomipz 10y ago
>"DC/IO is quickly becoming the go-to solution for database deployments." It is? Can you provide any evidence supporting this claim? Mesos is mostly used to deploy stateless services.
mej10 10y ago
I wish there was more information on how much work it takes to maintain a DC/OS setup. All the marketing makes it out to be the easiest thing in the world.

weitzj 10y ago

Did you look at nsq.io or NATS?

tapirl 10y ago
Do you have any experience on NATS? It (with NATS steam) looks great.
tjholowaychuk 10y ago
+1 for NSQ, it's not a magic bullet in terms of scalability but you can get quite far. When I was at Segment we were pushing an easy 2-3B messages per day through it, if not more with message "amplification" internally.

TheHydroImpulse 10y ago

FYI, Kafka doesn't need to fetch from disk every time as it caches the logs pretty aggressively, as long as you have enough memory. Running Zk and Kafka on the same nodes is likely not the best thing.

im_down_w_otp 10y ago
Why? I would think that, as long as there wasn't massive I/O contention between the two, that co-locating Kafka and Zookeeper on the same machines would mitigate a whole massive class of weird edge cases by removing one of the failure modes; the network boundary between the two critical components. Though for my part I still don't understand why Zookeeper wasn't built as a library to add distributed strongly consistent coordination to software that needs/benefits from it rather than being an external service that needs to be connected to, and thus introduces a gnarly mess of new failure modes that make Zookeeper client behavior extremely critical and often fragile. Something that's more like a "libpaxos/libraft" (e.g. serf for Go-lang or riak_ensemble for Erlang) seems a lot more valuable. /shrug
- TheHydroImpulse 10y ago
  But co-locating them won't actually remove a class of errors because Zk is not HA. The Kafka brokers need to communicate with the leader in the Zk cluster. If we have K1,Z1 -- K2,Z2 -- K3,Z3 -- and one node goes down, you've now taken down both a broker and a Zk node. Remember, the brokers don't care about connecting to any Zk node, they want the leader. So you aren't gaining any more fault tolerant by co-locating them. If there's a network partition between the leader Zk node and other nodes, the local Kafka broker won't actually be able to do much because the Zk cluster will elect a new leader, on another node, so again, you aren't gaining anything. Moreover, you're now tying the scalability of Kafka with Zk. Zk doesn't scale linearly, so there's only so many nodes you may have in a cluster. Kafka, on the other hand, scales linearly. So if you're colocating them and you have to bump up Kafka, do you still start up Zk for those nodes (but they don't actually join the cluster)? You're now special casing and adding more edge cases.

jpgvm 10y ago

Probably should have been running ZK and Kafka queues separate to CoreOS/container shenanigans. If deployed using the Netflix co-processes both are very durable.

wcdolphin 10y ago

Did you ever try running 5 ZK's in the ensemble? 3 is the absolute minimum to survive a single machine failure. If you are having trouble with availability, it seems natural to increase your safety factor there. I was surprised by the contrasting sense of importance of delivery guarantees in the article. At the start, losing a message was akin to the death of a child. At the end, shrug. Now every single machine failure (or even ømq process restart) failure will lose you log messages stored in memory :(. Glad to hear you found a solution that worked for you though! Would love to hear about difficulties you had with the new system, in particular adding brokers.

_qc3o 10y ago
They said availability was "death of a child", not dropping log messages. The trade-off they've made here in terms of being available with some potential loss of visibility is the right one. The system overall is clearly simpler and simpler systems have simpler failure modes and so it is easier to add mitigation components on top that can recover from those failure modes to guarantee higher uptime. I've never heard anyone say managing a production Kafka cluster was easy or simple. Well, anyone who has had to actually maintain such clusters hasn't said it anyway.
- kod 10y ago
  I've managed a production kafka cluster at my current gig for over two years. It has been easy and trouble free with the exception of one incident, which was ultimately our fault.
- fauigerzigerk 10y ago
  >They said availability was "death of a child", not dropping log messages. True, but it appears to me that availability problems and dropped log messages often have the same root cause - network issues. So whenever they do have availability issues (and dying babies) they won't be able to investigate properly because log messages are being lost as well. That's obviously a very general observation. It may well be that in their architecture availability issues are mostly caused by something unrelated to networking (e.g. the database).

bachback 10y ago

With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - much more powerful and the sane default. It seems at some point people confused internal networking solutions with the Internet.

PieterH 10y ago
If you try to use ZeroMQ to replace TCP with the same semantics, then no, it won't work.
- bachback 10y ago
  How should ZeroMQ replace TCP? Internet runs on TCP/IP (last time I checked). In private networks many problems of public networks don't appear. I'm aware that organisations use it for their internal systems. As soon as one bridges to the outside world, one is going to hit with a problem.
  - PieterH 10y ago
    ZeroMQ is a messaging layer that runs over numerous network protocols (TCP, UDP, PGM, TIPC,...), as well as inter-process and inter-thread. It provides abstractions (like pub-sub) that work over all these protocols, and it shows the developer the abstraction using a socket-like API. In practice you build services which are internally composed and scaled using threads and processes, and which talk to each other over plain (unencrypted) protocols, e.g. TCP between boxes, IPC between processes on the same box. And when you need to talk to the outside world you build bridges that can speak any protocol you like, such as HTTP, or ZeroMQ's encrypted TCP protocol (CurveZMQ). From the developer's point of view, it's mostly transparent. Obviously you need security infrastructure, e.g. key/certificate management. You can build entire ZeroMQ applications using only encrypted TCP and they will run on public infrastructure. You can develop and test the same apps on a laptop. You might run the same code using IPC on a laptop, and CurveZMQ over the Internet. So though ZeroMQ definitely uses TCP as one of its transport protocols, it's not a replacement for it. Which means that doing things like asking for the IP address of a peer make no sense. What's the IP address of a thread, or another process on the same box? If you need identity information, you should not use the transport protocol for that.

thomaslee 10y ago

I used to be on a team responsible for a single small-ish Kafka cluster (between 6-12 nodes) doing non-trivial throughput on bare metal. Without commenting on whether ZeroMQ is the right alternative: I can understand being scared off. Our hand was forced such that we had to go the other way and understand what was going on in Kafka. The kicker is that Kafka can be rock solid in terms of handling massive throughput and reliability when the wheels are well greased, but there are a lot of largely undocumented lessons to learn along the way RE: configuration and certain surprising behavior that can arise at scale (such as https://issues.apache.org/jira/browse/KAFKA-2063 https://issues.apache.org/jira/browse/KAFKA-2063, which our team ran into maybe a year ago & is only being fixed now). Symptoms of these issues can cause additional knock-on effects with respect to things like leader election (we wound up with a "zombie leader" in our cluster that caused all sorts of bizarre problems) and graceful shutdowns. Add to that the fact the software is still very much under active development (sporadic partition replica drops after an upgrade from 0.8.1 to 0.8.2; we had to apply some small but crucial patches from Uber's fork) & that it needs a certain level of operational maturity to monitor it all ... it's easy to get nervous about what the next "surprise" will be. Having said all that, I'd use Kafka again in a heartbeat for those high volume use cases where reliability matters. Not sure I'd advise others without similar operational experience to do the same for anything mission critical, though -- unless you like stress. That stress is why Confluent is in business. :)

BrandonBradley 10y ago
I can attest to 'getting nervous about what the next surprise will be' with Kafka. And I'm only dealing with a single node right now. Kafka and Confluent Platform are very much still works in progress. I had to patch Kafka Connect HDFS connector because a fix I needed was left out of the last release. Be prepared to do something similar with any of Kafka's components.

buster 10y ago

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. How can you claim to run a serious business and say "babies will die" when you can't be sure to be able to find problems? Because, when will you lose logs? Not in normal operation, but when weird things happen. When networking has a hiccup. When Load on the system is too high, so most likely when many people are using your service. Exactly when shit hits the fan. And you just made the decision that it's ok to drop log messages in such cases? That's not good. I think you should either dive into Kafka/Zookeeper and fix your problems or switch to another logging solution. You should probably just drop that non-sense "streaming and real-time logs" requirement and live with a log delay of a few seconds and build something really stable instead of building something inherently unstable. Honestly, just collecting syslogs on the core vm and sending them to a central server would have been the better solution. Better then looking into fancy real-time, streaming logs on a sunday night because the system is having a breakdown and you can't even be sure that you are not missing essential logs.

macns 10y ago
[..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that. It seems that when shit hits the fan for this reason ZMQ really is a more reliable choice because it's more flexible.
- buster 10y ago
  No. When you can't rely on your collected ZMQ logs and need to "access historical logs" by some other means, why use the ZMQ logs at all? You usually don't know that something was not logged. Also, as he describes in the article, historical logs are scoped out and it is "likely" they they will develop something for those logs in the future. Again it looks like the plan is to use ZMQ and a subscriber to put those logs into logstash. That doesn't solve the problem i mentioned at all. ZMQ may still drop the logs! So, as far as i understand they don't have a plan for reliable logging. Even if they would, they'd have one reliable solution and an unreliable solution. The unreliable ZMQ based approach is probably neat and leads to fancy realtime dashboard stuff, but since it's not a reliable source of information it's not a good solution for operating a system where "babies will die".

k__ 10y ago

I'm a total message queue noob. What are the usecases for them? I used MQTT but only as a message bus.

macns 10y ago
You should have a look at http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all It's a great read and describes most scenarios well and easy to understand.
zo1 10y ago
From my point of view, the main things behind message queues (Not zMQ specifically) is guaranteed delivery, persistence, multiple-message atomicity, message passing/forwarding, and sometimes guaranteed message ordering. Other than that, all it does is facilitate communication between different actors. Nothing magical/weird about it, just depends on whether or not you've got a nail to hammer with your MQ-hammer.

deleted 10y ago

[deleted]

StreamBright 10y ago

The author correctly points out that he is comparing apples to oranges. Kafka gives you features that certain systems cannot live without, like on disk persistence (saved my life couple of times) and topics. Filtering messages on the client side like ZeroMQ does it not an option in many cases, just think about security. I think Kafka has a long way to go before it can be used as a general message queue (many features are not there yet like visibility timeout for example) but if you can manage Zookeeper and have means to work with it (somebody understands it and knows its quirks) it can provide a reliable platform for distributing a large number of messages with low latency and high throughput, just like it does at LinkedIN.

agentgt 10y ago

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k messages/s when embarrassingly had debug turned on in production and RabbitMQ did not fail but Elastic Search and sadly Flume did as well (I tried to get rid of flume with a custom Rust AMQP to Elastic Search client but at the time had some bugs with the libraries.. Maybe I will recheck out Mozilla Heka someday). There is this sort of beating of the developer chest with a lot of tech companies.. that hey listen we are ultra important and we are dealing with ridiculously traffic and we need ultra high performance. Please tell/show me these numbers.... Or maybe stop logging crap you don't need to log. Or maybe I'm wrong and we should log absolutely everything and Auth0 made the right choice given their needs (lets assume they have millions of messages a second), I still think I could make a sharded RabbitMQ go pretty far. This goes with other technology as well. You don't need to pick hot glamorous NoSQL when Postgresql or MySQL and a tiny bit of engineering will get the job done just fine particularly when mature solutions give you such many things free out of the box (RabbitMQ gives you a ton of stuff like a cool admin UI and routing that you would have to build in ZeroMQ).

brightball 10y ago
Agree with everything you said, just curious is the GT in your name for Georgia Tech?
- agentgt 10y ago
  Yep... I was the first agent@cc.gatech.edu circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
  - brightball 10y ago
    No apologies needed, lol. I went to Clemson. Just saw it and was curious.

efangs 10y ago

Anyone use collectd + rrd for this purpose? Still trying to understand at what level it's worth to move to something else.

markpapadakis 10y ago

Maybe TANK ( https://github.com/phaistos-networks/TANK https://github.com/phaistos-networks/TANK ) would have been a good alternative on there. No features parity with Kafka, but setting it up is a matter of running one binary and creating a few topics, and it is faster than Kafka for produce/consume operations. (disclosure: I am involved in its development).

asasidh 10y ago

So you used Kafka for something that should have been handled by a MQTT or ZeroMQ in the first place ?

cbsmith 10y ago
MQTT is just a protocol, so not sure how that helps. 0MQ doesn't sound like it is the right solution either, but yeah... often you pick the wrong tool and learn something in the process.

wanderr 10y ago

I came up with a very different solution for real time access to logs: tail them to slack. It's not an aggregation solution and doesn't work well if you have chatty logs with nothing to filter on, but if you just want to be notified when things are happening in the logs it's pretty nice and doesn't need any infrastructure. http://wanderr.com/jay/tail-error-logs-to-slack-for-fun-and-profit/2016/05/12/ http://wanderr.com/jay/tail-error-logs-to-slack-for-fun-and-...

wanderr 10y ago
why the downvote? the article says "Real-time access to server-side logs is what makes backend development palatable in the era of cloud computing. As a developer you want to be able to get real-time feedback from your server side code deployed to the actual execution environment in the cloud, especially during active development or staging." and this is another solution that provides that.

_halgari 10y ago

ZMQ's default behavior (and in some cases only behavior) of dropping new messages when buffers are full, made it a no-go for my client. We ended up switching away from ZMQ to a more traditional durable queue and ended up saving a ton of code complexity and got a lot of reliability in the process. Having now researched it I can't think of a reason I'd ever use ZMQ again. I'll either use a durable queue when I care about message delivery, or something much more traditional when I don't.

deleted 10y ago

[deleted]

manigandham 10y ago

Why dont all these companies ever just use real enterprise software? There are about a dozen message systems out there that will handle much more than Kafka with minimal or no operational overhead while supporting everything they need.