3 ms·
> Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place? Very few people roll out of bed and
by josephg 6y ago
> Why would you want to disagree with Kafka being the default option? What could you reasonably advocate for in its place?
Very few people roll out of bed and think "my application needs a high performance distributed event stream". For every problem that can be solved with kafka, there are other solutions that don't involve kafka. Many of the alternate solutions don't involve event streams at all. Are they better? I don't know. That question is unknowable without looking into the problems in particular, and looking at what other technology is out there. And I say this with a lot of fondness for kafka. I really like it. I just think there are always technical tradeoffs to make.
> any more than MongoDB competes with PostgreSQL
MongoDB and Postgres do compete with each other though! They work differently - so part of the competition isn't around features; its around how your people think about your data. "Think of your data in a way that makes sense for mongodb!" say mongodb sales reps. "Think of your problem as a message queue" say fans of rabbitmq. They aren't wrong in the general case. But there are lots of problems where I could use kafka, or I could use redis, or I could use a naked postgres instance. You need a much more complex heuristic to decide what technology to adopt than "Should I use kafka?" -> "Yes."[1]
And I say all that as a big fan of kafka. I think its API is delightful and just today I was plagerizing its API design to sort out backpressure in a system I'm building.
> What could you reasonably advocate for in its place?
I haven't used it but by all reports RedPanda is lovely: https://vectorized.io/ https://vectorized.io/
[1] Though I'm convinced the answer to "Should I use mongodb" is always "No."
- jpgvm 6y agoI think we are agreeing on a different point than I was making. Kafka isn't the solution to all problems, like I noted with WhatsApp not being the best fit. Though you can shoehorn it into the same problem space, this is what LINE Messenger did which is of equal scale to WhatsApp and runs entirely on Kafka. So I completely agree Kafka isn't the solution to all A needs to talk to B problems, even when Kafka might be able to do the job. I think what was perhaps lost is that I am not talking about adding Kafka to an arbitrary application but rather operating a Big Data pipeline - i.e data from applications, devices, sensors, whatever and transporting it to data lake, warehouse and online stores like Druid. While you can use Kafka in other contexts this is it's original purpose and main use case and where I don't think any other piece of software replaces it right now. Pulsar is on the way there but not quite ready yet. Redpanda is just Kafka without being the real thing - maybe you save some CPU, maybe you hit an incompatibility, really it's not worth the effort because Kafka in 99% of use cases is network bound as it's already efficient enough. (Not always true, there are some CPU bound cases with tons of producers but they are rare) If you chose anything else you are signing up for writing all of these integrations yourself and that just doesn't make sense. If you pick Kafka you get all the hookups for free and you only need to write custom code for your existing custom applications. If say you were to want something more "lightweight" sure you could use Redis PUBSUB but now you are writing code that dumps from Redis PUBSUB into S3 for archival instead of just pulling something like Secor or Kafka Connect and being done with it. Perhaps I should have been more clear that I think it's only in this specific domain/use case that I think it's completely dominant.
- agallego 6y agoredpanda is not just a performance (latency) gain. The main thing people pay us money for is not performance, it is operational simplicity - no zk, no jvm, nothing to tune (https://vectorized.io/docs/autotune https://vectorized.io/docs/autotune) . one binary you can `scp` around a few servers is really quite easy to work with. A different example of a system that can start small is Joe Hellertein's research (https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf) the point is that operational simplicity, often trumps performance for most ppl. In fact, that's what the parent argument suggests redpanda for, the parent comment was not about performance.
- jpgvm 6y agoI get that but often times you need or want ZK anyway. If you are running Druid, Pinot, Hadoop or anything else Big Data they generally have a ZK dependency. If you need service discovery, consensus or config storage ZK is generally great for all of those usecases too. I think if you want a completely standalone distributed log for an application then it could be a good fit but it's not going to replace Kafka as the goto for Big Data pipelines because it's advantages aren't worth it in this context. (i.e you are already running ZK, you are throughput and not latency bound, rest of the ecosystem is JVM anyway, etc)