3 ms·

I think we are agreeing on a different point than I was making. Kafka isn't the solution to all problems, like I noted with WhatsApp not being the best fit. Th

by jpgvm 6y ago

I think we are agreeing on a different point than I was making. Kafka isn't the solution to all problems, like I noted with WhatsApp not being the best fit. Though you can shoehorn it into the same problem space, this is what LINE Messenger did which is of equal scale to WhatsApp and runs entirely on Kafka. So I completely agree Kafka isn't the solution to all A needs to talk to B problems, even when Kafka might be able to do the job. I think what was perhaps lost is that I am not talking about adding Kafka to an arbitrary application but rather operating a Big Data pipeline - i.e data from applications, devices, sensors, whatever and transporting it to data lake, warehouse and online stores like Druid. While you can use Kafka in other contexts this is it's original purpose and main use case and where I don't think any other piece of software replaces it right now. Pulsar is on the way there but not quite ready yet. Redpanda is just Kafka without being the real thing - maybe you save some CPU, maybe you hit an incompatibility, really it's not worth the effort because Kafka in 99% of use cases is network bound as it's already efficient enough. (Not always true, there are some CPU bound cases with tons of producers but they are rare) If you chose anything else you are signing up for writing all of these integrations yourself and that just doesn't make sense. If you pick Kafka you get all the hookups for free and you only need to write custom code for your existing custom applications. If say you were to want something more "lightweight" sure you could use Redis PUBSUB but now you are writing code that dumps from Redis PUBSUB into S3 for archival instead of just pulling something like Secor or Kafka Connect and being done with it. Perhaps I should have been more clear that I think it's only in this specific domain/use case that I think it's completely dominant.

agallego 6y ago
redpanda is not just a performance (latency) gain. The main thing people pay us money for is not performance, it is operational simplicity - no zk, no jvm, nothing to tune (https://vectorized.io/docs/autotune https://vectorized.io/docs/autotune) . one binary you can `scp` around a few servers is really quite easy to work with. A different example of a system that can start small is Joe Hellertein's research (https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf) the point is that operational simplicity, often trumps performance for most ppl. In fact, that's what the parent argument suggests redpanda for, the parent comment was not about performance.
- jpgvm 6y ago
  I get that but often times you need or want ZK anyway. If you are running Druid, Pinot, Hadoop or anything else Big Data they generally have a ZK dependency. If you need service discovery, consensus or config storage ZK is generally great for all of those usecases too. I think if you want a completely standalone distributed log for an application then it could be a good fit but it's not going to replace Kafka as the goto for Big Data pipelines because it's advantages aren't worth it in this context. (i.e you are already running ZK, you are throughput and not latency bound, rest of the ecosystem is JVM anyway, etc)

by jpgvm 6y ago

agallego 6y ago

redpanda is not just a performance (latency) gain. The main thing people pay us money for is not performance, it is operational simplicity - no zk, no jvm, nothing to tune (https://vectorized.io/docs/autotune https://vectorized.io/docs/autotune) . one binary you can `scp` around a few servers is really quite easy to work with. A different example of a system that can start small is Joe Hellertein's research (https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf https://dsf.berkeley.edu/jmh/papers/anna_ieee18.pdf) the point is that operational simplicity, often trumps performance for most ppl. In fact, that's what the parent argument suggests redpanda for, the parent comment was not about performance.

jpgvm 6y ago
I get that but often times you need or want ZK anyway. If you are running Druid, Pinot, Hadoop or anything else Big Data they generally have a ZK dependency. If you need service discovery, consensus or config storage ZK is generally great for all of those usecases too. I think if you want a completely standalone distributed log for an application then it could be a good fit but it's not going to replace Kafka as the goto for Big Data pipelines because it's advantages aren't worth it in this context. (i.e you are already running ZK, you are throughput and not latency bound, rest of the ecosystem is JVM anyway, etc)