7 ms·

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ prob

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

oulipo2 1y ago
Have you tried Redpanda?
majormajor 1y ago
If you are using this sort of redis queue (https://redis.io/glossary/redis-queue/ https://redis.io/glossary/redis-queue/) with PUSH/POP vs fan-out you're working on a very different sort of problem than what Kafka is built for. Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java. However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks. But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).
njitbew 1y ago
> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
- slipperydippery 1y ago
  Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years. Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.
  - MathMonkeyMan 1y ago
    The JVM eats a chunk of memory in order to make its garbage collector more efficient. Think of it like Linux's page cache. I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations. The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.
  - bdangubic 1y ago
    you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :) jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory
    - js4ever 1y ago
      [flagged]
      - p2detar 1y ago
        Is this an AI-generated answer? Most of these are not even true, although I still would prefer Go for micro-services. I'll address just a bunch and to be clear - I'm not even a big Java fan. - Quarkus with GraalVM compiles your Java app to native code. There is no JIT or warm up, memory footprint is also low. By the way, the JVM Hotspot JIT can actually make your Java app faster than your Go or Rust app in many cases [citation needed] exactly due to the hot path optimizations it does. - GC tuning - I don't even know who does this. Maybe Netflix or some trading shops? Almost no one does this nowadays and with the new JVM ZGC [0] coming up, nobody would need to. > You can’t ship a minimal standalone binary without pulling in a JVM. - You'd need JRE actually, e.g., 27 MB .MSI for Windows. That's probably the easiest thing to install today and if you do this via your package manager, you also get regular security fixes. Build tools like Gradle generate a fully ready-to-execute directory structure for your app. If you got the JRE on your system, it will run. > Dependency management and classpath conflicts historically plagued Java The keyword here is "historically". Please try Maven or Gradle today and enjoy the modern dependency management. It just works. I won't delve into Java 9 modules, but it's been ages since I last saw a class path issue. > J2EE Is someone still using this? It is super easy writing a web app with Java+Javalin for example. The Java library and frameworks ecosystem is super rich. > “Write once, run anywhere” costs: The abstraction layers that make Java portable also add runtime weight and overhead. Like I wrote above, the HotSpot JIT is actually doing the heavy lifting for your in real time. These claims are baseless without pointing to what "overhead" is meant in practice. --- 0 - https://inside.java/2023/11/28/gen-zgc-explainer/ https://inside.java/2023/11/28/gen-zgc-explainer/ or https://www.youtube.com/watch?v=dSLe6G3_JmE https://www.youtube.com/watch?v=dSLe6G3_JmE
cortesoft 1y ago
It doesn't have to be 'hyper-scaled' to be needed, unless we have widely different definitions of hyper scale. Access logs from a few thousand servers with medium traffic will push you past any single instance service, and Kafka works great for that workload.
rvz 1y ago
> Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play. Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today. Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.
- AtlasBarfed 1y ago
  I thought Kafka ditched the zookeeper
  - mdaniel 1y ago
    I wish; KIP-500[1] was the "let's use native Raft" implementation but I have never once in my life seen anyone using Kafka in KIP-500 mode 1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+replace+zookeeper+with+a+self-managed+metadata+quorum https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...
    - enether 1y ago
      Confluent has shared they've migrated thousands of Kafka clusters (their whole cloud fleet) to KIP-500 https://www.confluent.io/blog/zookeeper-to-kraft-with-confluent-kubernetes/ https://www.confluent.io/blog/zookeeper-to-kraft-with-conflu...
    - 62951413 1y ago
      Very few people want to re-create their production AWS MSK cluster from scratch. And that's the only way currently. MSK can usually upgrade Kafka brokers with minor performance degradation but not for this particular change.
closeparen 1y ago
Publishing an event to Kafka puts it “out there” in a way that guarantees it won’t be lost and allows any number of interested consumers, including the data warehouse, to deal with it at their leisure (subject to retention period which is typically like 72h). For us, your Kafka topics and their schemas are as much a part of your API as your gRPC IDLs. Something like Redis or 0MQ feels more appropriate for internal coordination between instances of the same service, or at least a producer that has a specific consumer in mind.
haddr 1y ago
Couldn’t disagree more… if you go the ZMQ you are left alone handling many things you get in Kafka for free. If you have any sort of big data problems then good luck. You are going to reinvent the wheel.
gf000 1y ago
If you have this stance to technology ("feels bloated", meme-level knowledge on Java), then I really hope that you are not responsible for technical decisions.

oulipo2 1y ago

Have you tried Redpanda?

majormajor 1y ago

If you are using this sort of redis queue (https://redis.io/glossary/redis-queue/ https://redis.io/glossary/redis-queue/) with PUSH/POP vs fan-out you're working on a very different sort of problem than what Kafka is built for. Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java. However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks. But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).

njitbew 1y ago

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

slipperydippery 1y ago
Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years. Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.
- MathMonkeyMan 1y ago
  The JVM eats a chunk of memory in order to make its garbage collector more efficient. Think of it like Linux's page cache. I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations. The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.
- bdangubic 1y ago
  you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :) jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory
  - js4ever 1y ago
    [flagged]
    - p2detar 1y ago
      Is this an AI-generated answer? Most of these are not even true, although I still would prefer Go for micro-services. I'll address just a bunch and to be clear - I'm not even a big Java fan. - Quarkus with GraalVM compiles your Java app to native code. There is no JIT or warm up, memory footprint is also low. By the way, the JVM Hotspot JIT can actually make your Java app faster than your Go or Rust app in many cases [citation needed] exactly due to the hot path optimizations it does. - GC tuning - I don't even know who does this. Maybe Netflix or some trading shops? Almost no one does this nowadays and with the new JVM ZGC [0] coming up, nobody would need to. > You can’t ship a minimal standalone binary without pulling in a JVM. - You'd need JRE actually, e.g., 27 MB .MSI for Windows. That's probably the easiest thing to install today and if you do this via your package manager, you also get regular security fixes. Build tools like Gradle generate a fully ready-to-execute directory structure for your app. If you got the JRE on your system, it will run. > Dependency management and classpath conflicts historically plagued Java The keyword here is "historically". Please try Maven or Gradle today and enjoy the modern dependency management. It just works. I won't delve into Java 9 modules, but it's been ages since I last saw a class path issue. > J2EE Is someone still using this? It is super easy writing a web app with Java+Javalin for example. The Java library and frameworks ecosystem is super rich. > “Write once, run anywhere” costs: The abstraction layers that make Java portable also add runtime weight and overhead. Like I wrote above, the HotSpot JIT is actually doing the heavy lifting for your in real time. These claims are baseless without pointing to what "overhead" is meant in practice. --- 0 - https://inside.java/2023/11/28/gen-zgc-explainer/ https://inside.java/2023/11/28/gen-zgc-explainer/ or https://www.youtube.com/watch?v=dSLe6G3_JmE https://www.youtube.com/watch?v=dSLe6G3_JmE

cortesoft 1y ago

It doesn't have to be 'hyper-scaled' to be needed, unless we have widely different definitions of hyper scale. Access logs from a few thousand servers with medium traffic will push you past any single instance service, and Kafka works great for that workload.

rvz 1y ago

> Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play. Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today. Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.

AtlasBarfed 1y ago
I thought Kafka ditched the zookeeper
- mdaniel 1y ago
  I wish; KIP-500[1] was the "let's use native Raft" implementation but I have never once in my life seen anyone using Kafka in KIP-500 mode 1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+replace+zookeeper+with+a+self-managed+metadata+quorum https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...
  - enether 1y ago
    Confluent has shared they've migrated thousands of Kafka clusters (their whole cloud fleet) to KIP-500 https://www.confluent.io/blog/zookeeper-to-kraft-with-confluent-kubernetes/ https://www.confluent.io/blog/zookeeper-to-kraft-with-conflu...
  - 62951413 1y ago
    Very few people want to re-create their production AWS MSK cluster from scratch. And that's the only way currently. MSK can usually upgrade Kafka brokers with minor performance degradation but not for this particular change.

closeparen 1y ago

Publishing an event to Kafka puts it “out there” in a way that guarantees it won’t be lost and allows any number of interested consumers, including the data warehouse, to deal with it at their leisure (subject to retention period which is typically like 72h). For us, your Kafka topics and their schemas are as much a part of your API as your gRPC IDLs. Something like Redis or 0MQ feels more appropriate for internal coordination between instances of the same service, or at least a producer that has a specific consumer in mind.

haddr 1y ago

Couldn’t disagree more… if you go the ZMQ you are left alone handling many things you get in Kafka for free. If you have any sort of big data problems then good luck. You are going to reinvent the wheel.

gf000 1y ago

If you have this stance to technology ("feels bloated", meme-level knowledge on Java), then I really hope that you are not responsible for technical decisions.