10 ms·

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/trav

by tbrock 6y ago

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper dep, JVM foundation)

closeparen 6y ago
Keystores are clunky, but other than that, why exactly?
- tbrock 6y ago
  Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?
  - staticassertion 6y ago
    But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step. That's the power of having tunables.
    - tbrock 6y ago
      Then they chose better defaults because I never have to think about it.
      - growse 6y ago
        Just because the defaults are better for you doesn't mean they're better for everyone.
  - nhoughto 6y ago
    Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...
    - tptacek 6y ago
      They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be. Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.
      - nhoughto 6y ago
        yep so not 100% the same, but my point with linking to it is tunables / levers / control is useful. In other scenarios maybe you could get by without having to do a rewrite in Rust, maybe you could keep it in golang and tune the GC a bit. Whilst I like the memory ballast as a clever solution, it does smell like a workaround to a solvable problem if a parameter existed to tune it directly.
        tptacek 6y ago
        It's totally a workaround and I'm jumping on that last comment when I shouldn't be; it's just a reaction to the hyperbolic "you might have to rewrite" conclusion, because you won't have to rewrite, but indeed you might be annoyed by the lack of tunables.
      - jonathanoliver 6y ago
        I'm an active Go user and I've been working almost exclusively in Go since 2011. I have a feeling that over time, the number of knobs on Go will increase much like it has in Java. I'm trying to think back to my first experiences with Java in 1995. It was so simple compared to the overhead of getting things working in C/C++. So Java's got like 25+ years on Go and I would speculate that the number of knobs on Go in 25 years will be significantly higher. On the one hand, I don't like seeing all the crazy configurable stuff, e.g. CGO_....=1 until I actually need it.
  - vips7L 6y ago
    There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx<all_your_ram>.
    - simtel20 6y ago
      If you haven't done -Xmx=<all memory> before, you may not be aware that you're giving a bum steer here.
    - thayne 6y ago
      That is a terrible idea. The JVM basically never returns RAM to the OS, and in my experience, if you run a java process long enough, the max heap size ends up being the used heap size.
      - vips7L 6y ago
        G1 the default allocator returns heap to the os only when you set -G1PeriodicGCInterval and both ZGC and Shenandoah do it automatically.
    - bobbyi_settv 6y ago
      But to know whether you want that, you need to understand the application. Elasticsearch, for example, recommends using at most 50% of physical memory for heap: https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... You also often want to stay below 32 gb to get compressed OOPs
    - tbrock 6y ago
      Why isn’t that the default? Unless I say otherwise, use it. Imagine launching every app and having to say ok you get 300mb but not more!
      - rahilb 6y ago
        It’s the default because Java has a history, and it was set when everyone was deploying war files to machines with 512mb total memory and the default worked for 95% of people for the past 25 years. In fact, it’s still good enough for the chocolate factory that has brainwashed you into drinking the static binary milkshake. Of course there are ways to use all the available ram when you deploy your kubernetes pods, you just have to consult the documentation. Honestly this criticism is ridiculous I can’t believe I even bothered to reply.
  - Twirrim 6y ago
    There's a black box optimizer for databases, which is designed to take the user away from having to find and tweak all the hundreds of possible tunables in the database, https://github.com/cmu-db/ottertune https://github.com/cmu-db/ottertune / https://ottertune.cs.cmu.edu/ https://ottertune.cs.cmu.edu/ I keep wanting the same for JVM. It's probably nice that there are so many ways to adjust the JVM to best suit a very wide range of workloads, but from a practical perspective I rarely have spare time to go diving through the guts to experiment. Let me point something at a one-box, and it can tweak parameters to its hearts content, so I can export and apply to the rest of the fleet.
    - pjmlp 6y ago
      Well, ART auto tunes and probably not everyone will agree it does it perfectly. There are also plenty JVMs to pick and some are better than others working with their defaults.
  - nine_k 6y ago
    Normally you just run your one JVM app on your VM. The real balance is basically between the heap size and page cache size. You might want to tweak it, as much as the GC algorithm and other memory details. This applies if you understand them. If you don't, you just start the JVM with defaults, and it configures itself for optimal performance, using its idea of optimal.
agallego 6y ago
Agreed. We're working on this exactly - https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.
- manigandham 6y ago
  Nice, always like seeing new projects. Is this based on Seastar? Will this be open-source?
  - agallego 6y ago
    Yeah. Based on seastar. W.r.t OSS doing research on it atm. I had a VP at a major cloud say to me "we will try to use your tech" in person. So doing research before I make the decision
- snidane 6y ago
  Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.
  - agallego 6y ago
    Thanks! Yeh I think the same, specially with the rise of core count. I think an i3.metal is at 96vcpus. I estimate a machine like that should hold around 400K partitions just fine with us. I should measure this weekend.
  - sz4kerto 6y ago
    Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.
    - agallego 6y ago
      The JVM is fast. (though I'd say the best thing about the JVM is the amazingly large test suite it maintains). I think he was alluding to the fact it is much easier to develop predictable systems in a language that forces you to deal with the constraints up front. - i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. Not to mention that a lot of libraries that are immediately avail for c++ (see io_uring) take a while to get ported to java. The cost of JNI for a libaio wrapper is also expensive. Last i bench(few years back), switching to a crc32 with JNI switch alone was 30 microseconds - an eternity - before doing the work. In any case, we use seastar (seastar.io), which I'm not sure can actually be ported to java. The pinned thread per core makes a lot of sense for minimizing latency and cache pollution, etc. Externally, the feeling that apps in java are slow is real, less because JVM is slow per se, but because writing low latency apps in java is not the idiomatic way and those that do see to extract every ounce of performance of the hardware often look else where since the work is just about the same.
jonathanoliver 6y ago
Someone is adding journaling to the NATS protocol to make it Kakfa like: https://liftbridge.io/ https://liftbridge.io/ It was just released as v1.0 a few weeks ago.
- please-reread 6y ago
  > make it Kakfa like Kakfa?
skyde 6y ago
are you using Jocko in production?
- tbrock 6y ago
  No, it didn’t support consumer groups last time I checked (but could!).
elric 6y ago
Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them shifts some of the complexity of dealing with certificates from the developer to the system administrator. It's an implementation choice. The only other complaint of yours I could find ITT was about setting min/max heap size... not only is it extremely convenient to be able to do that, it also hasn't ever technically been required, and the defaults have been Good Enough for most uses since Java 5 came out in 2004. The JVM will figure it out for you, and if it's too conservative or too aggressive for you, you can tune it. If Kafka is a PITA to deploy, fine, maybe that's fair criticism. Not all things on the JVM are pain to deploy.
- axaxs 6y ago
  I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.
  - karmakaze 6y ago
    To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.
    - axaxs 6y ago
      This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.
      - weego 6y ago
DiabloD3 6y ago
All you're doing is exchanging one runtime, the JVM, that happens to be well understood, highly optimized, and has a decade long track record, and supported by every major company on Earth... for Go's runtime, one that isn't well understood, does not have a track record, and is a product of a company that is known to just abandon popular services for no apparent reason. You also seem to have misunderstood what causes some apps to be a pain to deploy: the apps, themselves, suck, and would suck no matter what language they had been written in. Apache tends to become the home for a lot of projects that exhibit that particular issue, and Zookeeper does not seem to be an exception. Kafka likely also has a case of the Apacheisms going on, but also apparently has a non-trivial amount of Scala in it. Complaining about Kafka fits your requirements far more than Zookeeper alone does.
- bananabreakfast 6y ago
  Complaining about Google sunsetting consumer products as a means to disparage their technical contributions to open source software is incredibly trite and a very tired meme. So, so much of Google depends on Go. It is extremely well understood, has an excellent track record and has zero chance of being "abandoned"
- tinco 6y ago
  What about the JVM is well understood? The most common trope about it is that it's supposed to be the fastest VM ever because, but Java devs always manage to build slow apps in it. So much for understanding it well. The claim that Go does not have a track record is ridiculous. It's whole premise is that it's made to power essential parts of the largest IT operation in the world. And there's nothing for Google to cancel, it's open source and they even made the effort of translating it to Go so it's super easy to work on. And I'm not fanboying over Go, haven't written a Go service in years, but denying Go is an excellent platform to build these kinds of apps on is ridiculous.

by tbrock 6y ago

closeparen 6y ago

Keystores are clunky, but other than that, why exactly?

tbrock 6y ago
Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?
- staticassertion 6y ago
  But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step. That's the power of having tunables.
  - tbrock 6y ago
    Then they chose better defaults because I never have to think about it.
    - growse 6y ago
      Just because the defaults are better for you doesn't mean they're better for everyone.
- nhoughto 6y ago
  Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to-rust-a190bbca2b1f https://blog.discord.com/why-discord-is-switching-from-go-to...
  - tptacek 6y ago
    They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be. Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.
    - nhoughto 6y ago
      yep so not 100% the same, but my point with linking to it is tunables / levers / control is useful. In other scenarios maybe you could get by without having to do a rewrite in Rust, maybe you could keep it in golang and tune the GC a bit. Whilst I like the memory ballast as a clever solution, it does smell like a workaround to a solvable problem if a parameter existed to tune it directly.
      - tptacek 6y ago
        It's totally a workaround and I'm jumping on that last comment when I shouldn't be; it's just a reaction to the hyperbolic "you might have to rewrite" conclusion, because you won't have to rewrite, but indeed you might be annoyed by the lack of tunables.
    - jonathanoliver 6y ago
      I'm an active Go user and I've been working almost exclusively in Go since 2011. I have a feeling that over time, the number of knobs on Go will increase much like it has in Java. I'm trying to think back to my first experiences with Java in 1995. It was so simple compared to the overhead of getting things working in C/C++. So Java's got like 25+ years on Go and I would speculate that the number of knobs on Go in 25 years will be significantly higher. On the one hand, I don't like seeing all the crazy configurable stuff, e.g. CGO_....=1 until I actually need it.
- vips7L 6y ago
  There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx<all_your_ram>.
  - simtel20 6y ago
    If you haven't done -Xmx=<all memory> before, you may not be aware that you're giving a bum steer here.
  - thayne 6y ago
    That is a terrible idea. The JVM basically never returns RAM to the OS, and in my experience, if you run a java process long enough, the max heap size ends up being the used heap size.
    - vips7L 6y ago
      G1 the default allocator returns heap to the os only when you set -G1PeriodicGCInterval and both ZGC and Shenandoah do it automatically.
  - bobbyi_settv 6y ago
    But to know whether you want that, you need to understand the application. Elasticsearch, for example, recommends using at most 50% of physical memory for heap: https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... You also often want to stay below 32 gb to get compressed OOPs
  - tbrock 6y ago
    Why isn’t that the default? Unless I say otherwise, use it. Imagine launching every app and having to say ok you get 300mb but not more!
    - rahilb 6y ago
      It’s the default because Java has a history, and it was set when everyone was deploying war files to machines with 512mb total memory and the default worked for 95% of people for the past 25 years. In fact, it’s still good enough for the chocolate factory that has brainwashed you into drinking the static binary milkshake. Of course there are ways to use all the available ram when you deploy your kubernetes pods, you just have to consult the documentation. Honestly this criticism is ridiculous I can’t believe I even bothered to reply.
- Twirrim 6y ago
  There's a black box optimizer for databases, which is designed to take the user away from having to find and tweak all the hundreds of possible tunables in the database, https://github.com/cmu-db/ottertune https://github.com/cmu-db/ottertune / https://ottertune.cs.cmu.edu/ https://ottertune.cs.cmu.edu/ I keep wanting the same for JVM. It's probably nice that there are so many ways to adjust the JVM to best suit a very wide range of workloads, but from a practical perspective I rarely have spare time to go diving through the guts to experiment. Let me point something at a one-box, and it can tweak parameters to its hearts content, so I can export and apply to the rest of the fleet.
  - pjmlp 6y ago
    Well, ART auto tunes and probably not everyone will agree it does it perfectly. There are also plenty JVMs to pick and some are better than others working with their defaults.
- nine_k 6y ago
  Normally you just run your one JVM app on your VM. The real balance is basically between the heap size and page cache size. You might want to tweak it, as much as the GC algorithm and other memory details. This applies if you understand them. If you don't, you just start the JVM with defaults, and it configures itself for optimal performance, using its idea of optimal.

agallego 6y ago

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

manigandham 6y ago
Nice, always like seeing new projects. Is this based on Seastar? Will this be open-source?
- agallego 6y ago
  Yeah. Based on seastar. W.r.t OSS doing research on it atm. I had a VP at a major cloud say to me "we will try to use your tech" in person. So doing research before I make the decision
snidane 6y ago
Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.
- agallego 6y ago
  Thanks! Yeh I think the same, specially with the rise of core count. I think an i3.metal is at 96vcpus. I estimate a machine like that should hold around 400K partitions just fine with us. I should measure this weekend.
- sz4kerto 6y ago
  Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.
  - agallego 6y ago
    The JVM is fast. (though I'd say the best thing about the JVM is the amazingly large test suite it maintains). I think he was alluding to the fact it is much easier to develop predictable systems in a language that forces you to deal with the constraints up front. - i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. Not to mention that a lot of libraries that are immediately avail for c++ (see io_uring) take a while to get ported to java. The cost of JNI for a libaio wrapper is also expensive. Last i bench(few years back), switching to a crc32 with JNI switch alone was 30 microseconds - an eternity - before doing the work. In any case, we use seastar (seastar.io), which I'm not sure can actually be ported to java. The pinned thread per core makes a lot of sense for minimizing latency and cache pollution, etc. Externally, the feeling that apps in java are slow is real, less because JVM is slow per se, but because writing low latency apps in java is not the idiomatic way and those that do see to extract every ounce of performance of the hardware often look else where since the work is just about the same.

jonathanoliver 6y ago

Someone is adding journaling to the NATS protocol to make it Kakfa like: https://liftbridge.io/ https://liftbridge.io/ It was just released as v1.0 a few weeks ago.

please-reread 6y ago
> make it Kakfa like Kakfa?

skyde 6y ago

are you using Jocko in production?

tbrock 6y ago
No, it didn’t support consumer groups last time I checked (but could!).

elric 6y ago

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them shifts some of the complexity of dealing with certificates from the developer to the system administrator. It's an implementation choice. The only other complaint of yours I could find ITT was about setting min/max heap size... not only is it extremely convenient to be able to do that, it also hasn't ever technically been required, and the defaults have been Good Enough for most uses since Java 5 came out in 2004. The JVM will figure it out for you, and if it's too conservative or too aggressive for you, you can tune it. If Kafka is a PITA to deploy, fine, maybe that's fair criticism. Not all things on the JVM are pain to deploy.

axaxs 6y ago
I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.
- karmakaze 6y ago
  To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.
  - axaxs 6y ago
    This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.
    - weego 6y ago

DiabloD3 6y ago

All you're doing is exchanging one runtime, the JVM, that happens to be well understood, highly optimized, and has a decade long track record, and supported by every major company on Earth... for Go's runtime, one that isn't well understood, does not have a track record, and is a product of a company that is known to just abandon popular services for no apparent reason. You also seem to have misunderstood what causes some apps to be a pain to deploy: the apps, themselves, suck, and would suck no matter what language they had been written in. Apache tends to become the home for a lot of projects that exhibit that particular issue, and Zookeeper does not seem to be an exception. Kafka likely also has a case of the Apacheisms going on, but also apparently has a non-trivial amount of Scala in it. Complaining about Kafka fits your requirements far more than Zookeeper alone does.

bananabreakfast 6y ago
Complaining about Google sunsetting consumer products as a means to disparage their technical contributions to open source software is incredibly trite and a very tired meme. So, so much of Google depends on Go. It is extremely well understood, has an excellent track record and has zero chance of being "abandoned"
tinco 6y ago
What about the JVM is well understood? The most common trope about it is that it's supposed to be the fastest VM ever because, but Java devs always manage to build slow apps in it. So much for understanding it well. The claim that Go does not have a track record is ridiculous. It's whole premise is that it's made to power essential parts of the largest IT operation in the world. And there's nothing for Google to cancel, it's open source and they even made the effort of translating it to Go so it's super easy to work on. And I'm not fanboying over Go, haven't written a Go service in years, but denying Go is an excellent platform to build these kinds of apps on is ridiculous.