7 ms·

What are the usual benefits of all those "x reimplemented in go" type of projects ? Do people see performance improvments in practice ? Or is it simpler to depl

by bsaul 10y ago

What are the usual benefits of all those "x reimplemented in go" type of projects ? Do people see performance improvments in practice ? Or is it simpler to deploy / monitor ? I know go has all the hype at the moment, but i'm curious to know what the benefits are in practice. Those real world reimplementation project make for a great benchmark.

antirez 10y ago
Here the point is more no Zookeeper needed apparently.
- travisjeffery 10y ago
  First, I'm a big fan of yours. I was recently interviewed by Go Time and I talked about how awesome Redis is, what a good example it/you've been for open source projects, and how funny your v4 RC blog post was. My goals for building Jocko to start: 1. Learn more about distributed systems to the point I could build one "What I cannot build, I do not understand." sort of thing. 2. Simplify running Kafka—so remove ZK, only need to install a binary, and so on 3. Maintain compatability—so people can just drop Jocko in, Kafka clients and anything else that uses Kafka's protocol works After that, I have some ideas on things to do but am waiting to get feedback from people. For instance, I think the configuration around setting how much disk space topics use could be better.
  - antirez 10y ago
    Hello, thanks for the kind words! And sorry for guessing what your goal was, I used the wrong words probably, what I meant is that for me the highest value of Jocko at a first glance is the tremendous operational simplification provided by the fact it does not need ZK while implementing the same protocol. Of course I guess the project is currently ongoing, but once it will be a drop in replacement for the Java implementation needing ZK this could be very interesting for many users. This project could also serve to Kafka developers as an inspiration to also remove ZK perhaps, I guess that they already reasoned towards this idea tons of times btw so there must be some rationale they have (based on their design choices) to keep it.
    - dastbe 10y ago
      Having distributed consensus as a distinct service (as opposed to directly building it into each application) is beneficial to a lot of companies because they can invest resources to make it operate well (in addition to being correct) and reap the benefits across all of their services. By well, I both mean having competent operations around the service as well as making it more resilient and faster to respond to various networking failures. the downside is that in order to use something like kafka locally (afaik?) you have to run zookeeper locally as well.
      - aries1980 10y ago
        There are other service discovery tools than Zookeeper such as Consul.io, Etcd, etc.
      - tcbawo 10y ago
        The biggest benefit IMHO would be from making this a pluggable service within the Kafka client and server. Then, you're not tied into ZooKeeper explicitly.
      - mioelnir 10y ago
        Also, speaking with my sysadmin hat here, Zookeeper is an insanely powerful introspection point for running distributed services. The number of services with internal cluster coordination protocols that offer similar capabilities is abysmal.
hashmp 10y ago
Getting rid of the zookeeper requirement is a huge plus. Adds complication to the stack when trying to automate the deployment process. Maybe you could achieve a similar thing with the Scala version however. For the Kafka deployment itself, you wouldn't need to install scala or java. Just download the go executable compiled for your architecture and away you go.
geodel 10y ago
Though author has not mentioned I think Go program would be using an order of magnitude less amount of memory compared to Java alternative. Besides of course you have mentioned ease of deployment. Also standard builtin http(1/2) client/server, json package can help a lot for setting up communication between various components.
- coredog64 10y ago
  Kafka isn't particularly memory intensive. Confluent suggests a small (5gb) heap and to let the OS filesystem caching handle the rest [0]. It is still early days in my deployment, but I believe IOPs are going to be the bottleneck. [0] http://docs.confluent.io/1.0/kafka/deployment.html http://docs.confluent.io/1.0/kafka/deployment.html
  - Sphax 10y ago
    Yeah same, the only problems we've had with our cluster has been disks, RAID 5 is a bad idea with Kafka. Other than that we've been running our cluster for almost 2 years now with the same 32GB servers, never once had a problem with memory or CPU, and we've been pushing more and more data into it.
    - deleted 10y ago
      [deleted]
  - geodel 10y ago
    It is in general Java itself thats memory intensive compare to Go.
- mdasen 10y ago
  Go memory usage will definitely not be an order of magnitude less. This is something that cannot be emphasized enough: the idea that a Go program of any substantial size would use an order of magnitude less memory than a Java one is completely wrong. First, the "proof": https://days2011.scala-lang.org/sites/days2011/files/ws3-1-Hundt.pdf https://days2011.scala-lang.org/sites/days2011/files/ws3-1-H.... That's a paper from Google. The memory footprint of the Go program was 501MB, the Java version 617MB, and the Scala version 293MB. Now, given that the Scala version and Java versions were so far off in memory just tells you that a bunch of it just depends on some specific things used. Most importantly, it should dispel the myth that Java uses more memory than Go. The JVM isn't perfect, but arguing that Java uses an order of magnitude more memory is simply wrong. Java has a bad reputation for memory usage among people who have never run substantial programs because a Java hello-world (or other small programs) will often allocate itself a decently sized heap. Java programs also take a while to start up because of the JVM, but that doesn't make Java slow. Go certainly has some good things. Structs can help with memory locality and avoiding some unnecessary pointers. But that's not going to help one with attaining an order of magnitude better memory usage. But Go's garbage collector isn't made for memory efficiency, but rather short pause times. That can be great, but Go seems to have made the choice that shorter pause times are preferable to throughput (actually accomplishing work) or using less memory (might as well throw more RAM at the problem). Now let's talk about Kafka specifically. It's been a while since I've looked at it, but Kafka actually doesn't keep the data on the JVM heap. It writes to the filesystem using the kernel's built-in async flushing and uses sendfile to write data directly from the OS page cache to sockets. In a lot of ways, Kafka bypasses its own runtime and just leans on the OS for the heavy lifting. I like Go, but it isn't magical. It certainly doesn't offer an order of magnitude memory improvement over Java. Go has some great bits, but if you're thinking that you'll get an order of magnitude memory improvement, it's not there. It might even use more memory than Java.
weberc2 10y ago
I think this is poor marketing; if you're familiar with Go, saying "written in Go", is shorthand for "low memory requirements, decent performance, no runtime dependencies / single binary deployment, straightforward compilation from source, easy to contribute to". But to everyone else it sounds like writing in Go is an end unto itself.
- apta 10y ago
  > decent performance Also in Java (if not better there). > single binary deployment Also possible in Java > straightforward compilation from source ditto > easy to contribute to ditto Edit: a much more expanded reply here: https://news.ycombinator.com/reply?id=13451098&goto=item%3Fid%3D13449728%2313451098 https://news.ycombinator.com/reply?id=13451098&goto=item%3Fi...
  - devoply 10y ago
    > single binary deployment > Also possible in Java Does not mean that's the default. Default matters quite a bit. Not to mention the verbosity of Java that I guess you can use other languages on top of Java to deal with. But Go makes a lot of sense for these sorts of projects which in the past would be written in C or C++
    - apta 10y ago
      Funnily enough, there are complaints about Go being verbose. Just look at the error handling discussion that was on here a few days ago.
  - closeparen 10y ago
    >Also possible in Java Really? When working with Java system-level tools I always need a shell script to manage JAVA_HOME and CLASSPATH, or an OS package where the maintainer has done that for me. I've seen these generated, but in a way that places an artificial upper bound on the number of command line arguments. Some programs like to have tons of these, so only going up to $10 doesn't cut it. With Go I can literally just copy the binary over and execute it.
tiger3 10y ago
From the README: "Learn a lot and have fun"

by bsaul 10y ago

antirez 10y ago

Here the point is more no Zookeeper needed apparently.

travisjeffery 10y ago
First, I'm a big fan of yours. I was recently interviewed by Go Time and I talked about how awesome Redis is, what a good example it/you've been for open source projects, and how funny your v4 RC blog post was. My goals for building Jocko to start: 1. Learn more about distributed systems to the point I could build one "What I cannot build, I do not understand." sort of thing. 2. Simplify running Kafka—so remove ZK, only need to install a binary, and so on 3. Maintain compatability—so people can just drop Jocko in, Kafka clients and anything else that uses Kafka's protocol works After that, I have some ideas on things to do but am waiting to get feedback from people. For instance, I think the configuration around setting how much disk space topics use could be better.
- antirez 10y ago
  Hello, thanks for the kind words! And sorry for guessing what your goal was, I used the wrong words probably, what I meant is that for me the highest value of Jocko at a first glance is the tremendous operational simplification provided by the fact it does not need ZK while implementing the same protocol. Of course I guess the project is currently ongoing, but once it will be a drop in replacement for the Java implementation needing ZK this could be very interesting for many users. This project could also serve to Kafka developers as an inspiration to also remove ZK perhaps, I guess that they already reasoned towards this idea tons of times btw so there must be some rationale they have (based on their design choices) to keep it.
  - dastbe 10y ago
    Having distributed consensus as a distinct service (as opposed to directly building it into each application) is beneficial to a lot of companies because they can invest resources to make it operate well (in addition to being correct) and reap the benefits across all of their services. By well, I both mean having competent operations around the service as well as making it more resilient and faster to respond to various networking failures. the downside is that in order to use something like kafka locally (afaik?) you have to run zookeeper locally as well.
    - aries1980 10y ago
      There are other service discovery tools than Zookeeper such as Consul.io, Etcd, etc.
    - tcbawo 10y ago
      The biggest benefit IMHO would be from making this a pluggable service within the Kafka client and server. Then, you're not tied into ZooKeeper explicitly.
    - mioelnir 10y ago
      Also, speaking with my sysadmin hat here, Zookeeper is an insanely powerful introspection point for running distributed services. The number of services with internal cluster coordination protocols that offer similar capabilities is abysmal.

hashmp 10y ago

Getting rid of the zookeeper requirement is a huge plus. Adds complication to the stack when trying to automate the deployment process. Maybe you could achieve a similar thing with the Scala version however. For the Kafka deployment itself, you wouldn't need to install scala or java. Just download the go executable compiled for your architecture and away you go.

geodel 10y ago

Though author has not mentioned I think Go program would be using an order of magnitude less amount of memory compared to Java alternative. Besides of course you have mentioned ease of deployment. Also standard builtin http(1/2) client/server, json package can help a lot for setting up communication between various components.

coredog64 10y ago
Kafka isn't particularly memory intensive. Confluent suggests a small (5gb) heap and to let the OS filesystem caching handle the rest [0]. It is still early days in my deployment, but I believe IOPs are going to be the bottleneck. [0] http://docs.confluent.io/1.0/kafka/deployment.html http://docs.confluent.io/1.0/kafka/deployment.html
- Sphax 10y ago
  Yeah same, the only problems we've had with our cluster has been disks, RAID 5 is a bad idea with Kafka. Other than that we've been running our cluster for almost 2 years now with the same 32GB servers, never once had a problem with memory or CPU, and we've been pushing more and more data into it.
  - deleted 10y ago
    [deleted]
- geodel 10y ago
  It is in general Java itself thats memory intensive compare to Go.
mdasen 10y ago
Go memory usage will definitely not be an order of magnitude less. This is something that cannot be emphasized enough: the idea that a Go program of any substantial size would use an order of magnitude less memory than a Java one is completely wrong. First, the "proof": https://days2011.scala-lang.org/sites/days2011/files/ws3-1-Hundt.pdf https://days2011.scala-lang.org/sites/days2011/files/ws3-1-H.... That's a paper from Google. The memory footprint of the Go program was 501MB, the Java version 617MB, and the Scala version 293MB. Now, given that the Scala version and Java versions were so far off in memory just tells you that a bunch of it just depends on some specific things used. Most importantly, it should dispel the myth that Java uses more memory than Go. The JVM isn't perfect, but arguing that Java uses an order of magnitude more memory is simply wrong. Java has a bad reputation for memory usage among people who have never run substantial programs because a Java hello-world (or other small programs) will often allocate itself a decently sized heap. Java programs also take a while to start up because of the JVM, but that doesn't make Java slow. Go certainly has some good things. Structs can help with memory locality and avoiding some unnecessary pointers. But that's not going to help one with attaining an order of magnitude better memory usage. But Go's garbage collector isn't made for memory efficiency, but rather short pause times. That can be great, but Go seems to have made the choice that shorter pause times are preferable to throughput (actually accomplishing work) or using less memory (might as well throw more RAM at the problem). Now let's talk about Kafka specifically. It's been a while since I've looked at it, but Kafka actually doesn't keep the data on the JVM heap. It writes to the filesystem using the kernel's built-in async flushing and uses sendfile to write data directly from the OS page cache to sockets. In a lot of ways, Kafka bypasses its own runtime and just leans on the OS for the heavy lifting. I like Go, but it isn't magical. It certainly doesn't offer an order of magnitude memory improvement over Java. Go has some great bits, but if you're thinking that you'll get an order of magnitude memory improvement, it's not there. It might even use more memory than Java.

weberc2 10y ago

I think this is poor marketing; if you're familiar with Go, saying "written in Go", is shorthand for "low memory requirements, decent performance, no runtime dependencies / single binary deployment, straightforward compilation from source, easy to contribute to". But to everyone else it sounds like writing in Go is an end unto itself.

apta 10y ago
> decent performance Also in Java (if not better there). > single binary deployment Also possible in Java > straightforward compilation from source ditto > easy to contribute to ditto Edit: a much more expanded reply here: https://news.ycombinator.com/reply?id=13451098&goto=item%3Fid%3D13449728%2313451098 https://news.ycombinator.com/reply?id=13451098&goto=item%3Fi...
- devoply 10y ago
  > single binary deployment > Also possible in Java Does not mean that's the default. Default matters quite a bit. Not to mention the verbosity of Java that I guess you can use other languages on top of Java to deal with. But Go makes a lot of sense for these sorts of projects which in the past would be written in C or C++
  - apta 10y ago
    Funnily enough, there are complaints about Go being verbose. Just look at the error handling discussion that was on here a few days ago.
- closeparen 10y ago
  >Also possible in Java Really? When working with Java system-level tools I always need a shell script to manage JAVA_HOME and CLASSPATH, or an OS package where the maintainer has done that for me. I've seen these generated, but in a way that places an artificial upper bound on the number of command line arguments. Some programs like to have tons of these, so only going up to $10 doesn't cut it. With Go I can literally just copy the binary over and execute it.

tiger3 10y ago

From the README: "Learn a lot and have fun"