6 ms·

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question o

by bsaul 7y ago

Sidenote question : Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack. It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ? Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?

matteomerli 7y ago
> is pulsar going to take the same road ? Yes, it's in the works
- FBISurveillance 7y ago
  Related Kafka KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... It doesn't look like it's going to be ready anytime soon though.
- whycombin8 7y ago
  Any plans for bookkeeper?
- wilhow 7y ago
  Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?
  - matteomerli 7y ago
    That's correct, we're moving to have a pluggable metadata store and coordination service.
    - senderista 7y ago
      Note that this is a different goal from the KIP referenced above, which is to entirely remove any dependency on an external configuration service. The idea of “pluggable consensus” is explicitly rejected in this KIP.
PaulHoule 7y ago
Zookeeper sux. If you are in the Java world you can often roll something better out using Hazelcast.
geodel 7y ago
This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java. Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.
- tyri_kai_psomi 7y ago
  NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are much more straightforward.
  - _frkl 7y ago
    This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?
    - tyri_kai_psomi 7y ago
      In my thinking, I think of an endpoint as something at either end of the communication channel (NATS in this case) where it is effectively terminal. Usually this is where the application logic lies. Dereck Collison (creator of NATS) brings this up in many of his talks about NATS, but I think the source of his thinking might come from “End-to-End Arguments in System Design” by Saltzer, Reed, & Clark. The core of it is this point: "Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level." That is, in order get that message redundancy or exactly once delivery, or message persistence, you pay a high cost, and you may be better off delegating to the endpoints. This blog provides a good overview https://blog.acolyer.org/2014/11/14/end-to-end-arguments-in-system-design/ https://blog.acolyer.org/2014/11/14/end-to-end-arguments-in-... Here is the original paper http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...
pm90 7y ago
It depends. We have a ton of Java apps running atop of kubernetes. All of them use zk, but every team operates their own mini zk cluster deployed on k8s. It’s worked fine except for certain hard to debug problems that hit a few teams occasionally. I guess my point is that k8s let’s you shift the operational burden to dev teams if they need it. If you have a centralized operations team running a giant, common zk/ etcd, yeah this would be additional operational burden.
nvarsj 7y ago
It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always seem to when I criticize the JVM), but Apache/JVM style architecture feels REALLY long in the tooth to me. I think you are committing to an outdated and very expensive approach to building software if you use anything running on the JVM, especially Apache based anything. Cassandra is a great example of this - out of the box it's a terribly performing database that is extremely expensive to run and tune. Throw enough resources and time at it and you can get it to acceptable scalability - but running on the JVM which is a huge memory hog will always make it expensive to run (and even then, you will always get terrible latency distributions with the JVM's awful GC). If I was building a business I would run far far away from any JVM based solution. The only thing it has going for it is momentum. If you need to hire 100s of engineers off the street for a large project, then a JVM based stack is about your only option unfortunately.
- squarecog 7y ago
  I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang. There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.
  - nvarsj 7y ago
    Sorry, you are right. It would probably be more accurate to use Spring actually. I was thinking of Cassandra/Zookeeper specifically when I said Apache.
- vips7L 7y ago
  For what its worth I have a java microservice running on 13mb of ram.
atombender 7y ago
I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.
- FridgeSeal 7y ago
  And I believe Kafka has a KIP in progress to remove their reliance on Zookeeper altogether but having the brokers communicate with each other directly.
  - JensRantil 7y ago
    https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...
namelosw 7y ago
It seems to be the trend. There would be more and more equivalents in Go in a foreseeable future. There would be a split. I generally don't use Java or Go in my applications, but Go components are usually more lightweight and easier to use if you're not majorly working with those runtimes. JVM has a big overhead, and Java applications usually make things worse if they use things like dependency injection.

by bsaul 7y ago

matteomerli 7y ago

> is pulsar going to take the same road ? Yes, it's in the works

FBISurveillance 7y ago
Related Kafka KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A... It doesn't look like it's going to be ready anytime soon though.
whycombin8 7y ago
Any plans for bookkeeper?
wilhow 7y ago
Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?
- matteomerli 7y ago
  That's correct, we're moving to have a pluggable metadata store and coordination service.
  - senderista 7y ago
    Note that this is a different goal from the KIP referenced above, which is to entirely remove any dependency on an external configuration service. The idea of “pluggable consensus” is explicitly rejected in this KIP.

PaulHoule 7y ago

Zookeeper sux. If you are in the Java world you can often roll something better out using Hazelcast.

geodel 7y ago

This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java. Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.

tyri_kai_psomi 7y ago
NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are much more straightforward.
- _frkl 7y ago
  This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?
  - tyri_kai_psomi 7y ago
    In my thinking, I think of an endpoint as something at either end of the communication channel (NATS in this case) where it is effectively terminal. Usually this is where the application logic lies. Dereck Collison (creator of NATS) brings this up in many of his talks about NATS, but I think the source of his thinking might come from “End-to-End Arguments in System Design” by Saltzer, Reed, & Clark. The core of it is this point: "Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level." That is, in order get that message redundancy or exactly once delivery, or message persistence, you pay a high cost, and you may be better off delegating to the endpoints. This blog provides a good overview https://blog.acolyer.org/2014/11/14/end-to-end-arguments-in-system-design/ https://blog.acolyer.org/2014/11/14/end-to-end-arguments-in-... Here is the original paper http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...

pm90 7y ago

It depends. We have a ton of Java apps running atop of kubernetes. All of them use zk, but every team operates their own mini zk cluster deployed on k8s. It’s worked fine except for certain hard to debug problems that hit a few teams occasionally. I guess my point is that k8s let’s you shift the operational burden to dev teams if they need it. If you have a centralized operations team running a giant, common zk/ etcd, yeah this would be additional operational burden.

nvarsj 7y ago

It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always seem to when I criticize the JVM), but Apache/JVM style architecture feels REALLY long in the tooth to me. I think you are committing to an outdated and very expensive approach to building software if you use anything running on the JVM, especially Apache based anything. Cassandra is a great example of this - out of the box it's a terribly performing database that is extremely expensive to run and tune. Throw enough resources and time at it and you can get it to acceptable scalability - but running on the JVM which is a huge memory hog will always make it expensive to run (and even then, you will always get terrible latency distributions with the JVM's awful GC). If I was building a business I would run far far away from any JVM based solution. The only thing it has going for it is momentum. If you need to hire 100s of engineers off the street for a large project, then a JVM based stack is about your only option unfortunately.

squarecog 7y ago
I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang. There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.
- nvarsj 7y ago
  Sorry, you are right. It would probably be more accurate to use Spring actually. I was thinking of Cassandra/Zookeeper specifically when I said Apache.
vips7L 7y ago
For what its worth I have a java microservice running on 13mb of ram.

atombender 7y ago

I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.

FridgeSeal 7y ago
And I believe Kafka has a KIP in progress to remove their reliance on Zookeeper altogether but having the brokers communicate with each other directly.
- JensRantil 7y ago
  https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

namelosw 7y ago

It seems to be the trend. There would be more and more equivalents in Go in a foreseeable future. There would be a split. I generally don't use Java or Go in my applications, but Go components are usually more lightweight and easier to use if you're not majorly working with those runtimes. JVM has a big overhead, and Java applications usually make things worse if they use things like dependency injection.