7 ms·

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

nicobn 10y ago
For better or for worse, Java runs a big chunk of this world. Not running a Java application because "ewww Java is gross" will prevent you from enjoying some of the most exciting database technologies out there.
asimuvPR 10y ago
Mind sharing a bit more?
- ryanobjc 10y ago
  The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people have noted above as well.
  - eropple 10y ago
    You're right in that Zookeeper can be a hassle in a dynamic environment, but in my experience Exhibitor largely removes that concern.
  - rad_gruchalski 10y ago
    Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
    - ryanobjc 10y ago
      i think you do know the problem, which is not everyone has that set up. Yes, if you are cloud, and can roll with static IPs then great for you. Some people still have to be on-prem and reusing IPs isn't always an option.
    - sciurus 10y ago
      How is this an improvement over using DNS? When you swap in a new EC2 instance it has different IP addresses from the old instance. So now not only do you have to restart your other Zookeeper servers in the cluster, you have to update their configuration then restart them.
      - curun1r 10y ago
        He's probably using ENIs to decouple the IP address from the instance.
  - neeleshs 10y ago
    https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZOOKEEPER-1506 https://issues.apache.org/jira/plugins/servlet/mobile#issue/...
  - neilc 10y ago
    This behavior should be fixed in Zk 3.4.7+ and 3.5.0+
  - ttwwmm 10y ago
    We run ZooKeeper (and Kafka) within Docker containers and were able to solve this problem with a patch to the client: https://issues.apache.org/jira/browse/ZOOKEEPER-2184 https://issues.apache.org/jira/browse/ZOOKEEPER-2184 You also need to disable the JVM's insane DNS-caching behavior (config change only).
gshx 10y ago
Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.
- slantedview 10y ago
  General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on. Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any system is potentially asking a lot of users/operators.
  - growse 10y ago
    Well, like many things, if you deploy it on sensible hardware and put some thought into how its operated, zookeeper isn't particularly problematic. It's certainly not a 'nightmare'.
  - sciurus 10y ago
    As an operator I do feel that pain but the alternative of every piece of software that needs distributed coordination reimplementing it, likely poorly, it isn't great either.
    - curun1r 10y ago
      There are (IMHO, superior) alternatives like Etcd and Consul that could be used. Ideally, these kinds systems should be built such that the distributed coordination piece is pluggable and the implementation can be chosen based on deployment concerns.
jlisam13 10y ago
'plus systems software in Java', mind sharing an explanation?
- glibgil 10y ago
  Java has garbage collection and is about two times slower than C
  - sgt101 10y ago
    For good C, which is hard to write - for me and other mortal humans. On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code. The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.
    - glibgil 10y ago
      You can call the comparison pointless, but it is the answer to the question of why Java is not a systems language
  - Chyzwar 10y ago
    C dont have garbage collection. In some cases Java can outperform C.
    - glibgil 10y ago
      Yeah, that's what I said. C doesn't have gc so it is a systems language. Java has gc so it is not a systems language
- ryanobjc 10y ago
  Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong. Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to. Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks. It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.
slantedview 10y ago
Agree re: Zookeeper. It looks like DistributedLog uses BookKeeper, which I'm guessing is where the Zookeeper usage comes from. In terms of simplifying the deployment, they could have went embedded with Atomix[1] instead. Perhaps next time. [1]: http://atomix.io/ http://atomix.io/
kinkdr 10y ago
Exactly this. When I saw the header I thought to myself, "Yay! Now I can run something like Kafka, without depending on Zookeeper or having to install JVM!! Give me stable drivers for Python and Go and I am sold!" If I have to install Zookeeper and JVM, why not use Kafka?
vruiz 10y ago
I have similar worries with kafka, that’s why I’ve been working on and off in my own log based message queue for while now [0]. So far is nothing but a humble beginning to something targeting very simple use cases, but AFAIK there aren't any lightweight solutions in this space. [0] https://github.com/ninibe/netlog https://github.com/ninibe/netlog
frugalmail 10y ago
Rails, node? Web scalable, eh?

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

nicobn 10y ago

For better or for worse, Java runs a big chunk of this world. Not running a Java application because "ewww Java is gross" will prevent you from enjoying some of the most exciting database technologies out there.

asimuvPR 10y ago

Mind sharing a bit more?

ryanobjc 10y ago
The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people have noted above as well.
- eropple 10y ago
  You're right in that Zookeeper can be a hassle in a dynamic environment, but in my experience Exhibitor largely removes that concern.
- rad_gruchalski 10y ago
  Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
  - ryanobjc 10y ago
    i think you do know the problem, which is not everyone has that set up. Yes, if you are cloud, and can roll with static IPs then great for you. Some people still have to be on-prem and reusing IPs isn't always an option.
  - sciurus 10y ago
    How is this an improvement over using DNS? When you swap in a new EC2 instance it has different IP addresses from the old instance. So now not only do you have to restart your other Zookeeper servers in the cluster, you have to update their configuration then restart them.
    - curun1r 10y ago
      He's probably using ENIs to decouple the IP address from the instance.
- neeleshs 10y ago
  https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZOOKEEPER-1506 https://issues.apache.org/jira/plugins/servlet/mobile#issue/...
- neilc 10y ago
  This behavior should be fixed in Zk 3.4.7+ and 3.5.0+
- ttwwmm 10y ago
  We run ZooKeeper (and Kafka) within Docker containers and were able to solve this problem with a patch to the client: https://issues.apache.org/jira/browse/ZOOKEEPER-2184 https://issues.apache.org/jira/browse/ZOOKEEPER-2184 You also need to disable the JVM's insane DNS-caching behavior (config change only).

gshx 10y ago

Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.

slantedview 10y ago
General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on. Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any system is potentially asking a lot of users/operators.
- growse 10y ago
  Well, like many things, if you deploy it on sensible hardware and put some thought into how its operated, zookeeper isn't particularly problematic. It's certainly not a 'nightmare'.
- sciurus 10y ago
  As an operator I do feel that pain but the alternative of every piece of software that needs distributed coordination reimplementing it, likely poorly, it isn't great either.
  - curun1r 10y ago
    There are (IMHO, superior) alternatives like Etcd and Consul that could be used. Ideally, these kinds systems should be built such that the distributed coordination piece is pluggable and the implementation can be chosen based on deployment concerns.

jlisam13 10y ago

'plus systems software in Java', mind sharing an explanation?

glibgil 10y ago
Java has garbage collection and is about two times slower than C
- sgt101 10y ago
  For good C, which is hard to write - for me and other mortal humans. On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code. The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.
  - glibgil 10y ago
    You can call the comparison pointless, but it is the answer to the question of why Java is not a systems language
- Chyzwar 10y ago
  C dont have garbage collection. In some cases Java can outperform C.
  - glibgil 10y ago
    Yeah, that's what I said. C doesn't have gc so it is a systems language. Java has gc so it is not a systems language
ryanobjc 10y ago
Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong. Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to. Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks. It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.

slantedview 10y ago

Agree re: Zookeeper. It looks like DistributedLog uses BookKeeper, which I'm guessing is where the Zookeeper usage comes from. In terms of simplifying the deployment, they could have went embedded with Atomix[1] instead. Perhaps next time. [1]: http://atomix.io/ http://atomix.io/

kinkdr 10y ago

Exactly this. When I saw the header I thought to myself, "Yay! Now I can run something like Kafka, without depending on Zookeeper or having to install JVM!! Give me stable drivers for Python and Go and I am sold!" If I have to install Zookeeper and JVM, why not use Kafka?

vruiz 10y ago

I have similar worries with kafka, that’s why I’ve been working on and off in my own log based message queue for while now [0]. So far is nothing but a humble beginning to something targeting very simple use cases, but AFAIK there aren't any lightweight solutions in this space. [0] https://github.com/ninibe/netlog https://github.com/ninibe/netlog

frugalmail 10y ago

Rails, node? Web scalable, eh?