3 ms·

I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.

by matt_wulfeck 9y ago

jdormit 9y ago
As opposed to the time required to implement, debug, and support a custom solution?
- ckocagil 9y ago
  I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust. Simple code with "obviously no bugs" vs complex code with "no obvious bugs".
  - smugworth 9y ago
    > I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust. This is one of our biggest headaches, and it's not even that a Kafka server itself is so configurable. We have hundreds of teams writing client applications, and jumping on bridges because Kafka clients have poor configurations is getting old. Too many knobs to twiddle, but I guess that's what happens if you're expecting to be able to tweak for high performance.
549362-30499 9y ago
Even if that's the case, deploying and scaling a Kafka cluster is something that hundreds of companies have figured out and publicly written about. It's something that you can hire an experienced engineer to fix. When this thing runs into problems, they will be all new ones.
- bonesss 9y ago
  These days you can also find kubernetes scripts that handle the Kafka installation, setup, routing, etc. That just leaves the "simple work" of administering and tuning which, as you pointed out, is competence that's steadily growing in the industry.
nevi-me 9y ago
I'm a self-taught, have a single dedicated server, I have a single-instance Kafka running on top of ZK. Yes, I lose the benefits of replication, failover, etc. I don't need that though. The whole installation took me half an hour, learning Kafka took maybe 3 hours, and as long as my server's been up, Kafka's been up. Granted, I am not monitoring Kafka, but I do other processes. The other nice thing is that now that I have ZK, other software that need it can just reuse the same process. I think using the maintenance cost as a reason to write your own tool, is a short-sighted decision.

I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.

jdormit 9y ago

As opposed to the time required to implement, debug, and support a custom solution?

ckocagil 9y ago
I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust. Simple code with "obviously no bugs" vs complex code with "no obvious bugs".
- smugworth 9y ago
  > I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust. This is one of our biggest headaches, and it's not even that a Kafka server itself is so configurable. We have hundreds of teams writing client applications, and jumping on bridges because Kafka clients have poor configurations is getting old. Too many knobs to twiddle, but I guess that's what happens if you're expecting to be able to tweak for high performance.

549362-30499 9y ago

Even if that's the case, deploying and scaling a Kafka cluster is something that hundreds of companies have figured out and publicly written about. It's something that you can hire an experienced engineer to fix. When this thing runs into problems, they will be all new ones.

bonesss 9y ago
These days you can also find kubernetes scripts that handle the Kafka installation, setup, routing, etc. That just leaves the "simple work" of administering and tuning which, as you pointed out, is competence that's steadily growing in the industry.

nevi-me 9y ago

I'm a self-taught, have a single dedicated server, I have a single-instance Kafka running on top of ZK. Yes, I lose the benefits of replication, failover, etc. I don't need that though. The whole installation took me half an hour, learning Kafka took maybe 3 hours, and as long as my server's been up, Kafka's been up. Granted, I am not monitoring Kafka, but I do other processes. The other nice thing is that now that I have ZK, other software that need it can just reuse the same process. I think using the maintenance cost as a reason to write your own tool, is a short-sighted decision.