4 ms·

Kafka on Kubernetes: What could go wrong?

35 points by g0xA52A2A 4y ago

ckdarby 4y ago
TL;DR This is a plug for Red panda. If you're going to look at Kafka inside of Kubernetes at all just look at running Apache Pulsar in K8s. One of the main arguments in this article is worry about exposing the broker but in Pulsar there is a proxy for this and the helm chart supports ingress public endpoint by just enabling a flag.
- bsaul 4y ago
  Is there somewhere a real list of pros of cons of pulsar vs kafka ( or redpanda) , from people having real life experience of the two ( and not just reading the documentation) ?
  - elcapitan 4y ago
    Btw, speaking of pros and cons, it would be really nice to have a feature comparison website for different types of technologies (e.g. relational databases, messaging systems, etc). For some, there are Wikipedia "Comparison of X" pages, and to some degree you can extract and discover via Stackshare, but Wikipedia is far from complete and up to date, and Stackshare mostly just shows you who uses what and describes technologies via marketing buzzwords rather than somewhat "normalized" descriptions.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
shitloadofbooks 4y ago
Doesn't Strimzi make this completely painless?
- deleted 4y ago
  [deleted]
- smartbit 4y ago
  No, not completely painless, but in my experience it makes running kafka quite smoothly. Can highly recommend it.
javier2 4y ago
We run Kafka with kubernetes, but its so locked down/customized we might as well run it on bare metal. We started out on rook network storage, but after multiple fs corruption incidents, we now run every broker with local storage. Kube still gives us the benefit of re-using our deployment software and many monitoring conventions, so we do get some benefit from kubernetes. The only reason we did not lose data on the rook setup, is we run 5 brokers with replication factor 4, with 2 in sync.
js4ever 4y ago
This is an ad for Redpanda and it's BSL, please stay away
- voganmother42 4y ago
  BSL?
  - js4ever 4y ago
    Business source license = fakely open source
    - lostcolony 4y ago
      While still meeting most of the OSI criteria for being open source. It basically says "you can use the features covered by this license for most uses, except creating your own SaaS offering of this product", plus some of the typical boilerplate about ensuring the license travels with the code. It's basically a license that was created in response to how Amazon has been operating, co-opting open source technologies (whose business models focused on creating SaaS offerings of their open source tech) and creating their own managed services, directly competing with the creators (Redis even called this out a few years ago when they changed their license, see https://techcrunch.com/2019/02/21/redis-labs-changes-its-open-source-license-again/ https://techcrunch.com/2019/02/21/redis-labs-changes-its-ope...). They also indicate that at some point the license will change to Apache 2 ( https://github.com/redpanda-data/redpanda/tree/dev/licenses https://github.com/redpanda-data/redpanda/tree/dev/licenses ); it's BSL right now likely to give them enough legal protection to keep competitors who'd operate Redpanda as a SaaS away, while they build up the enterprise (paid) features (which has a separate, far more wordy, license). I'd imagine they'd look to make the switch when they feel they have enough value and customers of the enterprise features. I don't work on open source, or anything licensed by BSI, and from such an outsider perspective you sound...very dogmatic.
      - slimsag 4y ago
        "The BSL is not an Open Source license and we do not claim it to be one." "The BSL does not meet the Open Source Definition (OSD) maintained by the Open Source Initiative (OSI). OSD does not allow limitations on specific kinds of such, such as production use." > Q: Can I use BSL products to develop software that will be licensed under different licenses? > > A: Yes, as long as you don’t include any of the BSL code in the code for the software that’s being developed, and you do not make production use of the BSL code. From the creators of the BSL license[0]: https://mariadb.com/bsl-faq-adopting/ https://mariadb.com/bsl-faq-adopting/ BSL is not open source, period. It's just another startup source-available license parading "I want to be open source on GitHub.. but gosh please don't use my code without paying me!!" Companies should stop instructing Heather to create these licenses IMO, and just split their code into open source and enterprise-only parts. It's easier, more honest, and ends this whole debate. It's the GitLab/Sourcegraph model too.
bogomipz 4y ago
I would recommend looking at Strmzi for running Kafka on K8S. It works well and addresses all the issues mentioned in this piece. There's no reason to switch to RedPanda because you're running Kubernetes. https://strimzi.io/ https://strimzi.io/
- js4ever 4y ago
  Totally agree, and it's really open source unlike redpanda. For me really anything else is better than redpanda and their BSL license
mt42or 4y ago
It's always funny to see attacks comming from vendors which are struggling in the real life because of Kubernetes success.
- the_mitsuhiko 4y ago
  I’m not sure how that connects to the blog post. Redpanda works fine with Kubernetes and I assume by vendor you mean them.
jon-wood 4y ago
Did I seriously just read through the very worst kind of fiction, business process fiction, just to be bait and switched with an ad for something that isn’t actually Kafka? I don’t even use Kafka or Kubernetes but I thought this might be an interesting look at the possible failure modes and how to handle them.
gattacamovie 4y ago
as some mentioned, there are benefits and there are costs applying the 'lift and shift' for big things like kafka (and elastic search, DBs, etc). The main assumption is that MOST of your apps (especially the ones affected by kafka access latency) run in k8s already. Access from outside should be mainly for: a. integrations with other systems which do not have very strict latency constraints b. replication to disaster recovery site The benefits are listed (between lines) in both article and below. Price is is usually still treating those pods like pets: - one pod per k8s node (taints&node selectors) - special sizing & tunings of the targeted nodes (resources, kernel params, etc) - one LB per Pod. yes, costly and against what you would expect, but that's what is required for a bullet proof deploy. (delay is not always there, especially in clouds, LB have super efficient implementations (especially gcp) - bullet proof storage, with the required performance computed in your sizing phase.
osigurdson 4y ago
I know many caution against doing this kind of thing but what is not to like about the concept? Why can’t we have a single, reliable and well understood “substrate” for which to deploy backend infrastructure for our apps?

35 points by g0xA52A2A 4y ago

TL;DR This is a plug for Red panda. If you're going to look at Kafka inside of Kubernetes at all just look at running Apache Pulsar in K8s. One of the main arguments in this article is worry about exposing the broker but in Pulsar there is a proxy for this and the helm chart supports ingress public endpoint by just enabling a flag.

bsaul 4y ago
Is there somewhere a real list of pros of cons of pulsar vs kafka ( or redpanda) , from people having real life experience of the two ( and not just reading the documentation) ?
- elcapitan 4y ago
  Btw, speaking of pros and cons, it would be really nice to have a feature comparison website for different types of technologies (e.g. relational databases, messaging systems, etc). For some, there are Wikipedia "Comparison of X" pages, and to some degree you can extract and discover via Stackshare, but Wikipedia is far from complete and up to date, and Stackshare mostly just shows you who uses what and describes technologies via marketing buzzwords rather than somewhat "normalized" descriptions.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]

shitloadofbooks 4y ago

Doesn't Strimzi make this completely painless?

deleted 4y ago
[deleted]
smartbit 4y ago
No, not completely painless, but in my experience it makes running kafka quite smoothly. Can highly recommend it.

javier2 4y ago

We run Kafka with kubernetes, but its so locked down/customized we might as well run it on bare metal. We started out on rook network storage, but after multiple fs corruption incidents, we now run every broker with local storage. Kube still gives us the benefit of re-using our deployment software and many monitoring conventions, so we do get some benefit from kubernetes. The only reason we did not lose data on the rook setup, is we run 5 brokers with replication factor 4, with 2 in sync.

js4ever 4y ago

This is an ad for Redpanda and it's BSL, please stay away

voganmother42 4y ago
BSL?
- js4ever 4y ago
  Business source license = fakely open source
  - lostcolony 4y ago
    While still meeting most of the OSI criteria for being open source. It basically says "you can use the features covered by this license for most uses, except creating your own SaaS offering of this product", plus some of the typical boilerplate about ensuring the license travels with the code. It's basically a license that was created in response to how Amazon has been operating, co-opting open source technologies (whose business models focused on creating SaaS offerings of their open source tech) and creating their own managed services, directly competing with the creators (Redis even called this out a few years ago when they changed their license, see https://techcrunch.com/2019/02/21/redis-labs-changes-its-open-source-license-again/ https://techcrunch.com/2019/02/21/redis-labs-changes-its-ope...). They also indicate that at some point the license will change to Apache 2 ( https://github.com/redpanda-data/redpanda/tree/dev/licenses https://github.com/redpanda-data/redpanda/tree/dev/licenses ); it's BSL right now likely to give them enough legal protection to keep competitors who'd operate Redpanda as a SaaS away, while they build up the enterprise (paid) features (which has a separate, far more wordy, license). I'd imagine they'd look to make the switch when they feel they have enough value and customers of the enterprise features. I don't work on open source, or anything licensed by BSI, and from such an outsider perspective you sound...very dogmatic.
    - slimsag 4y ago
      "The BSL is not an Open Source license and we do not claim it to be one." "The BSL does not meet the Open Source Definition (OSD) maintained by the Open Source Initiative (OSI). OSD does not allow limitations on specific kinds of such, such as production use." > Q: Can I use BSL products to develop software that will be licensed under different licenses? > > A: Yes, as long as you don’t include any of the BSL code in the code for the software that’s being developed, and you do not make production use of the BSL code. From the creators of the BSL license[0]: https://mariadb.com/bsl-faq-adopting/ https://mariadb.com/bsl-faq-adopting/ BSL is not open source, period. It's just another startup source-available license parading "I want to be open source on GitHub.. but gosh please don't use my code without paying me!!" Companies should stop instructing Heather to create these licenses IMO, and just split their code into open source and enterprise-only parts. It's easier, more honest, and ends this whole debate. It's the GitLab/Sourcegraph model too.

bogomipz 4y ago

I would recommend looking at Strmzi for running Kafka on K8S. It works well and addresses all the issues mentioned in this piece. There's no reason to switch to RedPanda because you're running Kubernetes. https://strimzi.io/ https://strimzi.io/

js4ever 4y ago
Totally agree, and it's really open source unlike redpanda. For me really anything else is better than redpanda and their BSL license

mt42or 4y ago

It's always funny to see attacks comming from vendors which are struggling in the real life because of Kubernetes success.

the_mitsuhiko 4y ago
I’m not sure how that connects to the blog post. Redpanda works fine with Kubernetes and I assume by vendor you mean them.

jon-wood 4y ago

Did I seriously just read through the very worst kind of fiction, business process fiction, just to be bait and switched with an ad for something that isn’t actually Kafka? I don’t even use Kafka or Kubernetes but I thought this might be an interesting look at the possible failure modes and how to handle them.

gattacamovie 4y ago

as some mentioned, there are benefits and there are costs applying the 'lift and shift' for big things like kafka (and elastic search, DBs, etc). The main assumption is that MOST of your apps (especially the ones affected by kafka access latency) run in k8s already. Access from outside should be mainly for: a. integrations with other systems which do not have very strict latency constraints b. replication to disaster recovery site The benefits are listed (between lines) in both article and below. Price is is usually still treating those pods like pets: - one pod per k8s node (taints&node selectors) - special sizing & tunings of the targeted nodes (resources, kernel params, etc) - one LB per Pod. yes, costly and against what you would expect, but that's what is required for a bullet proof deploy. (delay is not always there, especially in clouds, LB have super efficient implementations (especially gcp) - bullet proof storage, with the required performance computed in your sizing phase.

osigurdson 4y ago

I know many caution against doing this kind of thing but what is not to like about the concept? Why can’t we have a single, reliable and well understood “substrate” for which to deploy backend infrastructure for our apps?