3 ms·

Anyone have any opinions on the long-term viability of Kafka? I've been lurking on the kafka dev mailing lists and I'm fairly turned off by the attitude from C

Anyone have any opinions on the long-term viability of Kafka? I've been lurking on the kafka dev mailing lists and I'm fairly turned off by the attitude from Confluent's employees that I've read from. There was a recent thread about their open source status and how they backhand Apache's open source philosophies, I'm wondering if they are thinking of moving away from open source in the future.

winteriscoming 10y ago
I am not associated with the Kafka dev team or Confluent in any way but have been using Kafka since their 0.7.x days. Speaking of their open source commitment so far, I haven't seen any kind of problems in the way they deal with the community. They have been open to bug fixes, feature enhancements and other contributions. Many of the Confluent employees you see currently, started off by contributing to the open source code and still do, from what I can see. >> There was a recent thread about their open source status I am not sure which thread you are talking about, but if it is that thread which involved adding REST server within Kafka core then I completely back what many of the Confluent employees and other community members decided on that topic. From what I could see in that thread, the whole reason of "we should bring in REST server within Kafka core" was not related to technical reasons but hypothetical reasons like "there's a project out there which already supports this REST feature but what if they don't like my contributions and don't allow me to push features that I like into that repo". That proposal of bringing in the REST server within Kafka core was, IMO, rightly rejected but at the same time, the users were allowed to state the technical reasons why they want that feature within core. Given any production usable project that has a large user base, discussions and decisions like these are common and that doesn't essentially mean they are moving away from open source. Overall, I have high respect to many of the members of Kafka dev team, many of whom are currently employed at Confluent, for the way they have so far dealt with suggestions to enhancements in the project. Of course, Confluent builds on top of Kafka and has/will have they own commercial interests, so some of the features that they develop might/will be commercial.
- pfarnsworth 10y ago
  Yes, I think that's the one but my biggest takeaway from what I remember was "Apache is okay but..." I remember reading about half a dozen Confluent employees towing the party line and parroting the exact same argument, about how being under Apache was stifling their innovation. And yet Kafka is thriving under Apache, so their arguments smelled fishy as hell. It sounds like they are setting up the foundation to pull something like what Oracle did with MySQL, and try to take control after countless outside helpers turned the product into something rock solid.
  - SEJeff 10y ago
    There are plenty of outside helpers, but the heavyweights doing the overwhelming majority of the work are either Jay, Jun, or Neha and/or work for one of those 3... All at Confluent. I've met both Jay and Jun in person, and watched Neha present @ MesosCon last year. If they're doing something with tech, it is going to be good.
boredandroid 10y ago
I'm one of the original Kafka creators and am an evil corporate shill at Confluent :-) That was a super silly discussion and I'd be annoyed reading it too. It does not, however, represent any kind of "move away from open source" by Confluent. The discussion was not actually about open source vs non-open source but rather around how to curate the Kafka ecosystem. Currently there are several hundred independent open source projects that act as connectors, clients, monitoring tools, processing libraries and frameworks, etc. Some of these are fantastic and active communities, some are built by companies that use Kafka, some by vendors that use Kafka in a product in some way. This has always been the structure of this ecosystem, since even before we moved Kafka into Apache. These projects are mostly Apache licensed and on github, but mostly not part of the ASF. You can see a subsample of them here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem The proposal from a group at Hortonworks was to duplicate one of the modules Confluent built, a REST layer, and build a new one in the main project. I don't think this is inherently a bad thing to do, but there wasn't really any concrete rationale...there was no complaint with the code in that module, nor any kind of complaint with the governance, nor any complaint with the community, etc. As a result a lot of us at Confluent kind of felt it was an odd exercise that had more to do with Hortonworks' policy of only shipping Apache projects, which in turn comes out of the weird fractious politics of the Hadoop ecosystem. The reality is that policy just doesn't fit with the Kafka ecosystem as it stands today.

winteriscoming 10y ago

I am not associated with the Kafka dev team or Confluent in any way but have been using Kafka since their 0.7.x days. Speaking of their open source commitment so far, I haven't seen any kind of problems in the way they deal with the community. They have been open to bug fixes, feature enhancements and other contributions. Many of the Confluent employees you see currently, started off by contributing to the open source code and still do, from what I can see. >> There was a recent thread about their open source status I am not sure which thread you are talking about, but if it is that thread which involved adding REST server within Kafka core then I completely back what many of the Confluent employees and other community members decided on that topic. From what I could see in that thread, the whole reason of "we should bring in REST server within Kafka core" was not related to technical reasons but hypothetical reasons like "there's a project out there which already supports this REST feature but what if they don't like my contributions and don't allow me to push features that I like into that repo". That proposal of bringing in the REST server within Kafka core was, IMO, rightly rejected but at the same time, the users were allowed to state the technical reasons why they want that feature within core. Given any production usable project that has a large user base, discussions and decisions like these are common and that doesn't essentially mean they are moving away from open source. Overall, I have high respect to many of the members of Kafka dev team, many of whom are currently employed at Confluent, for the way they have so far dealt with suggestions to enhancements in the project. Of course, Confluent builds on top of Kafka and has/will have they own commercial interests, so some of the features that they develop might/will be commercial.

pfarnsworth 10y ago
Yes, I think that's the one but my biggest takeaway from what I remember was "Apache is okay but..." I remember reading about half a dozen Confluent employees towing the party line and parroting the exact same argument, about how being under Apache was stifling their innovation. And yet Kafka is thriving under Apache, so their arguments smelled fishy as hell. It sounds like they are setting up the foundation to pull something like what Oracle did with MySQL, and try to take control after countless outside helpers turned the product into something rock solid.
- SEJeff 10y ago
  There are plenty of outside helpers, but the heavyweights doing the overwhelming majority of the work are either Jay, Jun, or Neha and/or work for one of those 3... All at Confluent. I've met both Jay and Jun in person, and watched Neha present @ MesosCon last year. If they're doing something with tech, it is going to be good.

boredandroid 10y ago

I'm one of the original Kafka creators and am an evil corporate shill at Confluent :-) That was a super silly discussion and I'd be annoyed reading it too. It does not, however, represent any kind of "move away from open source" by Confluent. The discussion was not actually about open source vs non-open source but rather around how to curate the Kafka ecosystem. Currently there are several hundred independent open source projects that act as connectors, clients, monitoring tools, processing libraries and frameworks, etc. Some of these are fantastic and active communities, some are built by companies that use Kafka, some by vendors that use Kafka in a product in some way. This has always been the structure of this ecosystem, since even before we moved Kafka into Apache. These projects are mostly Apache licensed and on github, but mostly not part of the ASF. You can see a subsample of them here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem The proposal from a group at Hortonworks was to duplicate one of the modules Confluent built, a REST layer, and build a new one in the main project. I don't think this is inherently a bad thing to do, but there wasn't really any concrete rationale...there was no complaint with the code in that module, nor any kind of complaint with the governance, nor any complaint with the community, etc. As a result a lot of us at Confluent kind of felt it was an odd exercise that had more to do with Hortonworks' policy of only shipping Apache projects, which in turn comes out of the weird fractious politics of the Hadoop ecosystem. The reality is that policy just doesn't fit with the Kafka ecosystem as it stands today.