6 ms·

I work with Kafka every day and don't really think the OP's concerns (it's too complex and there isn't third party driver support to the degree of Redis) are to

I work with Kafka every day and don't really think the OP's concerns (it's too complex and there isn't third party driver support to the degree of Redis) are too serious. They both will be solved with time. My bigger fear is Confluent - the private company founded by many Kafka core committers and employing many Kafka committers. Confluent offers open source extensions to Kafka's core in the form of connectors (boiler plate code to connect to common sources and sinks like JDBC databases, files, and Hadoop). Confluent also offers (as of right now) one closed source product extension (Control Center - a cluster monitoring system similar to the management UI of RabbitMQ, etc) that requires enterprise subscription (several thousand dollars per node per year) after a 30 day trial. $30.9MM for a service/support based company seems like a lot of money and a drives a very high valuation that needs to show return. I personally am skeptical of service/support venture backed model[0]. My fear is that Kafka will increasingly require "enterprise" support tools with less and less support and features available to people who do not pay for enterprise support. The amount of documentation of the 0.10 release (particularly the Streams API) that resided on the Confluent page versus the Kafka page is a HUGE red flag to me. I have all the respect in the world for Jay, and the Kafka/Confluent team, but I find myself avoiding Confluent's tools (Kafka Connect and Schema Registry) because of fear that those will eventually be closed source or require an enterprise subscription. [0] I'm not an investor but I haven't seen many these models work out in the long run. A recent Podcast by A16Z touches on this subject very well, with an A16Z partner saying he believes exactly one company has pulled this model off well at venture scale - Red Hat. http://a16z.com/2016/08/19/pricing-freemium-premium-opensource/ http://a16z.com/2016/08/19/pricing-freemium-premium-opensour...

djhworld 10y ago
My fear is that Kafka will increasingly require "enterprise" support tools with less and less support and features available to people who do not pay for enterprise support. I thought Kafka was an Apache project, don't they have rules against this sort of thing?
- x0x0 10y ago
  Sure, the Apache distro of kafka won't require them. Users who buy from Confluent will download Confluent's build of kafka. Similar to how things work in the hadoop ecosystem: You can get Apache hadoop, but most users get one of Cloudera, Hortonworks, or MapR.
  - ptrincr 10y ago
    As it stands though, kakfa is no where near as complicated as setting up and managing a hadoop/yarn/spark(etc etc) cluster. Without using something like Hortonwork's Ambari, it's a right pain trying to get everything installed, working together and managing it. Then you have to worry about how you are going to upgrade it. Currently you can use this chef cookbook https://github.com/mthssdrbrg/kafka-cookbook https://github.com/mthssdrbrg/kafka-cookbook , along with the yahoo kafka-manager application and you can get a cluster running without much effort or complication. The cookbook has a coordination feature which allows you to use a locking mechanism ( I use consul as we use it for dns/auto registration), which allows you to roll out changes across the cluster, restarting each broker one at a time, it also allows you to upgrade in a similar fashion.
gfodor 10y ago
Kafka Connect is part of Apache Kafka, so the odds of that becoming closed source are basically nil. At worst, you could imagine a Confluent-owned fork. Kafka Streams (another Kafka extension that Confluent has spearheaded) is also part of Apache Kafka. The Schema Registry and Kafka REST Proxy are confluent projects but are Apache licensed. I'd imagine the reason for this is less "Confluent might want to lock it down" and more "getting these projects adopted into the Apache Kafka project is more trouble than it is worth." Either way, worst case, just fork it. The schema registry is simple enough that it's hard to imagine it really needing much feature work from here anyway. The REST proxy I could imagine having enterprise features down the road though. It's hard to know how exactly Confluent could walk back the OSS nature of these projects though without a major backlash (not to mention the Apache license may actually prevent this? I am not sure.)

djhworld 10y ago

My fear is that Kafka will increasingly require "enterprise" support tools with less and less support and features available to people who do not pay for enterprise support. I thought Kafka was an Apache project, don't they have rules against this sort of thing?

x0x0 10y ago
Sure, the Apache distro of kafka won't require them. Users who buy from Confluent will download Confluent's build of kafka. Similar to how things work in the hadoop ecosystem: You can get Apache hadoop, but most users get one of Cloudera, Hortonworks, or MapR.
- ptrincr 10y ago
  As it stands though, kakfa is no where near as complicated as setting up and managing a hadoop/yarn/spark(etc etc) cluster. Without using something like Hortonwork's Ambari, it's a right pain trying to get everything installed, working together and managing it. Then you have to worry about how you are going to upgrade it. Currently you can use this chef cookbook https://github.com/mthssdrbrg/kafka-cookbook https://github.com/mthssdrbrg/kafka-cookbook , along with the yahoo kafka-manager application and you can get a cluster running without much effort or complication. The cookbook has a coordination feature which allows you to use a locking mechanism ( I use consul as we use it for dns/auto registration), which allows you to roll out changes across the cluster, restarting each broker one at a time, it also allows you to upgrade in a similar fashion.

gfodor 10y ago

Kafka Connect is part of Apache Kafka, so the odds of that becoming closed source are basically nil. At worst, you could imagine a Confluent-owned fork. Kafka Streams (another Kafka extension that Confluent has spearheaded) is also part of Apache Kafka. The Schema Registry and Kafka REST Proxy are confluent projects but are Apache licensed. I'd imagine the reason for this is less "Confluent might want to lock it down" and more "getting these projects adopted into the Apache Kafka project is more trouble than it is worth." Either way, worst case, just fork it. The schema registry is simple enough that it's hard to imagine it really needing much feature work from here anyway. The REST proxy I could imagine having enterprise features down the road though. It's hard to know how exactly Confluent could walk back the OSS nature of these projects though without a major backlash (not to mention the Apache license may actually prevent this? I am not sure.)