3 ms·

At Landoop we submitted our proposal to present our KSQL at the Kafka summit but we were rejected. Ah, if we only knew... :)

rollulus 9y ago
Hmm, that's "interesting". Can you elaborate a bit on how yours and theirs differ, if any?
- stefansb 9y ago
  There is a difference of course. While we are still learning about their implementation I think the statements below are true 1. We don't support just stream. You can throw a SQL at a kafka topic as easy as SELECT * FROM `topic` [WHERE ] 2. We support selecting or filter on the record metadata: offset/timestamp/partition (Haven't seen something similar in Confluent KSQL) 3. We integrate with Schema Registry for Avro. We hope to support Hortonworks schema registry soon as well as protobuf. 4. We allow for injecting fields in the Kafka Key part. For example: SELECT _offset as `_key.offset`, field2 * field3 - abs(field4.field5) as total FROM `magic-topic` 5. Just quickly looking at the Confluent KSQL "abs" function i see it accepts Double only. It doublt that everything is converted to Double before it hits the method and then converted back. (too short of a time to understand the whole implementation). 6. Filters: is related to point 2. We allow filter on message metadata. For example: SELECT * FROM topicA WHERE (a.d.e + b) /c = 100 and _offset > 2000 and partition in (2,6,9) Also not sure if they have customers yet using it. We do.
- nehanarkhede 9y ago
  There's a pretty big difference to a point that Landoop's KCQL (Kafka Connect Query Language) and Confluent's KSQL (Streaming SQL for Apache Kafka) are two different products. - KSQL is a full-fledged Streaming SQL engine for all kinds of stream processing operations from windowed aggregations, stream-table joins, sessionization and much more. So it does more powerful stream processing on Kafka than what Landoop's product supports which is simple projections and filters. - KSQL can do that because it supports streams and tables as first-class constructs and tightly integrates with Kafka's Streams API and the Kafka log itself. We are not aware of any other products that do that today, including Landoop's tool. - We will add support for Kafka connectors so you can stream data from different systems into Kafka through KSQL. This will cover what Landoop intended with KCQL (Kafka Connect Query Language. - Confluent works with several very large enterprises and many of the companies that have adopted Kafka. We worked with those customers to learn what would solve real business problems and used that feedback to build KSQL. So it . models on real-world customer feedback. - We'd love to hear feedback. Here's the repository https://github.com/confluentinc/ksql https://github.com/confluentinc/ksql and here's the Slack Channel slackpass.io/confluentcommunity - #ksql Hope that helps!
  - edan0 9y ago
    KSQL is not a full-fledged streaming SQL engine. If you look at its syntax reference page (https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-reference.md#syntax-reference https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-...), you can see that it only has a small number of scalar functions, a handful of aggregate functions, and only four major SELECT operators including an invented and nonstandard WINDOW clause (even though ANSI standard SQL has had a WINDOW clause for 18 years, since SQL:99). Contrast that with SQLstream Blaze, which is a full-fledged streaming SQL engine with over a hundred scalar functions, operators, CEP operators/temporal predicates, aggregate functions, analytic functions, UDXes/UDFs, and major statement operators. (See http://sqlstream.com/docs/index.html?conc_transforming_and_analyzing_dat.html http://sqlstream.com/docs/index.html?conc_transforming_and_a... for details.) This all comes into play when an experienced SQL developer wants to write a real-world query or port an existing business application from an RBDMS to a stream processor. Outside of toy applications, this just can't be done with without a full-fledged streaming SQL engine.
nehanarkhede 9y ago
An independent program committee makes these calls and had to pick from 166 submissions. The program committee picks talks that are insightful and technically challenging.
- andrewstevenson 9y ago
  I think a bit of clarification is required. KCQL, kafka connect query language was jointly developed with Landoop when Confluent commissioned DataMountaineer to write the JDBC sink. The reason we added it was to simplify the configuration but it also enabled us to filter and support various options of the many data sources/sinks we have connectors for. Confluent removed the kcql from the sink and reverted to a flume style configuration we were trying to avoid. It's good to see Confluent plans to support their Ksql in Connect, following DataMountaineers lead. We hope this is optional. We can look at migrating if it supports all the options we need that KCQL provides. Regarding Landoops proposal. This was not about KCQL but SQL on Streams, also named KSQL which is integrated with their new product Kafka Lenses. We'll look at Confluents SQL and see which one to go forward with, maybe both but we have happy customers using our version. But congratulations on your KSQL very nice! We (Landoop and DM staff) have proved many ex colleagues in the Investment bank world wrong about Kafka and this cements our decisions to use it. Thanks.
- deleted 9y ago
  [deleted]

rollulus 9y ago

Hmm, that's "interesting". Can you elaborate a bit on how yours and theirs differ, if any?

stefansb 9y ago
There is a difference of course. While we are still learning about their implementation I think the statements below are true 1. We don't support just stream. You can throw a SQL at a kafka topic as easy as SELECT * FROM `topic` [WHERE ] 2. We support selecting or filter on the record metadata: offset/timestamp/partition (Haven't seen something similar in Confluent KSQL) 3. We integrate with Schema Registry for Avro. We hope to support Hortonworks schema registry soon as well as protobuf. 4. We allow for injecting fields in the Kafka Key part. For example: SELECT _offset as `_key.offset`, field2 * field3 - abs(field4.field5) as total FROM `magic-topic` 5. Just quickly looking at the Confluent KSQL "abs" function i see it accepts Double only. It doublt that everything is converted to Double before it hits the method and then converted back. (too short of a time to understand the whole implementation). 6. Filters: is related to point 2. We allow filter on message metadata. For example: SELECT * FROM topicA WHERE (a.d.e + b) /c = 100 and _offset > 2000 and partition in (2,6,9) Also not sure if they have customers yet using it. We do.
nehanarkhede 9y ago
There's a pretty big difference to a point that Landoop's KCQL (Kafka Connect Query Language) and Confluent's KSQL (Streaming SQL for Apache Kafka) are two different products. - KSQL is a full-fledged Streaming SQL engine for all kinds of stream processing operations from windowed aggregations, stream-table joins, sessionization and much more. So it does more powerful stream processing on Kafka than what Landoop's product supports which is simple projections and filters. - KSQL can do that because it supports streams and tables as first-class constructs and tightly integrates with Kafka's Streams API and the Kafka log itself. We are not aware of any other products that do that today, including Landoop's tool. - We will add support for Kafka connectors so you can stream data from different systems into Kafka through KSQL. This will cover what Landoop intended with KCQL (Kafka Connect Query Language. - Confluent works with several very large enterprises and many of the companies that have adopted Kafka. We worked with those customers to learn what would solve real business problems and used that feedback to build KSQL. So it . models on real-world customer feedback. - We'd love to hear feedback. Here's the repository https://github.com/confluentinc/ksql https://github.com/confluentinc/ksql and here's the Slack Channel slackpass.io/confluentcommunity - #ksql Hope that helps!
- edan0 9y ago
  KSQL is not a full-fledged streaming SQL engine. If you look at its syntax reference page (https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-reference.md#syntax-reference https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-...), you can see that it only has a small number of scalar functions, a handful of aggregate functions, and only four major SELECT operators including an invented and nonstandard WINDOW clause (even though ANSI standard SQL has had a WINDOW clause for 18 years, since SQL:99). Contrast that with SQLstream Blaze, which is a full-fledged streaming SQL engine with over a hundred scalar functions, operators, CEP operators/temporal predicates, aggregate functions, analytic functions, UDXes/UDFs, and major statement operators. (See http://sqlstream.com/docs/index.html?conc_transforming_and_analyzing_dat.html http://sqlstream.com/docs/index.html?conc_transforming_and_a... for details.) This all comes into play when an experienced SQL developer wants to write a real-world query or port an existing business application from an RBDMS to a stream processor. Outside of toy applications, this just can't be done with without a full-fledged streaming SQL engine.

nehanarkhede 9y ago

An independent program committee makes these calls and had to pick from 166 submissions. The program committee picks talks that are insightful and technically challenging.

andrewstevenson 9y ago
I think a bit of clarification is required. KCQL, kafka connect query language was jointly developed with Landoop when Confluent commissioned DataMountaineer to write the JDBC sink. The reason we added it was to simplify the configuration but it also enabled us to filter and support various options of the many data sources/sinks we have connectors for. Confluent removed the kcql from the sink and reverted to a flume style configuration we were trying to avoid. It's good to see Confluent plans to support their Ksql in Connect, following DataMountaineers lead. We hope this is optional. We can look at migrating if it supports all the options we need that KCQL provides. Regarding Landoops proposal. This was not about KCQL but SQL on Streams, also named KSQL which is integrated with their new product Kafka Lenses. We'll look at Confluents SQL and see which one to go forward with, maybe both but we have happy customers using our version. But congratulations on your KSQL very nice! We (Landoop and DM staff) have proved many ex colleagues in the Investment bank world wrong about Kafka and this cements our decisions to use it. Thanks.
deleted 9y ago
[deleted]