4 ms·

This looks like an amazing piece of software! I have a couple of questions: - Does it (out of the box) support mapping JSON messages on a Kafka Topic to a tabl

This looks like an amazing piece of software! I have a couple of questions: - Does it (out of the box) support mapping JSON messages on a Kafka Topic to a table accessible from SQL? - How would you support the following use case: run SQL queries on one or more Kafka Topics and expose the results in (near) real-time through a JDBC supporting BI tool? I assume you'd use, AthenaX to run the queries and store the results in a standard RDBMS like PostgreSQL?

haohui 9y ago
(1) Yes. AthenaX provides a thin wrapper of KafkaJsonTableSource[1] out-of-the-box. As long as you specify the schema it should be good to go. (2) Your guess is correct. Internally we have AthenaX streaming data to MemSQL, Pinot or MySQL to support the use case you described. Some of them go through the JDBCTableSink [2], the others go through a customized connectors. AthenaX is designed to allow plugging in connectors when required. [1] https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaJsonTableSource.java https://github.com/apache/flink/blob/master/flink-connectors... [2] https://github.com/apache/flink/blob/master/flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCAppendTableSink.java https://github.com/apache/flink/blob/master/flink-connectors...