2 ms·
An option, albeit a bit complex would be: Debezium => Kafka => Parquet in Google Cloud Storage => BigQuery external queries.
by mrbungie 5y ago
An option, albeit a bit complex would be:
Debezium => Kafka => Parquet in Google Cloud Storage => BigQuery external queries.
- gunnarmorling 5y agoWhy the Parquet step? You should be able to do straight Debezium -> Kafka -> BQ, using the BQ sink connector for Kafka Connect (https://github.com/confluentinc/kafka-connect-bigquery); https://github.com/confluentinc/kafka-connect-bigquery); we have users using this with the Debezium MySQL connector, I'd expect this to work equally for Postgres. Disclaimer: working on Debezium