7 ms·
GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam). There's a new featu
by manigandham 7y ago
GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam).
There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job.
https://cloud.google.com/dataflow/docs/guides/sql/dataflow-sql-intro https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...
- chrisjc 7y agoThank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.