3 ms·
Isn't BigQuery real-time as well? For me, the wow factor is that you can host this yourself.
by programmarchy 7y ago
Isn't BigQuery real-time as well? For me, the wow factor is that you can host this yourself.
- chrisjc 7y agoI'm unfamiliar with BQ, but from what I understand BQ doesn't have any concept of streams... with maybe an exception for change-streams (change tracking) and that's very different and less powerful than what Materialize is able to do. After all, in BQ you're only able to query what has been already has been ingested or is being digested and that is somewhat by definition not 'real-time'.
- manigandham 7y agoGCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam). There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job. https://cloud.google.com/dataflow/docs/guides/sql/dataflow-sql-intro https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...
- chrisjc 7y agoThank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.
- manigandham 7y agoBigQuery does have streaming ingest and can pickup new rows immediately but you still have to run a query to get the latest results. This system is more like BigQuery with materialized views that are constantly updated as new data arrives.