12 ms·
We extensively explored this approach for a use-case and dropped it in favour of BigQuery external data query for one core reason: It can't do streaming reads i
by lysecret 3y ago
We extensively explored this approach for a use-case and dropped it in favour of BigQuery external data query for one core reason: It can't do streaming reads if you have to do any kind of operation on the underlying data, for almost everything it will have to load all the relevant parquet files locally to do the operations. You might have been able to solve this using row partitions on the underlying files beforehand, but it was too late for us.
Also, it actually turned out to be much more expensive to run the operations in cloud run, than relying on BQ (due to the fact it wouldn't stream as we wished).
More than anything it gave me renewed appreciation for BQ.
- chrisjc 3y agoVery interesting. Would you mind mentioning which external table format you were using? Iceberg by any chance? I wonder if there wasn't a middle-ground that could still utilize DuckDB or similar. You still rely on BQ to deliver the subset of data needed, and then rely on DuckDB to do the "last-mile" analytics: grouping, filtering, ordering, etc...?
- tobilg 3y agoI don't think this is correct, or I didn't really understand your use case. Could you give an example for "any kind of operation on the underlying data"?