4 ms·

I do the support for my department's trino cluster. We move ~1tb (and growing) in ETL jobs and support interactive queries for the data scientists/analysts. It

by dmead 4y ago

I do the support for my department's trino cluster. We move ~1tb (and growing) in ETL jobs and support interactive queries for the data scientists/analysts. It would be super good if you guys added big query write support. Its really annoying to have to run a hive cluster in google to act as a proxy for this.

atwebb 4y ago
Any chance you have an overview of the architecture and operations support required? How many data sources are you pinging?
- dmead 4y ago
  like 20 different sources. I nag my service reps about it. I"m sure it's been filed in your jira.
hashhar 4y ago
BigQuery very recently announced their Storage Write API which is one of the ways we were looking to implement this but there are some issues with the latency and consistency guarantees that it offers. But, yes, we do plan to add that eventually after ironing out all the kinks. See https://github.com/trinodb/trino/pull/13094 https://github.com/trinodb/trino/pull/13094
- bitsondatadev 4y ago
  Also, you can keep track of all the BQ progress here: https://github.com/trinodb/trino/issues/6867 https://github.com/trinodb/trino/issues/6867
  - dmead 4y ago
    thanks. If we can eliminate the costs and upkeep of hive in gcp, it would make my life easier for sure.