4 ms·
Where are said files coming from? What sorta latency requirements do you have?
by mattbillenstein 7y ago
Where are said files coming from? What sorta latency requirements do you have?
- s-xyz 7y agoCloud Storage, no latency requirements
- mattbillenstein 7y agoymmv but I kinda like Airflow for automating data ingestion tasks - mainly using its scheduling to run Python scripts that ETL data into bigquery. BigQuery can also directly query data on gcs now I believe, so maybe that's an option if you don't need to do any transforms.