We used to pump CSV files into BigQuery from remote locations. Data availability became a problem because of choppy/low bandwidth internet in many of the remote

We used to pump CSV files into BigQuery from remote locations. Data availability became a problem because of choppy/low bandwidth internet in many of the remote wind/solar sites we work in. We explored converting CSV’s to Parquet format mainly for the reduced size Parquet offers. BigQuery accepts Parquet files but still in beta. We then just zipped the CSV files which reduced the size to almost 1/8 and BigQuery accepts zipped files directly. Works like a charm, only downside is if your CSV files have zagged rows then errors are thrown up. You can change some settings and have BigQuery accept a certain number of zagged rows. If you are considering only file sizes then i would recommend this zipping method. If you are more interested in query time capabilities/advantages then you must consider Parquet. We are still experimenting with Parquet because our dataset keeps increasing by about 100GB everyday.