4 ms·

The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_guide#mul

by ryguyrg 14y ago

The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_guide#multithreading https://developers.google.com/storage/docs/gsutil_reference_... @iskander - do you have other approaches for uploading which you'd prefer?

iskander 14y ago
Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?