3 ms·

If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a

If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.

bockris 14y ago
Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR. Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there. http://stackoverflow.com/questions/11990839/is-there-a-way-to-upload-to-cloudstorage-from-a-url http://stackoverflow.com/questions/11990839/is-there-a-way-t... This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.
- proppy 14y ago
  There is publicdata.samples.trigrams in the public datasets.
  - bockris 14y ago
    Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.
sp332 14y ago
Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.
- alooPotato 14y ago
  Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.
  - ryguyrg 14y ago
    FYI - you can technically upload directly to BigQuery and not go through cloud storage. It just tends to be a bit more error-prone in normal implementations. We're working on documenting resumable upload to handle error cases better.
ryguyrg 14y ago
The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_guide#multithreading https://developers.google.com/storage/docs/gsutil_reference_... @iskander - do you have other approaches for uploading which you'd prefer?
- iskander 14y ago
  Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?

If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.

bockris 14y ago

Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR. Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there. http://stackoverflow.com/questions/11990839/is-there-a-way-to-upload-to-cloudstorage-from-a-url http://stackoverflow.com/questions/11990839/is-there-a-way-t... This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.

proppy 14y ago
There is publicdata.samples.trigrams in the public datasets.
- bockris 14y ago
  Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.

sp332 14y ago

Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.

alooPotato 14y ago
Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.
- ryguyrg 14y ago
  FYI - you can technically upload directly to BigQuery and not go through cloud storage. It just tends to be a bit more error-prone in normal implementations. We're working on documenting resumable upload to handle error cases better.

ryguyrg 14y ago

The binary gsutil does have a multi-threaded mode for uploading many files simultaneously: https://developers.google.com/storage/docs/gsutil_reference_guide#multithreading https://developers.google.com/storage/docs/gsutil_reference_... @iskander - do you have other approaches for uploading which you'd prefer?

iskander 14y ago
Let me turn the question around. I have about 400GB of data (spread over ~3000 HDF files, each containing 42 compressed columns). The only way I can imagine using BigQuery on this data is if I convert it all to csv files (which will take a while) and upload them to google storage (which will also take a while). Is there any alternative?