4 ms·

I work at Google and authored some of those posts. The limitation is per table[0] and for "batch loads" only. If you have per row inserts, you can always use b

by vgt 9y ago

I work at Google and authored some of those posts. The limitation is per table[0] and for "batch loads" only. If you have per row inserts, you can always use bigquery's amazing streaming API, you can also do materialization and federated query loads. Batch limits are there to help folks from creating too many small files, which affects performance (I believe Jordan's book explains this in greater detail). Would love to know more about your use cases since this type of thing shouldn't be the deciding factor in choosing an analytics data warehouse and should be easy to architect for. As someone pointed out, most quotas are negotiable. If anything, bigquery ingest process is pretty amazing and unique in the industry [1]. Apologies for brevity, I'm out of office on paternity leave. [0]https://cloud.google.com/bigquery/quotas#import https://cloud.google.com/bigquery/quotas#import [1]https://medium.com/google-cloud/paying-it-forward-how-bigquerys-data-ingest-breaks-tech-norms-8bfe2341f5eb https://medium.com/google-cloud/paying-it-forward-how-bigque...

slap_shot 9y ago
All use cases were a firehose of raw data being pulled from a Kafka topic every 15 seconds - 1 minute, written to storage, and inserted into 1,000+ tables.
- slap_shot 9y ago
  Also, if the quota can be negotiated, I would reccomended verbiage closer to that of the GA reporting API limits: 50,000 requests per project per day, which can be increased.
- vgt 9y ago
  Yeah definitely streaming API. This use case is MADE for BigQuery :) Check out, as well as the blog posts on the required reading list on the topic of inserts. There's also a pretty good Kafka-bq connector. It sounds like "ingest" was the big factor in your choice of a data warehouse. Regardless, I think bigquery has the best story in the industry here (but I'm biased). I'd help more hands on but sadly not online for a while :)
isp 9y ago
Thank you for the posts that you authored. These were very helpful for me at $DAY_JOB. BigQuery itself is a very impressive piece of tech.
- vgt 9y ago
  Thanks for reading and using Google cloud :)

by vgt 9y ago

slap_shot 9y ago

All use cases were a firehose of raw data being pulled from a Kafka topic every 15 seconds - 1 minute, written to storage, and inserted into 1,000+ tables.

slap_shot 9y ago
Also, if the quota can be negotiated, I would reccomended verbiage closer to that of the GA reporting API limits: 50,000 requests per project per day, which can be increased.
vgt 9y ago
Yeah definitely streaming API. This use case is MADE for BigQuery :) Check out, as well as the blog posts on the required reading list on the topic of inserts. There's also a pretty good Kafka-bq connector. It sounds like "ingest" was the big factor in your choice of a data warehouse. Regardless, I think bigquery has the best story in the industry here (but I'm biased). I'd help more hands on but sadly not online for a while :)

isp 9y ago

Thank you for the posts that you authored. These were very helpful for me at $DAY_JOB. BigQuery itself is a very impressive piece of tech.

vgt 9y ago
Thanks for reading and using Google cloud :)