6 ms·
To clarify, the quota is 1,000 loads per table per day, and 50,000 loads per project per day[1] for batch loads. In practice, I've never hit either of these qu
by numbsafari 9y ago
To clarify, the quota is 1,000 loads per table per day, and 50,000 loads per project per day[1] for batch loads.
In practice, I've never hit either of these quotas, because our data architecture is such that we'd have to have some kind of major operational screw up to encounter them (e.g., a run-away failed import job that somehow manages to fail 50,000 times in a single day). Regardless, you can work around the per-table limit by using Wildcard Tables[2].
In my case, the source of my data is strictly batch-oriented, so importing data into distinct tables with a common prefix solves this problem for me. Where I do use partition tables, the likelihood that I'm going to update a single partition 1000 times in a day is highly unlikely (I'd be curious about any use case for that).
The DML features of standard SQL are probably best viewed as a "helpful backup" rather than a general solution. Yes, you can run UPDATE and INSERT statements against your non-partitioned tables, but you probably want to limit that to situations where you are trying to correct something specific, or on an ad-hoc basis, and not as a general solution.
In any case, there are a number of tools you can use to not be impacted by this particular limitation. For example, you could use streaming[3]. And while streaming has it's own quota (100,000 inserts per project per second)[4], it's straightforward to ask for an increased quota.
The trick is to think of BQ like you do Hive or other "non-typical" SQL databases: yes it has a SQL query interface, but if you treat it like PGSQL or Oracle, you aren't going to get the benefits out of it.
If you are going to try and perform OLTP operations against BQ you are going to fail. It's a data warehouse, not a transactional system. Working with log-structured data, or append-only data, is where it really shines.
[1]: https://cloud.google.com/bigquery/quotas#import https://cloud.google.com/bigquery/quotas#import
[2]: https://cloud.google.com/bigquery/docs/reference/standard-sql/wildcard-table-reference https://cloud.google.com/bigquery/docs/reference/standard-sq...
[3]: https://cloud.google.com/bigquery/streaming-data-into-bigquery https://cloud.google.com/bigquery/streaming-data-into-bigque...
[4]: https://cloud.google.com/bigquery/quotas#streaminginserts https://cloud.google.com/bigquery/quotas#streaminginserts
- slap_shot 9y agoWe evaluated streaming in each of the cases and it wasn't sufficient. We ended up doing, essentially, the same thing with Redshift and it worked like a charm.
- numbsafari 9y agoI’d be curious to know more about the workloads that failed. What kind of data were you streaming?