3 ms·
All use cases were a firehose of raw data being pulled from a Kafka topic every 15 seconds - 1 minute, written to storage, and inserted into 1,000+ tables.
by slap_shot 9y ago
All use cases were a firehose of raw data being pulled from a Kafka topic every 15 seconds - 1 minute, written to storage, and inserted into 1,000+ tables.
- slap_shot 9y agoAlso, if the quota can be negotiated, I would reccomended verbiage closer to that of the GA reporting API limits: 50,000 requests per project per day, which can be increased.
- vgt 9y agoYeah definitely streaming API. This use case is MADE for BigQuery :) Check out, as well as the blog posts on the required reading list on the topic of inserts. There's also a pretty good Kafka-bq connector. It sounds like "ingest" was the big factor in your choice of a data warehouse. Regardless, I think bigquery has the best story in the industry here (but I'm biased). I'd help more hands on but sadly not online for a while :)