4 ms·
Try to run a few petabyte-scale queries and you will be surprised when you see the invoice. If you switch to flat-rate pricing, don't expect the latency that yo
by buremba 6y ago
Try to run a few petabyte-scale queries and you will be surprised when you see the invoice. If you switch to flat-rate pricing, don't expect the latency that you mentioned.
- manigandham 6y agoYou dont need to scan the whole dataset. Partitioning + clustering, especially for time-series, should be very efficient. BQ flat-rate pricing starts at 10k/month for a fixed number of processing slots. You can buy more slots to fit your query load. Anyways the point is that it's well within the financial means of Twitter, especially when compared to developing and operating this proprietary system.
- buremba 6y agoIf I'm not mistaken Twitter is mostly on GCP so they could just go with BigQuery instead of developing an in-house solution. We probably need to ask them in order to get a proper answer. :)