2 ms·
About a year ago I spent $80 making a single query using BigQuery. There was a large public dataset and I did a query that spanned several months of data. My qu
by reilly3000 7y ago
About a year ago I spent $80 making a single query using BigQuery. There was a large public dataset and I did a query that spanned several months of data. My query was something like SELECT fields FROM table.* WHERE date = ... the problem was that WHERE still scans data from ALL date partitioned tables then returns a filtered result. What I should have done is FROM table.date - anyhow I am glad I checked the cost before I hit it with more queries. At $5/TB scanned is remarkably cheap for small data but not for large data. In my case it was 16TB.
Still, it was pretty amazing that the query returned data within ~2 seconds.
So be careful kids. Public datasets don’t cost anything to store, but your cloud account is going to get pummeled for your exploratory data analysis and if you haven’t set any billing controls or alerts you’re in for a nasty surprise.