3 ms·
> There are around 29 million 810-by-810 meter grid cells nationwide and more than 1.4 billion rows of data for a single year. Even using the largest database i
by danso 4y ago
> There are around 29 million 810-by-810 meter grid cells nationwide and more than 1.4 billion rows of data for a single year. Even using the largest database instance available on Amazon Web Services, it took up to a week to run queries on the data. Often, our queries took days simply to fail. It was a long, demotivating slog.
> That’s when some colleagues told us about Google BigQuery, which is a Google Cloud services product that allows you to do SQL-style queries on very large data sets. Using BigQuery, code that once took a week to run finished in minutes.
I use BigQuery at work for all my SQL processing, but I'm usually working with data in the 100K to 1M range that take 1 to 10 seconds to process and don't have to think of costs. Does anyone have an idea of what minutes-long queries involving billions of rows would typically cost?