3 ms·
PSA: be very careful with what you query in BigQuery on this dataset! You will be billed $5/TB scanned. I was using this dataset a couple weeks ago for fun, and
by reilly3000 7y ago
PSA: be very careful with what you query in BigQuery on this dataset! You will be billed $5/TB scanned. I was using this dataset a couple weeks ago for fun, and ran up a $60 bill in about 10 mins. It’s important to understand that using WHERE doesn’t shrink the size of your query; instead your FROM clause should only be limited to a single date or month... not ‘*’
- jloveless 7y agoThis is EXACTLY why we moved this data into our own database for Crux.run. We took the JSON export of the Crux database and cleaned and loaded it into a database on a single server. E.g each of those queries are hitting a single box in us central
- jloveless 7y agoThere’s some of that information in the docs : https://docs.crux.run/#/introduction?id https://docs.crux.run/#/introduction?id But yeah- BQ can get very expensive very quickly if miss step
- reilly3000 7y agoThanks for doing so!