3 ms·
We just ported our ~500gb data warehouse from an 8 core postgres rds instance to bigquery. Rebuilds went from 5 hours to 11 minutes and costs are about the same
by efxhoy 2y ago
We just ported our ~500gb data warehouse from an 8 core postgres rds instance to bigquery. Rebuilds went from 5 hours to 11 minutes and costs are about the same or lower. and that’s with caching of some very big tables in postgres and rebuilding from scratch in bq.
I much prefer postgres as a tool I can run myself but the order of magnitude performance benefits are hard to argue with.
- valyala 2y agoDid you try ClickHouse? It should provide data ingestion and querying performance comparable to Google BigQuery, while it should cost less, since you can run it on your hardware.
- efxhoy 2y agoNo unfortunately not. We tried out snowflake and bigquery and landed on bigquery mainly because of how well it integrated with existing company workflows (a looot of google sheets floating around the org).