4 ms·
They have >1PB of data to ETL, with some queries hitting 450TB of pure shuffle. It's very true that most users don't need something like BigQuery or Snowflake.
by winwang 1y ago
They have >1PB of data to ETL, with some queries hitting 450TB of pure shuffle.
It's very true that most users don't need something like BigQuery or Snowflake. That's why some startups have come up to save Snowflake cost by "simply" putting a postgres instance in front of it!
In fact, I just advised someone recently to simply use Postgres instead of BigQuery since they had <1TB and their queries weren't super intensive.