5 ms·

I knew my bet to sticking with Postgres would pay off! This looks super exciting. I thought of doing something similar for our data warehouse with AWS Fargate

I knew my bet to sticking with Postgres would pay off! This looks super exciting. I thought of doing something similar for our data warehouse with AWS Fargate and Postgres but the cold starts and limited disk space required too much engineering on top to make it work. Moving to Snowflake comes at the cost of losing so many Posgtres features in exchange for speed. Things like foreign keys, constraints, extensions etc which requires so much engineering to replace in Snowflake. I would be happy to pay 25x the price for a 10x speed increase for a specific query.

thejosh 4y ago
Snowflake is a data warehouse though. Completely different use case. If your data can be done via PG, highly recommend that over SF. Especially with this concept. Snowflake is great when you use a tool like dbt, their modern SQL approach and functions are fantastic. Downsides is it's pretty pricey, and can catch you out.
- iknownothow 4y ago
  We already use DBT and the data is less than 10TB, something Postgres can handle well. And most of the data is concentrated in a few tables. With a serverless approach I'd be happy to allocate 10x resources for just a query or two and for the rest a minimal server is fine. I manage the data warehouse mostly alone because Postgres offers guarantees, unique constraints, triggers and relationships between columns of different tables. It does the work of two engineers. Snowflake is fast but not Postgres compatible. In order to move to Snowflake, I have to write tests and maintain them which Postgres does for me for free. I'd stick with Postgres at least until 20TB before considering Snowflake.
manigandham 4y ago
Snowflake is an OLAP system. It's an entirely different kind of "speed" designed for analyzing vast amounts of data through scans and aggregations.
- iknownothow 4y ago
  Neon now opens the door (at least in my mind) for Postgres to be used for analytics or a data warehouse for almost an order of magnitude more data before having to consider Snowflake. Basically if someone is already using Postgres as a warehouse, then they can prolong their migration to Snowflake by at least a year by using something like Neon.
  - manigandham 4y ago
    Sure but there are plenty of OLAP solutions like Greenplum, and extension-based offerings like Citus and Timescale, that can all partition and scale across nodes to massive datasets with column-oriented storage. AWS Redshift is also built on Postgres (although a much older and customized version).
nikita 4y ago
Thank you! Snowflake is a better cloud data warehouse than Postgres, but of course Postgres is so versatile. Neon will give you some of the Snowflake features: time machine, cloning - we call this branching, data sharing.

thejosh 4y ago

Snowflake is a data warehouse though. Completely different use case. If your data can be done via PG, highly recommend that over SF. Especially with this concept. Snowflake is great when you use a tool like dbt, their modern SQL approach and functions are fantastic. Downsides is it's pretty pricey, and can catch you out.

iknownothow 4y ago
We already use DBT and the data is less than 10TB, something Postgres can handle well. And most of the data is concentrated in a few tables. With a serverless approach I'd be happy to allocate 10x resources for just a query or two and for the rest a minimal server is fine. I manage the data warehouse mostly alone because Postgres offers guarantees, unique constraints, triggers and relationships between columns of different tables. It does the work of two engineers. Snowflake is fast but not Postgres compatible. In order to move to Snowflake, I have to write tests and maintain them which Postgres does for me for free. I'd stick with Postgres at least until 20TB before considering Snowflake.

manigandham 4y ago

Snowflake is an OLAP system. It's an entirely different kind of "speed" designed for analyzing vast amounts of data through scans and aggregations.

iknownothow 4y ago
Neon now opens the door (at least in my mind) for Postgres to be used for analytics or a data warehouse for almost an order of magnitude more data before having to consider Snowflake. Basically if someone is already using Postgres as a warehouse, then they can prolong their migration to Snowflake by at least a year by using something like Neon.
- manigandham 4y ago
  Sure but there are plenty of OLAP solutions like Greenplum, and extension-based offerings like Citus and Timescale, that can all partition and scale across nodes to massive datasets with column-oriented storage. AWS Redshift is also built on Postgres (although a much older and customized version).

nikita 4y ago

Thank you! Snowflake is a better cloud data warehouse than Postgres, but of course Postgres is so versatile. Neon will give you some of the Snowflake features: time machine, cloning - we call this branching, data sharing.