3 ms·

You can pretty easily store most of these data in any database. Don't need something like Snowflake - postgres stock can handle about any analytics queries, es

You can pretty easily store most of these data in any database. Don't need something like Snowflake - postgres stock can handle about any analytics queries, especially if you're not writing too much.

ramraj07 6y ago
So the issue is not paying for compute when you're not using it. Firstly I've tried running postgres on a low powered lightsail instance and while it runs, it's kinda insane that it takes a ton of time to load data and then to query it. Indexless scans of data in large tables (let's say a db with all my emails) can take a while! I've tried the same in snowflake, and I'm only using their smallest warehouse, and everything happens in seconds. Loading GBS of data is seconds, scanning GBs of data is seconds. Their smallest compute warehouse is still significantly beefier than the cpu you get with a puny lightsail with a shared vcpu and while it's disk io is shit (since it's reading from S3) the parallelism makes up for it. In the end I think it's the pricing model that makes the difference. With snowflake you seamlessly launch a very powerful machine to run your query and only get billed for a minute, and with running postgres you need to run your machine all the time. Also hot storage (as opposed to s3) is also non trivially expensive. Haven't seen any way to get 200gb of ssd storage without spending that 25 dollar minimum that snowflake costs anyway. This is data I intend to not even update or query more than a few times a month, so I really don't want to pay for compute running all the time in its name.
- edmundsauto 6y ago
  That's a fair point - the 200GB disk costs a bit more if you need an SSD (although my experience shows an SSD is not super necessary unless you have write heavy workloads or otherwise unusual access patterns). If you drop the SSD requirement, a cheap dedicated server will blow Snowflake out of the water. As a bonus, you can run your own code (ETL, scrapers, dataviz). Kimsufi has 4 core 2TB (spinning) w/ 16GB RAM for $17/mo. Personally, that is the route I go for my personal warehouse, as I find I almost always want to put an API or django app in front (and other software like Celery, scrapers, etc.)

You can pretty easily store most of these data in any database. Don't need something like Snowflake - postgres stock can handle about any analytics queries, especially if you're not writing too much.

ramraj07 6y ago

So the issue is not paying for compute when you're not using it. Firstly I've tried running postgres on a low powered lightsail instance and while it runs, it's kinda insane that it takes a ton of time to load data and then to query it. Indexless scans of data in large tables (let's say a db with all my emails) can take a while! I've tried the same in snowflake, and I'm only using their smallest warehouse, and everything happens in seconds. Loading GBS of data is seconds, scanning GBs of data is seconds. Their smallest compute warehouse is still significantly beefier than the cpu you get with a puny lightsail with a shared vcpu and while it's disk io is shit (since it's reading from S3) the parallelism makes up for it. In the end I think it's the pricing model that makes the difference. With snowflake you seamlessly launch a very powerful machine to run your query and only get billed for a minute, and with running postgres you need to run your machine all the time. Also hot storage (as opposed to s3) is also non trivially expensive. Haven't seen any way to get 200gb of ssd storage without spending that 25 dollar minimum that snowflake costs anyway. This is data I intend to not even update or query more than a few times a month, so I really don't want to pay for compute running all the time in its name.

edmundsauto 6y ago
That's a fair point - the 200GB disk costs a bit more if you need an SSD (although my experience shows an SSD is not super necessary unless you have write heavy workloads or otherwise unusual access patterns). If you drop the SSD requirement, a cheap dedicated server will blow Snowflake out of the water. As a bonus, you can run your own code (ETL, scrapers, dataviz). Kimsufi has 4 core 2TB (spinning) w/ 16GB RAM for $17/mo. Personally, that is the route I go for my personal warehouse, as I find I almost always want to put an API or django app in front (and other software like Celery, scrapers, etc.)