3 ms·

As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The

As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The main problem is most companies need the security of a managed service for cloud computing, and don't want to be locked out of scaling to a very large scale (with distributed compute). Unfortunately, I don't think there are a lot of options that meet those pretty simple requirements that aren't databricks or snowflake[1]. Sure, you could put your compute on single docker containers, but when your data gets too big your stuck and have to get someone expensive in to manage kubernetes, all that time, you can't compute on your data. Which is sorta the crux of it: databricks and snowflake are expensive, but not nearly as expensive as finding out you need them and not having them. [1] if you're using python on was/gcp, I think coiled (coiled.io) is a rare exception to that pattern

eitally 3y ago
I think your second point is vastly overstated for 99% of companies and their use cases. There are precious few situations that can't be handled decently well by a simple RDBMS, and a simple ETL process into an analytical data store for reporting. Almost everything that exist fits one of three buckets: 1) CRUD app, 2) reporting tool, 3) business process app. And almost all of this is for internal use only. It's really only a handful of enterprise things that need massive scalability and performance, and consumer-oriented stuff (much of which is pretty niche and also doesn't need massive global scalability).
- benrutter 3y ago
  I think you're right and I probably came over a bit strong in how likely I made out companies would be to use that level of scale. The point I was trying to make was more that because using something like RDBMS (maybe + VMs or something for data science) has a hard ceiling in terms of scale, even if companies are really unlikely to hit that ceiling, there is a weird incentive structure. The costs of databricks or snowflake are really high, but probably not enough to bankrupt a company or division. If you're a small to midsize company and you realise you can't compute on your data, that could cause much more serious issues. Even if those issues are only feasible 1/100 times, it's a pretty scary risk to expect a CTO to ignore.

eitally 3y ago

I think your second point is vastly overstated for 99% of companies and their use cases. There are precious few situations that can't be handled decently well by a simple RDBMS, and a simple ETL process into an analytical data store for reporting. Almost everything that exist fits one of three buckets: 1) CRUD app, 2) reporting tool, 3) business process app. And almost all of this is for internal use only. It's really only a handful of enterprise things that need massive scalability and performance, and consumer-oriented stuff (much of which is pretty niche and also doesn't need massive global scalability).

benrutter 3y ago
I think you're right and I probably came over a bit strong in how likely I made out companies would be to use that level of scale. The point I was trying to make was more that because using something like RDBMS (maybe + VMs or something for data science) has a hard ceiling in terms of scale, even if companies are really unlikely to hit that ceiling, there is a weird incentive structure. The costs of databricks or snowflake are really high, but probably not enough to bankrupt a company or division. If you're a small to midsize company and you realise you can't compute on your data, that could cause much more serious issues. Even if those issues are only feasible 1/100 times, it's a pretty scary risk to expect a CTO to ignore.