21 ms·

Most companies do not need Snowflake or Databricks

179 points by whoiskatrin 3y ago

petetnt 3y ago
Would be really wild if 20%, a truly whopping number, of the companies did need Showflake or Databricks.
- nycdatasci 3y ago
  I think the author means 80% of existing databricks/snowflake customers, since they say “the rest are overpaying”.
code51 3y ago
Which use cases cannot do without Snowflake or Databricks?
- nycdatasci 3y ago
  Zero-copy, zero-etl data sharing.
- iknownothow 3y ago
  Any typical analytics use case, where a single dataset (think a single table) is less than 2TB and the total data size is less than 10-20 TB in the organization. If that's you, you very likely don't need Snowflake and definitely not Databricks. An RDBMS would suffice, I recommend Postgres. These sizes are just my thumb rules. Your mileage may vary. I have people in my network use Snowflake with a total data size of 50GB!!! It makes my blood boil.
  - catchnear4321 3y ago
    years ago, had this one college professor, students would spend the time before class asking him to divide two fractional numbers, and he would rattle off a few decimal places after calculating, to cheers from the students that had their calculators out with the answers. the excitement would build with every correct decimal point. he didn’t actually need a calculator. but he did own one. hopefully this does not cause your blood to go through a state change.
    - code51 3y ago
      Am I right in understanding that all these companies are actually making deals for on-demand use in the future?
      - catchnear4321 3y ago
        you can get started with many of these companies with just a credit card with monthly billing. just like the major cloud vendors.
  - bigbillheck 3y ago
    > I have people in my network use Snowflake with a total data size of 50GB!!! It makes my blood boil. I keep a whole bunch of small-ish datasets in Snowflake because it means I don't have to do all the work to janitor my own postgres install. I go with the db someone else maintains for me that I have, not the one that I wish I had.
kfk 3y ago
Agreed, not completely on duckdb but we used it for consolidating billing data from 10+ ERP systems and it works, so I see his point. Just to add to his points: - Integrations are still one of the hardest things in Enterprise IT. Snowflake/Databricks/etc in fact add to the number of systems to integrate, they make this problem worst most of the times - Governance in a self-service data ecosystem gets complicated fast, especially if you need to stay compliant with data privacy, gdpr, etc.. And amazingly again neither Snowflake nor Databricks solve this. In fact, they make it worst by sucking up budget away from governance initiatives
- polskibus 3y ago
  I guess the ETL orchestration was the hard part? How did you do it?
  - kfk 3y ago
    Connectivity was the hardest part, we had to write Python connectors for a variety of ERP systems. We have a 1 server setup where we run duckdb and Python scripts, we monitor and orchestrate with Prefect but you can use one of the many Python orchestration tools. We load the finished data marts to a MS SQL server and users connect to it via PowerBI or Excel.
    - polskibus 3y ago
      I don't understand why would you use duckdb as an intermediate step to fill Ms SQL DW. Surely you could just go to Ms SQL directly?
      - chrisjc 3y ago
        I'm sure DuckDB was used to transform/prepare the data from the ERP extract format to an MSSQL ingestion format. There are plenty of arguments and reasons why you would use DuckDB to do this esp if you're preparing the data for Analytical/OLAP use-cases. Perhaps a more relevant question might be why they didn't use DataFactory or some other ETL tool/service. DuckDB is rising the occasion for these kinds of use-cases though.
tremon 3y ago
Of course they do not need it. 80% of companies can also do without AWS, they "just" need to hire the people themselves, and developer a core competency running and administering those services. It doesn't automatically mean that it makes financial sense for said company. I guess "don't need" == "must not use" when you're selling yourself as filling that self-hosted gap.
- toyg 3y ago
  This is one of the things I hate of this industry: service providers oversell their proposition and overcharge to infinity, while independent professionals often undersell that same proposition so that they can capture some of that overcharging for themselves. If you are an honest actor, you're left pulling your hair out at the overall mendacity of the whole market. Profit can be such a cancer.
nottorp 3y ago
99.99% of companies are not Google sized and don't need Google's solutions. Period.
- marginalia_nu 3y ago
  Google-sized solutions tend to create the need for google sized solutions though. Take a basic CRUD TODO app. Add metrics to every user interaction, every mouse movement. Now you need a time series database, kafka and a data lake. Break it down into microservices. Now you need kubernetes, and also structured logging, and probably an ELK stack. You also probably need Graphana to monitor your services. Because you have introduced hundreds of failure points, you also need high-availability, multiple regions, elastic scaling. Eventual consistency.
  - marginalia_nu 3y ago
    Thinking about it, this may be a corollary of Parkinson's law.
    - eitally 3y ago
      It definitely is. It also sort of applies to legacy enterprise, too, in a different flavor: "If the tool or system is easy to use and accessible (thing classic LAMP stack), then shadow IT is guaranteed to exist and start using it." The more modern version flavor is that "if a cloud service exists, then your enterprise architecture will expand to consume it [whether it's necessary or not]."
anonzzzies 3y ago
We make a product that smaller companies use; it’s much simpler to use and no consultants needed. We notice that even in bigger corps, departments import into our system and use that because they find the de facto systems painful to use and they don’t actually need that much power, ever.
dna_polymerase 3y ago
73.6% of all statistics are made up
- anonzzzies 3y ago
  Just like the statistics in the title; in reality that number is vastly higher.
- Spooky23 3y ago
  9/10 dentists prefer snowflake.
deleted 3y ago
[deleted]
totetsu 3y ago
For the non data experts, from elsewhere on medium: “ Semi-structured data tends to evolve over time. Systems that generate data add new columns to accommodate additional information, which requires downstream tables to evolve accordingly. The structure of tables in Snowflake can evolve automatically to support the structure of new data received from the data sources” “When dealing with large datasets, the processing power of individual machines can become limiting, necessitating the use of distributed and parallel processing capabilities provided by platforms such as Databricks“
joduplessis 3y ago
heresy!
YetAnotherNick 3y ago
Managing data yourself is hard. It is hard not because it can't scale, but for more boring reasons like harder DB upgrades and migrations, snapshotting, access control, lacking libraries/SDK, lack of documentation, harder training of new employees etc. The number one reason for extended downtime I have seen in companies is that data is in some bad state. And good data engineers who could do all the things authors expects them to do are not easy to find and expensive.
- underdeserver 3y ago
  Yeah. This post reads like the Dropbox comment. I've worked with ad-hoc data solutions, and now I work with analysts who swear by Snowflake. It's the organization, the administration, the maintenance that Snowflake is solving for us, and it's turn-key. It's not easy or cheap to get that done with a local solution.
krasznahorkai 3y ago
I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny kickbacks, but who knows? For all my clients Snowflake is overkill when you look from the perspective of growth and scale. They'll never use that part of Snowflake. They might just do as well with DuckDB, Azure Synapse or any other analytical-oriented platform laying around. What I do like with less-than-big use-cases is that (at least at Snowflake) you pay relatively little if you do relatively little data processing. It's not free, but it doesn't break the bank either.
- FridgeSeal 3y ago
  > and at least in my circles many Snowflake sales employees are ex-Oracle. given my previous experiences with Snowflake sales, this would not surprise me in the slightest. A small list of the incredibly pushy things SF sales teams did: - called up junior team members and pushed them to sign a renewal deal - point blank refused to explain how credits translated to real world money when pressed about how much usage we were actually getting for our money - after being told not to, they continued to call team members and pressured them to commit to a renewal - TAC/support gave consistently vague and unhelpful explanations. So yeah, Snowflake is an awful company and an overpriced product, and I hope it fails.
  - catchnear4321 3y ago
    sorry, but, there’s something about this that doesn’t smell right. junior ic being asked to sign a contract makes zero sense. that’s insanity. plenty of managers at plenty of companies lack that level of authority. sales teams use slimy tactics. that’s a given. time limits, multi-years, etc, etc, etc. trying to get the wrong person to sign is a different category entirely. maybe you ran across the “best of the best” sales people there, but, this sounds sufficiently outlandish that i have to question it a bit.
dickersnoodle 3y ago
Word. Most companies don't have nearly enough data to begin to justify them.
intellectronica 3y ago
True from a purely technical point-of-view, but doesn't take into account how companies make decisions about adopting platform tech and that they're making these choices for good reasons. When companies choose SF or DB they don't make a decision to install one of these platforms because there is no combination of open-source components and bespoke engineering work that could replace them and even be more flexible and efficient. They are choosing these platforms because they make it easy to manage data operations holistically, because it's easier to hire people who already know them well, because they are opinionated and restrict the range of bad or just strange decisions someone in the organisation could be making if they were not restricted, and of course for enterprise support, available from a single reputable vendor. See also: 80% of companies don't need Kubernetes, 80% of companies don't need a big-3 cloud, 80% of companies don't need managed security solutions, 80% of companies don't need SAP, 80% of companies don't need Salesforce, etc etc etc...
- ptero 3y ago
  Many companies also install DB, k8, etc. because costs don't matter as much. In the times of easy money they focus on running fast, borrowing if needed. When money is harder to borrow, as it is now, some of those decisions get reassessed. My 2c.
  - catchnear4321 3y ago
    cost of migration is part of any assessment worth anything. rarely cheap, rarely on time. not that some companies won’t bleed themselves out in a panicked effort to try and stop a paper cut.
benjaminwootton 3y ago
I would turn this on it’s head and say that most companies do need one or the other. My first premise is that most companies will have a BI or Data and Analytics problem, whether it’s analysing their spend, revenue, operations, customer churn or something more interesting. At that point, having an industry standard, fully managed, fully elastic and resilient platform with consumption based pricing sounds pretty appealing. Yes I can run and administer a warehouse on EC2, but the total cost of manpower and servers with full resilience is going to be high, especially as you’ll have to add in analytics tools, ETL tools etc which Snowflake or Databricks might have included. I’m a huge believer in both Snowflake and Databricks. Snowflake for BI and Databricks for anything more funky. The technology is on point and the business case stacks up for the most part.
lr4444lr 3y ago
...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. This reads like a flavor of the same argument I've heard many times over. You know what the biggest cost of an org. really is as we head into a recession? Labor. Stop wasting it on stuff done better by specialized companies that free you up to get stuff done.
- rglullis 3y ago
  > we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid of single vendor lock-in.
  - bushbaba 3y ago
    Would it be cheaper? Cloudera fits the bill and it’s for sure not cheaper
    - rglullis 3y ago
      Because Cloudera is aimed at a market that is not the same that would benefit from "most companies" being talked about in TFA? Plenty of companies that would be fine by using a managed postgresql database for their "data warehouse" and you can get, e.g, one reasonably powerful server at Digital Ocean for $60/month with managed backups - and that already has quite a big markup compared to their droplets because they are just packaging some monitoring tools on top of their existing systems.
  - hodgesrm 3y ago
qrios 3y ago
> The Fortune 100 have a use case for these companies, the rest are overpaying. There is a difference between "a use case for $data_platform" and "a data use case for $data_platform". Scope on the first one is the platform in $data_platform" and second is the data specifics requirements in $data_platform. Working on a non-Fortune100 insurance company in Europe, almost all our use cases can be easily done on traditional RDBMS like SQL Server, or on BI tools like SAS. Thankfully with higher granularity over time, Excel usage is fading out constantly. No big data, no heavy computing necessary - at least from my point of view. All setups in place today, can be called self-service platforms. With cautious estimations we have at least 100 of such "platforms" running since years, or even decades. This situation implies direct, we have a use case for a $data_platform itself. Costs are the biggest driver here, mainly due to the hidden costs of keeping these 100 systems up and running. Governance and management of the data, locked-in in all these stores, processed by slow and ugly SQLs nobody understands anymore, and with an unknown state of data quality, is key today.
catchnear4321 3y ago
> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management understand. justification is part of the role. how long does that take, and how much effort? (the answer is subjective and contextual.) to be blunt, this kind of advice is barely more than “do better.” it ignores the situational example of using snowflake cheaply. it acts like most devs aren’t just going to go fire up a postgres rds - you said open source! it oversimplifies all problems, implicitly, to you. > Pay to make a problem disappear. no, pay to change the parameters of the problem. this is a fundamental misunderstanding of how to get things done in a constrained environment. it isn’t either/or, and every saas comes with problems. you pay to trade problems. otherwise it wouldn’t be worth a blog post name dropping and shitting on databricks and snowflake costs. in the hands of an untrained user or in a sufficiently constrained environment, they cost a lot - that’s one of the problems you buy. cost management. > a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations again, hope you’ve taken the time to build these bridges. the author does indicate this is most important. and it has a cost. someone is building that bridge, and likely someone from each side. you sometimes pay vendors to deal with technical problems so you can deal with non-technical ones. most importantly, don’t assume a random stranger on the internet has sufficient context to give you worthwhile recommendations.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- pydry 3y ago
  >open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? Funny. I heard this same kind of argument used against replacing oracle with postgres. It reminds me of Microsoft's "TCO" PR offensive back when they were public about how much they loathed open source competition. Thing is that it wasnt just a straw man (nobody that needs to be convinced is under any illusions that software has to be maintained), Oracle was also way more expensive to maintain ON top of being expensive to run. They had their hooks into that organization pretty good though and once that happens technology choices become highly political - which experts are you going to fire and which ones are you going to hire? The oracle lot obviously didnt want it to be them that get managed out.
mrweasel 3y ago
Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine. Once you hit the scale where you have "enough" data I'd agree with many of the other comments: Managing open source or home grown solutions quickly become more expensive than just paying for a service. Not quite the same, but we considered deploying a few solutions on OpenStack, but once you paid for training and staff it turned out to be cheaper to just give VMWare more money.
- Aurornis 3y ago
  > We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine. I did some work with a company that talked about their “big data”, bragged about their data lake, and made grandiose statements about how they had big plans to use “all this data” to get an edge up on their competitors. They talked about the big data platforms and technologies they were evaluating and how they’d probably have to hire more people to handle it all. When I pressed them for details on it, they eventually explained the data they had on each customer and the approximate number of customers. I did some quick math and calculated the total amount of data on all of their customers was under 100MB. They had managed to blow it up to occupy a couple orders of magnitude more space in one situation (think storing time series data as JSON where every point had duplicated tags attached due to ineffective structure) but even that was small enough to fit entirely in RAM of a decent sized server.
  - peteradio 3y ago
    Remember some industries are perversely incentivized to drive up costs because profits are limited as a percentage of total cost.
willvarfar 3y ago
My experience is that there is a lot of milage for small and medium enterprises to use your normal RDBMS replication to create a copy of your OLTP DB that drives your business and run your analytics on that. And put up with complaining it can be slow to do inefficient analytics queries. Really, after spending silly amounts of time and hassle (and money) on the fancy snowflake or whatever, you discover its not massively faster for those small and medium businesses. And now you have to keep paying for it and keep maintaining it, which is a often actually a bigger burden than keeping a normal RDBMS replica alive.
- ozim 3y ago
  I think "put up with complaining it can be slow to do inefficient analytics queries" is why I would rather pay for some 3rd party tool that would get the complaints. Business people want some queries that are slow no matter what you do. Then expect these to run in under a second. They don't care about some abstract things like operations complexity or big O - they want it here and now. Even if your data fits on one server there are still queries that will take time and it is not going to be instant, even if you put whole db in RAM it is going to be fast but still I can imagine bunch of queries that will simply have to take couple of minutes anyway.
jojobas 3y ago
Most people shouldn't need a 3GHz 8-core CPU in their pocket to look at cats or check weather, but here we are. Simplicity by bloat come at a price.
iou 3y ago
https://archive.is/xNT28 https://archive.is/xNT28
HorizonXP 3y ago
My client is in the middle of migrating their Snowflake data to Databricks Unity Catalog. To this day, I still do not understand why they love Databricks so much. It just looks like a Jupyter notebook to me. I know, it has Spark. So? I’m trying to learn more so I can build something better, maybe.
- its_a_random_ac 3y ago
  Yeah, I'd like to know too, from everything I can see it's just a cloud-hosted notebook with a spark backend, which does have some benefits, but I can't tell why they got as big as they did. Also from a more cynical side, I've interviewed at a lot of companies that ask if I've used it before, and I'm really temped to just lie, since it feels like something that could be picked up really easily.
benrutter 3y ago
As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The main problem is most companies need the security of a managed service for cloud computing, and don't want to be locked out of scaling to a very large scale (with distributed compute). Unfortunately, I don't think there are a lot of options that meet those pretty simple requirements that aren't databricks or snowflake[1]. Sure, you could put your compute on single docker containers, but when your data gets too big your stuck and have to get someone expensive in to manage kubernetes, all that time, you can't compute on your data. Which is sorta the crux of it: databricks and snowflake are expensive, but not nearly as expensive as finding out you need them and not having them. [1] if you're using python on was/gcp, I think coiled (coiled.io) is a rare exception to that pattern
- eitally 3y ago
  I think your second point is vastly overstated for 99% of companies and their use cases. There are precious few situations that can't be handled decently well by a simple RDBMS, and a simple ETL process into an analytical data store for reporting. Almost everything that exist fits one of three buckets: 1) CRUD app, 2) reporting tool, 3) business process app. And almost all of this is for internal use only. It's really only a handful of enterprise things that need massive scalability and performance, and consumer-oriented stuff (much of which is pretty niche and also doesn't need massive global scalability).
  - benrutter 3y ago
    I think you're right and I probably came over a bit strong in how likely I made out companies would be to use that level of scale. The point I was trying to make was more that because using something like RDBMS (maybe + VMs or something for data science) has a hard ceiling in terms of scale, even if companies are really unlikely to hit that ceiling, there is a weird incentive structure. The costs of databricks or snowflake are really high, but probably not enough to bankrupt a company or division. If you're a small to midsize company and you realise you can't compute on your data, that could cause much more serious issues. Even if those issues are only feasible 1/100 times, it's a pretty scary risk to expect a CTO to ignore.
bjornsing 3y ago
Considering what you get I think Snowflake is rather decently priced. It’s not easy to integrate and operate a bunch of open source tools that replaces it, especially not on a small scale. Also, in most cases you’ll pay a lot more for the people that use Snowflake than you do for Snowflake, so it makes more sense to focus on productivity.
jboggan 3y ago
Modifying the argument, I'd say even the companies that do need Snowflake or Databricks for an analytical use case tend to get run over by the hype train post adoption and start using it everywhere as an anti-pattern and causing much bigger problems operationally.
- ledauphin 3y ago
  100%. as an organization, there's value in _limiting_ the number of tools you adopt. But you'll nearly always want to adopt a few right-sized tools rather than using the same sledgehammer for every single thing that looks remotely like a nail.
wnolens 3y ago
No one needs to evaluate all options against their current/known/possible needs when they can just pick a well vetted product and get on with business.
tootie 3y ago
> The global economy is headed for a recession. That’s not my opinion, that’s the Federal Reserve’s I know that's not the topic of the post but OP shows a lack of comprehension here. The Fed warned because that's what they do when the risk is non-negligible. They never say it is definitely going to happen because nobody knows.
deleted 3y ago
[deleted]
mxxc 3y ago
i am not sure why everyone (comments, this post, etc.) assume there is a one-size-fits-all solution to every problem, even this one that looks quite simple. companies have to align a few things, ranging from the skills of the current employees, hiring plans, investor/shareholder management, or how to make sure the CEO really gets that boat he really wants and deserves. not all businesses are the same. businesses with fat contracts but few users won't have massive operating costs, and they can use whichever easy and non-scalable technology they want, because once the business scales up, those fat contracts will pay for enough data engineers. a gaming startup will face high server costs right away, without any optimisation, while data platforms (e.g. bigquery) with a tiny bit of optimisation (materialising 2-3 summary tables, for example) will bring the cost down to "laughable" pretty easily. it is true that many of these things are choices, e.g. do you really want to spend a shit ton of money for looker when superset for most users is just as good? are you even able to make that choice? if these choices are hard to make because a potential user (or set of users) in the company really wants something instead of something else, well, that is not a technical choice, and the issue you have has nothing to do with the technology.
MrPowers 3y ago
I think most companies don't understand a large part of the Databricks offering and it should be used by way more organizations. Disclaimer: I was a Databricks user for 6 years and now work at Databricks. Yea, you can create your own Spark deployment, but it will run much slower than the Databricks Runtime (DBR) or the Databricks proprietary Spark Runtime (Photon). Computations that run slower cause you to have a larger cloud compute bill. Databricks rewrote Spark in C++ and it runs really fast and saves a lot on ec2 compute. > Define when you should compact files, when to Z-order Or don't consider these issues and use autocompaction / the new Liquid clustering. These are great examples of problems the platform should solve, so the user has time to focus on business logic. > If you can sniff out the inefficiencies in your Data early and make architecture that handles your specific data I don't know what this means. Are you going to build a deep learning model to make read/writes faster like Databricks predictive I/O? https://docs.databricks.com/en/optimizations/predictive-io.html https://docs.databricks.com/en/optimizations/predictive-io.h.... Probably not, you have a lot of business problems to solve. > Do the real work. Work with people. The Code will write itself. I've seen lots of DIY data platforms. They're horrible to work with and I can assure you that the code does not write itself. The data engineers have a lot less time to write code because they're constantly trying to stand the platform back up.
- spmurrayzzz 3y ago
  > Are you going to build a deep learning model to make read/writes faster like Databricks predictive I/O? It makes sense for Databricks to do this because they're building a product that needs to work for a high cardinality of datasets. Using a DNN for that is defensible because the input shapes are practically infinite. But for individual orgs, it seems much more likely that simple heuristics-driven access pattern optimizations can be done without throwing ML at the problem (though I'll say the predictive IO concept is a cool one, I've done similar ML work for network traffic QoS). > I've seen lots of DIY data platforms. They're horrible to work with and I can assure you that the code does not write itself. This one made me scratch my head a bit, because I have seen many DIY platforms as well over a 20 year career. Most of them have been amazing to work with. The code didn't write itself of course, but maintenance burdens were low and specialization/expertise within the org was high as a result. On top of that, I didn't need to argue with an AE about rising costs on a per-annum basis whenever renewal time comes around. I point this out not in the service of coloring your observations as wrong or misguided, but to highlight that there's going to be a spectrum of varied lived experiences people have with the build-vs-buy conundrum. I expect we'll never see genuine consensus on this issue as a result of that variance (and maybe we don't have to).
thiago_fm 3y ago
The thing is, you don't understand that a business needs to stay focused and that has a price. When you buy those solutions, you are paying for the price of staying focused. If the price for staying focused is $X, and X has a positive business impact and need, like Snowflake or Databricks could. And it doesn't mean you won't need to spend on HR, developers, management and so on to run your own OSS solution. It is actually great to give Snowflake and/or Databricks your money. It's a really expensive service, with a huge markup. That I understand. But the alternative doesn't look good at all. A company is better staying focused. Also, if the company sees that it expends a considerable amount of its budget in Snowflake/Databricks, they'll find solutions, that could be negotiating with them, figuring out how to optimize their use etc. I've worked and seen many companies optimizing their cost structure and run away from Datadog/Newrelic, Snowflake/Databricks and just fail miserably. The alternatives made developers simply use it less, and also spend more time to do the same that they did. They even sometimes call it successful, execs point out the money saved, but they don't see the subjective part: developers being less efficient, wasting their time with BS. There's a long tail of costs related to inefficient systems, it's wasting developer hours, that you need to manage, hire, train... Not to forget the burden that it is to fire or make people redundant, companies end up losing so much of their culture and soul doing such things. And this is what you need to typically do if you assume those expenses of running things for yourself. The ideal company are platforms that pays for many SaaS and stay focused on its core business, if it can make money, great! There is so much literature on cost optimization in business in general, stuff that have been studied for over a century. If people would read on that, they wouldn't be repeating such nonsense as this article. Also, the article says that you are OVERPAYING for it if you aren't a Fortune 100. It's much more like the opposite, if you have a small company with a small amount of data, it's very likely that Snowflake or Databricks won't cost you that much. Based also on the other comments on HN, I can only conclude that this is one of those articles that is so full of mistakes and plainly wrong, that there isn't even a real debate over here, just people saying how wrong it is.
datadrivenangel 3y ago
The dirty secret of modern data management is that very few people really have big enough data to justify major amounts of data infrastructure.
commandlinefan 3y ago
> charging their customers for Ferraris I've seen this sort of analogy before and I don't think it's the right one here: if I get a Ferrari, I know _exactly_ what I'm getting. I'm getting a really fast, really beautiful car. I'm not too familiar with Snowflake, but I've suffered under Databricks on and off for about a decade now, and as far as I can tell, it's just a more expensive, closed way to do what I could do a lot faster and a lot easier if I didn't have to work around the obstacles that Databricks puts in my way that don't have any value other than being something that Databricks can charge my employer a ton of money for putting in my way.
- onlyrealcuzzo 3y ago
  Theoretically, your employer is paying Databricks, because they'd rather pay them than have a harder time finding employees with actual skills (and paying them accordingly). Additionally, it's somewhat of a standard now - so it's easy to hire other people in the org with skills using it.
  - orochimaaru 3y ago
    But isn’t that something they would need to do anyway - find employees with actual skills? I mean sure, it’s how the snake oil is sold - “buy so and so saas product and you wouldn’t need to hire and can reduce cost”. I’m surprised that line can still sell today because in software it’s rarely true unless you can merge excellent talent with said product.
    - onlyrealcuzzo 3y ago
      If you work at a small startup - they are mostly parroting what large and mid-size companies are doing - regardless of the fact that they're likely to fail before they get to that size, and doing said thing might make their chances of failure higher. If you work at a large company - having a larger pool of candidates to hire from is definitely worthwhile. And, it is much easier to have a poor understanding of Databricks than a very good understanding of data engineering in general. Someone with a poor understanding of Databricks might be able to do 80% of what you can do with a great understanding of data engineering in general.
bob1029 3y ago
Why not give in and pay for the SQL Server Hyperscale instance? Isn't there enough BS to worry about? Why continue to waste time on tired old OLTP/OLAP/scalability/etc. conversations in 2023? Unless your business can continuously write >100 megabytes of transactional data per second, this solution would almost certainly address all of your needs forever and ever. Up to 100TB too. It just works. It offers transactions exactly the way most business expect them to be conducted. No weird code, no weird client libraries, nothing. It works more or less like it has for the last 30+ years. I can tell you for a fact that simply setting up a gigantic shared database called "company" and getting the team connected to it did wonders for us. When you stop worrying about "will it scale" you can start to collaborate and do amazing things again.
- bick_nyers 3y ago
  If you do magically hit the "unicorn vector" and 100x your customer-base then you can start worrying about "will it scale", and thankfully you will have new revenue that can be deployed in the form of new hardware and new hires.
rmbyrro 3y ago
Most companies don't use Snowflake or Databricks
adam_oxla 3y ago
While I do agree that for most of companies those large SaaS solutions are overkill I do not think that DuckDB or similar is sufficient. Nowadays more of the companies really need to process large datasets. I meet regularly companies that used PostgreSQL or something similar up to some point but then they have grown and it is not sufficient anymore. They need something scalable. It does not have to be large SaaS: in many cases small Clickhouse cluster is sufficient. Nevertheless not everything can be done using single server. Also even if customer knows exactly well what are their needs right now does needs will grow and change over time so it is reasonable to build something that is not only good enough for now. Of course building something absolutely "future proof" leads to extremes and high bills.
slotrans 3y ago
Basically correct. If you have less than tens-of-terabytes the big exciting stuff just isn't needed and isn't particularly helpful. Focus on understanding your data and building a useful model. Use Postgres if you can. Supplement with DuckDB and/or ClickHouse as needed. Or, use your cloud's columnar DB (Redshift, BQ, Azure whatever) because you can start and stop using it as you please without talking to sales or signing a contract. If your data team doesn't have the requisite skills... well... consider that tech selection might not be your problem.
savrajsingh 3y ago
This reminds me of something I've seen too -- companies paying for enterprise wordpress (>$25k a year) when they could easily have say a $100 wordpress plan with all the features they need, behind free or cheap cloudflare.
pradeepchhetri 3y ago
I would consider following things before selecting a database offering: - Whether the database vendor is a lock-in. It would be a straight "NO" for me if the database isn't open-source with proper license, since I can't self-host it in case, I need to move away from their SaaS offering due to various unexpected foreseen reasons: vendor decided to increase pricing, vendor has hidden pricing, vendor has reliability issues etc etc. - How big is the community behind the database. Check their public forums to understand how community feels about the database and how their requirements are considered. - Don't believe any random benchmarking post online but by doing my own benchmarking for my use-case. - Check which other companies have adopted that database and read their experiences.
ouraf 3y ago
That's common practice with mid sized companies and above, though: - find something that can probably give some competitive advantage or please shareholders - Identify it as "not part of our core business" - Pay a third party company to do it even if it would be cheaper to things in house. If something goes wrong and the shareholders ask questions, the CEO can blame them instead of their own IT department and reassure them they're "focusing on what generates most value for the company".
tjhunter 3y ago
This article has valid points but does not understand the perspective of companies. Companies do not buy technologies. Companies buy solutions. - Companies do not buy Spark, they buy the ability to process their data and to have multiple personas collaborate (data scientists, data engineers, ...) - You can do it yourself. It will be cheaper but it will require time, expertise and money, all things that companies do not give easily - Snowflake and Databricks are elastic: you can start small and grow as you need. This is much easier than justifying the upfront cost of hiring specialized people or asking for trust that your ad-hoc solution will respect whatever enterprise governance rules (disclaimer: I worked at Databricks for 6 years and talked to hundreds of prospect and actual Databricks users and customers)

179 points by whoiskatrin 3y ago

petetnt 3y ago

Would be really wild if 20%, a truly whopping number, of the companies did need Showflake or Databricks.

nycdatasci 3y ago
I think the author means 80% of existing databricks/snowflake customers, since they say “the rest are overpaying”.

code51 3y ago

Which use cases cannot do without Snowflake or Databricks?

nycdatasci 3y ago
Zero-copy, zero-etl data sharing.
iknownothow 3y ago
Any typical analytics use case, where a single dataset (think a single table) is less than 2TB and the total data size is less than 10-20 TB in the organization. If that's you, you very likely don't need Snowflake and definitely not Databricks. An RDBMS would suffice, I recommend Postgres. These sizes are just my thumb rules. Your mileage may vary. I have people in my network use Snowflake with a total data size of 50GB!!! It makes my blood boil.
- catchnear4321 3y ago
  years ago, had this one college professor, students would spend the time before class asking him to divide two fractional numbers, and he would rattle off a few decimal places after calculating, to cheers from the students that had their calculators out with the answers. the excitement would build with every correct decimal point. he didn’t actually need a calculator. but he did own one. hopefully this does not cause your blood to go through a state change.
  - code51 3y ago
    Am I right in understanding that all these companies are actually making deals for on-demand use in the future?
    - catchnear4321 3y ago
      you can get started with many of these companies with just a credit card with monthly billing. just like the major cloud vendors.
- bigbillheck 3y ago
  > I have people in my network use Snowflake with a total data size of 50GB!!! It makes my blood boil. I keep a whole bunch of small-ish datasets in Snowflake because it means I don't have to do all the work to janitor my own postgres install. I go with the db someone else maintains for me that I have, not the one that I wish I had.

kfk 3y ago

Agreed, not completely on duckdb but we used it for consolidating billing data from 10+ ERP systems and it works, so I see his point. Just to add to his points: - Integrations are still one of the hardest things in Enterprise IT. Snowflake/Databricks/etc in fact add to the number of systems to integrate, they make this problem worst most of the times - Governance in a self-service data ecosystem gets complicated fast, especially if you need to stay compliant with data privacy, gdpr, etc.. And amazingly again neither Snowflake nor Databricks solve this. In fact, they make it worst by sucking up budget away from governance initiatives

polskibus 3y ago
I guess the ETL orchestration was the hard part? How did you do it?
- kfk 3y ago
  Connectivity was the hardest part, we had to write Python connectors for a variety of ERP systems. We have a 1 server setup where we run duckdb and Python scripts, we monitor and orchestrate with Prefect but you can use one of the many Python orchestration tools. We load the finished data marts to a MS SQL server and users connect to it via PowerBI or Excel.
  - polskibus 3y ago
    I don't understand why would you use duckdb as an intermediate step to fill Ms SQL DW. Surely you could just go to Ms SQL directly?
    - chrisjc 3y ago
      I'm sure DuckDB was used to transform/prepare the data from the ERP extract format to an MSSQL ingestion format. There are plenty of arguments and reasons why you would use DuckDB to do this esp if you're preparing the data for Analytical/OLAP use-cases. Perhaps a more relevant question might be why they didn't use DataFactory or some other ETL tool/service. DuckDB is rising the occasion for these kinds of use-cases though.

tremon 3y ago

Of course they do not need it. 80% of companies can also do without AWS, they "just" need to hire the people themselves, and developer a core competency running and administering those services. It doesn't automatically mean that it makes financial sense for said company. I guess "don't need" == "must not use" when you're selling yourself as filling that self-hosted gap.

toyg 3y ago
This is one of the things I hate of this industry: service providers oversell their proposition and overcharge to infinity, while independent professionals often undersell that same proposition so that they can capture some of that overcharging for themselves. If you are an honest actor, you're left pulling your hair out at the overall mendacity of the whole market. Profit can be such a cancer.

nottorp 3y ago

99.99% of companies are not Google sized and don't need Google's solutions. Period.

marginalia_nu 3y ago
Google-sized solutions tend to create the need for google sized solutions though. Take a basic CRUD TODO app. Add metrics to every user interaction, every mouse movement. Now you need a time series database, kafka and a data lake. Break it down into microservices. Now you need kubernetes, and also structured logging, and probably an ELK stack. You also probably need Graphana to monitor your services. Because you have introduced hundreds of failure points, you also need high-availability, multiple regions, elastic scaling. Eventual consistency.
- marginalia_nu 3y ago
  Thinking about it, this may be a corollary of Parkinson's law.
  - eitally 3y ago
    It definitely is. It also sort of applies to legacy enterprise, too, in a different flavor: "If the tool or system is easy to use and accessible (thing classic LAMP stack), then shadow IT is guaranteed to exist and start using it." The more modern version flavor is that "if a cloud service exists, then your enterprise architecture will expand to consume it [whether it's necessary or not]."

anonzzzies 3y ago

We make a product that smaller companies use; it’s much simpler to use and no consultants needed. We notice that even in bigger corps, departments import into our system and use that because they find the de facto systems painful to use and they don’t actually need that much power, ever.

dna_polymerase 3y ago

73.6% of all statistics are made up

anonzzzies 3y ago
Just like the statistics in the title; in reality that number is vastly higher.
Spooky23 3y ago
9/10 dentists prefer snowflake.

deleted 3y ago

[deleted]

totetsu 3y ago

For the non data experts, from elsewhere on medium: “ Semi-structured data tends to evolve over time. Systems that generate data add new columns to accommodate additional information, which requires downstream tables to evolve accordingly. The structure of tables in Snowflake can evolve automatically to support the structure of new data received from the data sources” “When dealing with large datasets, the processing power of individual machines can become limiting, necessitating the use of distributed and parallel processing capabilities provided by platforms such as Databricks“

joduplessis 3y ago

heresy!

YetAnotherNick 3y ago

Managing data yourself is hard. It is hard not because it can't scale, but for more boring reasons like harder DB upgrades and migrations, snapshotting, access control, lacking libraries/SDK, lack of documentation, harder training of new employees etc. The number one reason for extended downtime I have seen in companies is that data is in some bad state. And good data engineers who could do all the things authors expects them to do are not easy to find and expensive.

underdeserver 3y ago
Yeah. This post reads like the Dropbox comment. I've worked with ad-hoc data solutions, and now I work with analysts who swear by Snowflake. It's the organization, the administration, the maintenance that Snowflake is solving for us, and it's turn-key. It's not easy or cheap to get that done with a local solution.

krasznahorkai 3y ago

I work as a solution architect at a consulting firm that builds analytical data platforms for customers. Our company has a partnership with Snowflake, which means all the solutions we build are pushed to use Snowflake. Their sales strategy is very Oracle-like and at least in my circles many Snowflake sales employees are ex-Oracle. This means our sales and Snowflake sales are the best of friends. Formally they'll deny kickbacks, but who knows? For all my clients Snowflake is overkill when you look from the perspective of growth and scale. They'll never use that part of Snowflake. They might just do as well with DuckDB, Azure Synapse or any other analytical-oriented platform laying around. What I do like with less-than-big use-cases is that (at least at Snowflake) you pay relatively little if you do relatively little data processing. It's not free, but it doesn't break the bank either.

FridgeSeal 3y ago
> and at least in my circles many Snowflake sales employees are ex-Oracle. given my previous experiences with Snowflake sales, this would not surprise me in the slightest. A small list of the incredibly pushy things SF sales teams did: - called up junior team members and pushed them to sign a renewal deal - point blank refused to explain how credits translated to real world money when pressed about how much usage we were actually getting for our money - after being told not to, they continued to call team members and pressured them to commit to a renewal - TAC/support gave consistently vague and unhelpful explanations. So yeah, Snowflake is an awful company and an overpriced product, and I hope it fails.
- catchnear4321 3y ago
  sorry, but, there’s something about this that doesn’t smell right. junior ic being asked to sign a contract makes zero sense. that’s insanity. plenty of managers at plenty of companies lack that level of authority. sales teams use slimy tactics. that’s a given. time limits, multi-years, etc, etc, etc. trying to get the wrong person to sign is a different category entirely. maybe you ran across the “best of the best” sales people there, but, this sounds sufficiently outlandish that i have to question it a bit.

dickersnoodle 3y ago

Word. Most companies don't have nearly enough data to begin to justify them.

intellectronica 3y ago

True from a purely technical point-of-view, but doesn't take into account how companies make decisions about adopting platform tech and that they're making these choices for good reasons. When companies choose SF or DB they don't make a decision to install one of these platforms because there is no combination of open-source components and bespoke engineering work that could replace them and even be more flexible and efficient. They are choosing these platforms because they make it easy to manage data operations holistically, because it's easier to hire people who already know them well, because they are opinionated and restrict the range of bad or just strange decisions someone in the organisation could be making if they were not restricted, and of course for enterprise support, available from a single reputable vendor. See also: 80% of companies don't need Kubernetes, 80% of companies don't need a big-3 cloud, 80% of companies don't need managed security solutions, 80% of companies don't need SAP, 80% of companies don't need Salesforce, etc etc etc...

ptero 3y ago
Many companies also install DB, k8, etc. because costs don't matter as much. In the times of easy money they focus on running fast, borrowing if needed. When money is harder to borrow, as it is now, some of those decisions get reassessed. My 2c.
- catchnear4321 3y ago
  cost of migration is part of any assessment worth anything. rarely cheap, rarely on time. not that some companies won’t bleed themselves out in a panicked effort to try and stop a paper cut.

benjaminwootton 3y ago

I would turn this on it’s head and say that most companies do need one or the other. My first premise is that most companies will have a BI or Data and Analytics problem, whether it’s analysing their spend, revenue, operations, customer churn or something more interesting. At that point, having an industry standard, fully managed, fully elastic and resilient platform with consumption based pricing sounds pretty appealing. Yes I can run and administer a warehouse on EC2, but the total cost of manpower and servers with full resilience is going to be high, especially as you’ll have to add in analytics tools, ETL tools etc which Snowflake or Databricks might have included. I’m a huge believer in both Snowflake and Databricks. Snowflake for BI and Databricks for anything more funky. The technology is on point and the business case stacks up for the most part.

lr4444lr 3y ago

...a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations This right here is exactly why Snowflake is a good fit for my org.: we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time on the Data Science's team's stuff, which frees them up to move faster on the core platform features bringing in the big bucks. This reads like a flavor of the same argument I've heard many times over. You know what the biggest cost of an org. really is as we head into a recession? Labor. Stop wasting it on stuff done better by specialized companies that free you up to get stuff done.

rglullis 3y ago
> we could pay someone's salary to install, maintain, and upgrade some open source alternative (and the VPS to run it), or we could just pay for Snowflake and stop wasting engineering's time You are missing the "we could pay someone that provides a commodified, open-source alternative to Snowflake" option. It would be cheaper, it wouldn't require any of your engineering team resources and it would help us all get rid of single vendor lock-in.
- bushbaba 3y ago
  Would it be cheaper? Cloudera fits the bill and it’s for sure not cheaper
  - rglullis 3y ago
    Because Cloudera is aimed at a market that is not the same that would benefit from "most companies" being talked about in TFA? Plenty of companies that would be fine by using a managed postgresql database for their "data warehouse" and you can get, e.g, one reasonably powerful server at Digital Ocean for $60/month with managed backups - and that already has quite a big markup compared to their droplets because they are just packaging some monitoring tools on top of their existing systems.
- hodgesrm 3y ago

qrios 3y ago

> The Fortune 100 have a use case for these companies, the rest are overpaying. There is a difference between "a use case for $data_platform" and "a data use case for $data_platform". Scope on the first one is the platform in $data_platform" and second is the data specifics requirements in $data_platform. Working on a non-Fortune100 insurance company in Europe, almost all our use cases can be easily done on traditional RDBMS like SQL Server, or on BI tools like SAS. Thankfully with higher granularity over time, Excel usage is fading out constantly. No big data, no heavy computing necessary - at least from my point of view. All setups in place today, can be called self-service platforms. With cautious estimations we have at least 100 of such "platforms" running since years, or even decades. This situation implies direct, we have a use case for a $data_platform itself. Costs are the biggest driver here, mainly due to the hidden costs of keeping these 100 systems up and running. Governance and management of the data, locked-in in all these stores, processed by slow and ugly SQLs nobody understands anymore, and with an unknown state of data quality, is key today.

catchnear4321 3y ago

> The cost for something that can be replicated free and open source is absurd. open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without. yes, yes, you just have to explain yourself. just help management understand. justification is part of the role. how long does that take, and how much effort? (the answer is subjective and contextual.) to be blunt, this kind of advice is barely more than “do better.” it ignores the situational example of using snowflake cheaply. it acts like most devs aren’t just going to go fire up a postgres rds - you said open source! it oversimplifies all problems, implicitly, to you. > Pay to make a problem disappear. no, pay to change the parameters of the problem. this is a fundamental misunderstanding of how to get things done in a constrained environment. it isn’t either/or, and every saas comes with problems. you pay to trade problems. otherwise it wouldn’t be worth a blog post name dropping and shitting on databricks and snowflake costs. in the hands of an untrained user or in a sufficiently constrained environment, they cost a lot - that’s one of the problems you buy. cost management. > a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations again, hope you’ve taken the time to build these bridges. the author does indicate this is most important. and it has a cost. someone is building that bridge, and likely someone from each side. you sometimes pay vendors to deal with technical problems so you can deal with non-technical ones. most importantly, don’t assume a random stranger on the internet has sufficient context to give you worthwhile recommendations.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
pydry 3y ago
>open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained? Funny. I heard this same kind of argument used against replacing oracle with postgres. It reminds me of Microsoft's "TCO" PR offensive back when they were public about how much they loathed open source competition. Thing is that it wasnt just a straw man (nobody that needs to be convinced is under any illusions that software has to be maintained), Oracle was also way more expensive to maintain ON top of being expensive to run. They had their hooks into that organization pretty good though and once that happens technology choices become highly political - which experts are you going to fire and which ones are you going to hire? The oracle lot obviously didnt want it to be them that get managed out.

mrweasel 3y ago

Most companies don't need a lot of things. I don't do a whole lot of data analytics anymore, but I'd say: Start with figuring out how much data you actually have. We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine. Once you hit the scale where you have "enough" data I'd agree with many of the other comments: Managing open source or home grown solutions quickly become more expensive than just paying for a service. Not quite the same, but we considered deploying a few solutions on OpenStack, but once you paid for training and staff it turned out to be cheaper to just give VMWare more money.

Aurornis 3y ago
> We still see companies claim to have vast amounts of data, but in reality they are talking about less than a TB of data, frequently just a few 100GB. When you operate at that scale, just chuck your data into whatever database you're comfortable with and do SQL queries, it's fine. I did some work with a company that talked about their “big data”, bragged about their data lake, and made grandiose statements about how they had big plans to use “all this data” to get an edge up on their competitors. They talked about the big data platforms and technologies they were evaluating and how they’d probably have to hire more people to handle it all. When I pressed them for details on it, they eventually explained the data they had on each customer and the approximate number of customers. I did some quick math and calculated the total amount of data on all of their customers was under 100MB. They had managed to blow it up to occupy a couple orders of magnitude more space in one situation (think storing time series data as JSON where every point had duplicated tags attached due to ineffective structure) but even that was small enough to fit entirely in RAM of a decent sized server.
- peteradio 3y ago
  Remember some industries are perversely incentivized to drive up costs because profits are limited as a percentage of total cost.

willvarfar 3y ago

My experience is that there is a lot of milage for small and medium enterprises to use your normal RDBMS replication to create a copy of your OLTP DB that drives your business and run your analytics on that. And put up with complaining it can be slow to do inefficient analytics queries. Really, after spending silly amounts of time and hassle (and money) on the fancy snowflake or whatever, you discover its not massively faster for those small and medium businesses. And now you have to keep paying for it and keep maintaining it, which is a often actually a bigger burden than keeping a normal RDBMS replica alive.

ozim 3y ago
I think "put up with complaining it can be slow to do inefficient analytics queries" is why I would rather pay for some 3rd party tool that would get the complaints. Business people want some queries that are slow no matter what you do. Then expect these to run in under a second. They don't care about some abstract things like operations complexity or big O - they want it here and now. Even if your data fits on one server there are still queries that will take time and it is not going to be instant, even if you put whole db in RAM it is going to be fast but still I can imagine bunch of queries that will simply have to take couple of minutes anyway.

jojobas 3y ago

Most people shouldn't need a 3GHz 8-core CPU in their pocket to look at cats or check weather, but here we are. Simplicity by bloat come at a price.

iou 3y ago

https://archive.is/xNT28 https://archive.is/xNT28

HorizonXP 3y ago

My client is in the middle of migrating their Snowflake data to Databricks Unity Catalog. To this day, I still do not understand why they love Databricks so much. It just looks like a Jupyter notebook to me. I know, it has Spark. So? I’m trying to learn more so I can build something better, maybe.

its_a_random_ac 3y ago
Yeah, I'd like to know too, from everything I can see it's just a cloud-hosted notebook with a spark backend, which does have some benefits, but I can't tell why they got as big as they did. Also from a more cynical side, I've interviewed at a lot of companies that ask if I've used it before, and I'm really temped to just lie, since it feels like something that could be picked up really easily.

benrutter 3y ago

As a data engineer who's worked in a bunch of different contexts/companies I 100% agree most of the time snowflake/databricks is an unnecessary money sink. The main problem is most companies need the security of a managed service for cloud computing, and don't want to be locked out of scaling to a very large scale (with distributed compute). Unfortunately, I don't think there are a lot of options that meet those pretty simple requirements that aren't databricks or snowflake[1]. Sure, you could put your compute on single docker containers, but when your data gets too big your stuck and have to get someone expensive in to manage kubernetes, all that time, you can't compute on your data. Which is sorta the crux of it: databricks and snowflake are expensive, but not nearly as expensive as finding out you need them and not having them. [1] if you're using python on was/gcp, I think coiled (coiled.io) is a rare exception to that pattern

eitally 3y ago
I think your second point is vastly overstated for 99% of companies and their use cases. There are precious few situations that can't be handled decently well by a simple RDBMS, and a simple ETL process into an analytical data store for reporting. Almost everything that exist fits one of three buckets: 1) CRUD app, 2) reporting tool, 3) business process app. And almost all of this is for internal use only. It's really only a handful of enterprise things that need massive scalability and performance, and consumer-oriented stuff (much of which is pretty niche and also doesn't need massive global scalability).
- benrutter 3y ago
  I think you're right and I probably came over a bit strong in how likely I made out companies would be to use that level of scale. The point I was trying to make was more that because using something like RDBMS (maybe + VMs or something for data science) has a hard ceiling in terms of scale, even if companies are really unlikely to hit that ceiling, there is a weird incentive structure. The costs of databricks or snowflake are really high, but probably not enough to bankrupt a company or division. If you're a small to midsize company and you realise you can't compute on your data, that could cause much more serious issues. Even if those issues are only feasible 1/100 times, it's a pretty scary risk to expect a CTO to ignore.

bjornsing 3y ago

Considering what you get I think Snowflake is rather decently priced. It’s not easy to integrate and operate a bunch of open source tools that replaces it, especially not on a small scale. Also, in most cases you’ll pay a lot more for the people that use Snowflake than you do for Snowflake, so it makes more sense to focus on productivity.

jboggan 3y ago

Modifying the argument, I'd say even the companies that do need Snowflake or Databricks for an analytical use case tend to get run over by the hype train post adoption and start using it everywhere as an anti-pattern and causing much bigger problems operationally.

ledauphin 3y ago
100%. as an organization, there's value in _limiting_ the number of tools you adopt. But you'll nearly always want to adopt a few right-sized tools rather than using the same sledgehammer for every single thing that looks remotely like a nail.

wnolens 3y ago

No one needs to evaluate all options against their current/known/possible needs when they can just pick a well vetted product and get on with business.

tootie 3y ago

> The global economy is headed for a recession. That’s not my opinion, that’s the Federal Reserve’s I know that's not the topic of the post but OP shows a lack of comprehension here. The Fed warned because that's what they do when the risk is non-negligible. They never say it is definitely going to happen because nobody knows.

deleted 3y ago

[deleted]

mxxc 3y ago

i am not sure why everyone (comments, this post, etc.) assume there is a one-size-fits-all solution to every problem, even this one that looks quite simple. companies have to align a few things, ranging from the skills of the current employees, hiring plans, investor/shareholder management, or how to make sure the CEO really gets that boat he really wants and deserves. not all businesses are the same. businesses with fat contracts but few users won't have massive operating costs, and they can use whichever easy and non-scalable technology they want, because once the business scales up, those fat contracts will pay for enough data engineers. a gaming startup will face high server costs right away, without any optimisation, while data platforms (e.g. bigquery) with a tiny bit of optimisation (materialising 2-3 summary tables, for example) will bring the cost down to "laughable" pretty easily. it is true that many of these things are choices, e.g. do you really want to spend a shit ton of money for looker when superset for most users is just as good? are you even able to make that choice? if these choices are hard to make because a potential user (or set of users) in the company really wants something instead of something else, well, that is not a technical choice, and the issue you have has nothing to do with the technology.

MrPowers 3y ago

I think most companies don't understand a large part of the Databricks offering and it should be used by way more organizations. Disclaimer: I was a Databricks user for 6 years and now work at Databricks. Yea, you can create your own Spark deployment, but it will run much slower than the Databricks Runtime (DBR) or the Databricks proprietary Spark Runtime (Photon). Computations that run slower cause you to have a larger cloud compute bill. Databricks rewrote Spark in C++ and it runs really fast and saves a lot on ec2 compute. > Define when you should compact files, when to Z-order Or don't consider these issues and use autocompaction / the new Liquid clustering. These are great examples of problems the platform should solve, so the user has time to focus on business logic. > If you can sniff out the inefficiencies in your Data early and make architecture that handles your specific data I don't know what this means. Are you going to build a deep learning model to make read/writes faster like Databricks predictive I/O? https://docs.databricks.com/en/optimizations/predictive-io.html https://docs.databricks.com/en/optimizations/predictive-io.h.... Probably not, you have a lot of business problems to solve. > Do the real work. Work with people. The Code will write itself. I've seen lots of DIY data platforms. They're horrible to work with and I can assure you that the code does not write itself. The data engineers have a lot less time to write code because they're constantly trying to stand the platform back up.

spmurrayzzz 3y ago
> Are you going to build a deep learning model to make read/writes faster like Databricks predictive I/O? It makes sense for Databricks to do this because they're building a product that needs to work for a high cardinality of datasets. Using a DNN for that is defensible because the input shapes are practically infinite. But for individual orgs, it seems much more likely that simple heuristics-driven access pattern optimizations can be done without throwing ML at the problem (though I'll say the predictive IO concept is a cool one, I've done similar ML work for network traffic QoS). > I've seen lots of DIY data platforms. They're horrible to work with and I can assure you that the code does not write itself. This one made me scratch my head a bit, because I have seen many DIY platforms as well over a 20 year career. Most of them have been amazing to work with. The code didn't write itself of course, but maintenance burdens were low and specialization/expertise within the org was high as a result. On top of that, I didn't need to argue with an AE about rising costs on a per-annum basis whenever renewal time comes around. I point this out not in the service of coloring your observations as wrong or misguided, but to highlight that there's going to be a spectrum of varied lived experiences people have with the build-vs-buy conundrum. I expect we'll never see genuine consensus on this issue as a result of that variance (and maybe we don't have to).

thiago_fm 3y ago

The thing is, you don't understand that a business needs to stay focused and that has a price. When you buy those solutions, you are paying for the price of staying focused. If the price for staying focused is $X, and X has a positive business impact and need, like Snowflake or Databricks could. And it doesn't mean you won't need to spend on HR, developers, management and so on to run your own OSS solution. It is actually great to give Snowflake and/or Databricks your money. It's a really expensive service, with a huge markup. That I understand. But the alternative doesn't look good at all. A company is better staying focused. Also, if the company sees that it expends a considerable amount of its budget in Snowflake/Databricks, they'll find solutions, that could be negotiating with them, figuring out how to optimize their use etc. I've worked and seen many companies optimizing their cost structure and run away from Datadog/Newrelic, Snowflake/Databricks and just fail miserably. The alternatives made developers simply use it less, and also spend more time to do the same that they did. They even sometimes call it successful, execs point out the money saved, but they don't see the subjective part: developers being less efficient, wasting their time with BS. There's a long tail of costs related to inefficient systems, it's wasting developer hours, that you need to manage, hire, train... Not to forget the burden that it is to fire or make people redundant, companies end up losing so much of their culture and soul doing such things. And this is what you need to typically do if you assume those expenses of running things for yourself. The ideal company are platforms that pays for many SaaS and stay focused on its core business, if it can make money, great! There is so much literature on cost optimization in business in general, stuff that have been studied for over a century. If people would read on that, they wouldn't be repeating such nonsense as this article. Also, the article says that you are OVERPAYING for it if you aren't a Fortune 100. It's much more like the opposite, if you have a small company with a small amount of data, it's very likely that Snowflake or Databricks won't cost you that much. Based also on the other comments on HN, I can only conclude that this is one of those articles that is so full of mistakes and plainly wrong, that there isn't even a real debate over here, just people saying how wrong it is.

datadrivenangel 3y ago

The dirty secret of modern data management is that very few people really have big enough data to justify major amounts of data infrastructure.

commandlinefan 3y ago

> charging their customers for Ferraris I've seen this sort of analogy before and I don't think it's the right one here: if I get a Ferrari, I know _exactly_ what I'm getting. I'm getting a really fast, really beautiful car. I'm not too familiar with Snowflake, but I've suffered under Databricks on and off for about a decade now, and as far as I can tell, it's just a more expensive, closed way to do what I could do a lot faster and a lot easier if I didn't have to work around the obstacles that Databricks puts in my way that don't have any value other than being something that Databricks can charge my employer a ton of money for putting in my way.

onlyrealcuzzo 3y ago
Theoretically, your employer is paying Databricks, because they'd rather pay them than have a harder time finding employees with actual skills (and paying them accordingly). Additionally, it's somewhat of a standard now - so it's easy to hire other people in the org with skills using it.
- orochimaaru 3y ago
  But isn’t that something they would need to do anyway - find employees with actual skills? I mean sure, it’s how the snake oil is sold - “buy so and so saas product and you wouldn’t need to hire and can reduce cost”. I’m surprised that line can still sell today because in software it’s rarely true unless you can merge excellent talent with said product.
  - onlyrealcuzzo 3y ago
    If you work at a small startup - they are mostly parroting what large and mid-size companies are doing - regardless of the fact that they're likely to fail before they get to that size, and doing said thing might make their chances of failure higher. If you work at a large company - having a larger pool of candidates to hire from is definitely worthwhile. And, it is much easier to have a poor understanding of Databricks than a very good understanding of data engineering in general. Someone with a poor understanding of Databricks might be able to do 80% of what you can do with a great understanding of data engineering in general.

bob1029 3y ago

Why not give in and pay for the SQL Server Hyperscale instance? Isn't there enough BS to worry about? Why continue to waste time on tired old OLTP/OLAP/scalability/etc. conversations in 2023? Unless your business can continuously write >100 megabytes of transactional data per second, this solution would almost certainly address all of your needs forever and ever. Up to 100TB too. It just works. It offers transactions exactly the way most business expect them to be conducted. No weird code, no weird client libraries, nothing. It works more or less like it has for the last 30+ years. I can tell you for a fact that simply setting up a gigantic shared database called "company" and getting the team connected to it did wonders for us. When you stop worrying about "will it scale" you can start to collaborate and do amazing things again.

bick_nyers 3y ago
If you do magically hit the "unicorn vector" and 100x your customer-base then you can start worrying about "will it scale", and thankfully you will have new revenue that can be deployed in the form of new hardware and new hires.

rmbyrro 3y ago

Most companies don't use Snowflake or Databricks

adam_oxla 3y ago

While I do agree that for most of companies those large SaaS solutions are overkill I do not think that DuckDB or similar is sufficient. Nowadays more of the companies really need to process large datasets. I meet regularly companies that used PostgreSQL or something similar up to some point but then they have grown and it is not sufficient anymore. They need something scalable. It does not have to be large SaaS: in many cases small Clickhouse cluster is sufficient. Nevertheless not everything can be done using single server. Also even if customer knows exactly well what are their needs right now does needs will grow and change over time so it is reasonable to build something that is not only good enough for now. Of course building something absolutely "future proof" leads to extremes and high bills.

slotrans 3y ago

Basically correct. If you have less than tens-of-terabytes the big exciting stuff just isn't needed and isn't particularly helpful. Focus on understanding your data and building a useful model. Use Postgres if you can. Supplement with DuckDB and/or ClickHouse as needed. Or, use your cloud's columnar DB (Redshift, BQ, Azure whatever) because you can start and stop using it as you please without talking to sales or signing a contract. If your data team doesn't have the requisite skills... well... consider that tech selection might not be your problem.

savrajsingh 3y ago

This reminds me of something I've seen too -- companies paying for enterprise wordpress (>$25k a year) when they could easily have say a $100 wordpress plan with all the features they need, behind free or cheap cloudflare.

pradeepchhetri 3y ago

I would consider following things before selecting a database offering: - Whether the database vendor is a lock-in. It would be a straight "NO" for me if the database isn't open-source with proper license, since I can't self-host it in case, I need to move away from their SaaS offering due to various unexpected foreseen reasons: vendor decided to increase pricing, vendor has hidden pricing, vendor has reliability issues etc etc. - How big is the community behind the database. Check their public forums to understand how community feels about the database and how their requirements are considered. - Don't believe any random benchmarking post online but by doing my own benchmarking for my use-case. - Check which other companies have adopted that database and read their experiences.

ouraf 3y ago

That's common practice with mid sized companies and above, though: - find something that can probably give some competitive advantage or please shareholders - Identify it as "not part of our core business" - Pay a third party company to do it even if it would be cheaper to things in house. If something goes wrong and the shareholders ask questions, the CEO can blame them instead of their own IT department and reassure them they're "focusing on what generates most value for the company".

tjhunter 3y ago

This article has valid points but does not understand the perspective of companies. Companies do not buy technologies. Companies buy solutions. - Companies do not buy Spark, they buy the ability to process their data and to have multiple personas collaborate (data scientists, data engineers, ...) - You can do it yourself. It will be cheaper but it will require time, expertise and money, all things that companies do not give easily - Snowflake and Databricks are elastic: you can start small and grow as you need. This is much easier than justifying the upfront cost of hiring specialized people or asking for trust that your ad-hoc solution will respect whatever enterprise governance rules (disclaimer: I worked at Databricks for 6 years and talked to hundreds of prospect and actual Databricks users and customers)