I've worked in this space in the past, and I think the big weakness of databricks is that for them to be successful as an independent company, they need to char

I've worked in this space in the past, and I think the big weakness of databricks is that for them to be successful as an independent company, they need to charge a lot more than AWS/GCloud/ect, and while historically they had better performance for a lot of Spark jobs, this hasn't been the case for several years now. What Databricks offers that you can't find in essentially the exact same products from other cloud providers, is that they have a pretty UI and a much more integrated setup for more advanced operations like upserts on a data lake instead of having to manage your own Hudi or Iceberg setup. Their notebook is a better offering than others have, but the money for big data companies is in the long running daily or weekly jobs, not one off queries. Historically, databricks had been offering major performance improvements to their customers while not updating the Spark open source code with all of the same query optimizations, and other various major spark performance improvements like Dynamic Pruning and Adaptive Execution, they've since done an about face for the most part and have started pushing these improvements into the open source. But as I noted before, for these longer running daily jobs, you'd spend 10x less per instance hour in some cases running on a cloud providers generic offering rather than paying for databricks, as they've made performance improvements of their own such that the price performance almost universally will always beat databricks. The point is that databricks is absolutely a very innovative company, it's just that it's impossible to compete against the bigger players. The only real major lasting strength of databricks is that Azure fully integrated them into their service as opposed to trying to compete directly like AWS and Gcloud have.