Things like Spark may be the main place where a lot of today's programmers really have to fight the "compiler" already compared to, say, writing Java for the JV

Things like Spark may be the main place where a lot of today's programmers really have to fight the "compiler" already compared to, say, writing Java for the JVM or writing plain Python, since these are being compiled down to parallel execution plans across distributed systems, which introduces fairly novel performance pitfalls compared to writing a linear Java or Python method that gets compiled down to basic native code. Adding a layer on top of it is certainly going to add more fun for the engineers who get the "hey, I need this to be fast" requests from analysts who have written their own query/notebook/English prompt to build a dashboard. It's not a bad thing, it's a very useful capability since if you're a generalist or a novice data analyst having to learn Spark or SQL to know how to do things like "get 4 week moving average sales by dept" is a big hurdle, but I think this is one of the more obvious examples of where these tools are going to result in more engineering demand in a lot of orgs, instead of less, even if things only go sideways or get crazy slow 0.1% of the time. At the meta-level Databricks or someone should be able to build some pretty good "optimizing compilers" that feed in all the info about your execution env, data, etc, to the code generator. But any time you need to override that you're gonna suddenly need a LOT of domain knowledge.