5 ms·
One thing I find somewhat amusing about this is that all of the generated code is against the PySpark API. And the PySpark API is itself an interop layer to the
by pixelmonkey 3y ago
One thing I find somewhat amusing about this is that all of the generated code is against the PySpark API. And the PySpark API is itself an interop layer to the native Scala APIs for Spark.
So you have LLM-based English prompts as an interop layer to Python + PySpark, which is itself an interop layer onto the Spark core. Also, the generated Spark SQL strings inside the DataFrame API have their own little compiler into Spark operations.
When Databricks wrote PySpark, it was because many programmers knew Python but weren't willing to learn Scala just to use Spark. Now, they are offering a way for programmers to not bother learning the PySpark APIs, and leverage the interop layers all the way down, starting from English prompts.
This makes perfect sense when you zoom out and think about what their goal is -- to get your data workflows running on their cluster runtime. But it does make a programmer like me -- who developed a lot of systems while Spark was growing up -- wonder just how many layers future programmers will be forced to debug through when things go wrong. Debugging PySpark code is hard enough, even when you know Python, the PySpark APIs, and the underlying Spark core architecture well. But if all the PySpark code I had ever written had started from English prompts, it might make debugging those inevitable job crashes even more bewildering.
I haven't, in this description, mentioned the "usual" programming layers we have to contend with, like Python's interpreter, the JVM, underlying operating system, cloud APIs, and so on.
If I were to take a guess, programmers of the future are going to need more help debugging across programming language abstractions, system abstraction layers, and various code-data boundaries than they currently make do with.
- nivekkevin 3y agoThe annoying (?) part of Scala Spark is the lack of notebook ecosystem. Also spark-submit requires a compiled jar for Scala yet only the main python script for Python. I would've loved Scala Spark if the eco system was in place.
- tbcj 3y agoWhat about Zeppelin?
- chaxor 3y agoAs has been pointed out many times, this is similar to the steps that have led to interpreted languages like python, R, Julia, etc that make calls to C, or use JVM/LLVM, etc on up to assembly or machine code. The leap made is certainly less defined than previous jumps, but there is some similarity in that the more specific a person writes the rules to define the program, the more potential there can be in making something powerful and efficient (if you know what you're doing). The next big gain in capability (other than the onvious short term goal of making an LLM output a full working code base) may be in LLMs being able to choose better design, without it being specified (for example having 'search for the best algorithm', and 'make it idempotent', etc added automatically to each prompt), and to potentially write the program automatically in something like assembly (or Rust or C for better readability) directly instead of preferring python as these models tend to right now.
- pixelmonkey 3y agoI don't think it's exactly the same because an LLM-based English=>Python translator is nowhere near as deterministic as compilers and assemblers. And English, being a language whose tokens are subject to wide interpretation of meaning, may be a source of byzantine complexity. Then, of course, there is the "moving target" introduced by model upgrades and evolution in the public crawl dataset rewiring the neural network for the model's world knowledge. There is a reason Python, as high level as it is, is still defined using an eBNF / PEG grammar[1] with only 35 or so keywords[2]. And there is a reason the Python bytecode interpreter is "just" a while loop on a minimal set of instructions[3]. All of this leads to a remarkable level of determinism, and determinism is your friend when trying to get code right. I haven't yet seen the equivalent in LLMs. I don't think it's an entirely intractable problem, but I'd be hesitant to leap straight into English language as a stable API today. I think code copilots are the right place to start. And maybe even copilots that help not just with code suggestions, but also with debug suggestions. [1]: https://docs.python.org/3/reference/grammar.html https://docs.python.org/3/reference/grammar.html [2]: https://docs.python.org/3/reference/lexical_analysis.html#keywords https://docs.python.org/3/reference/lexical_analysis.html#ke... [3]: https://devguide.python.org/internals/interpreter/ https://devguide.python.org/internals/interpreter/
- majormajor 3y ago
- KptMarchewa 3y ago> So you have LLM-based English prompts as an interop layer to Python + PySpark, which is itself an interop layer onto the Spark core. And the spark core "just" generates execution plan, which on databricks gets executed as native code. https://docs.databricks.com/runtime/photon.html https://docs.databricks.com/runtime/photon.html
- ryangibb 3y agoVernor Vinge wrote about 'programmer archaeologists' digging through 1000-year-old software stacks in https://en.wikipedia.org/wiki/A_Deepness_in_the_Sky#Interstellar_culture https://en.wikipedia.org/wiki/A_Deepness_in_the_Sky#Interste...
- whinvik 3y agoNitpick, but if you are using Photon, then the API is actually C++.