5 ms·
"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, whi
by spenrose 7y ago
"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, which is the core part of the compiler, very easy. Our original optimizer was based on the design of Catalyst, which is Spark SQL’s extensible optimizer. We moved away from Scala because it was too difficult to embed a JVM-based language into other runtimes and languages."
There is an important "contemporary history of computing" article to write about the evolution of the Spark project from "let's build a distributed filesystem for MapReduce in Java because we read those early Google papers" to "SQL is the right model for working with data so DataFrames" to "meet data scientists where they are: Python (and R)" to "make machine learning easy" and now to "LLVM, but for crunching big numeric arrays".
- deleted 7y ago[deleted]
- dpflan 7y agoVery interesting. Do you have any references to share relevant to the article you suggest to be written?
- vmchale 7y agoNot that you should replace Rust with Haskell, but Haskell would've been a better choice than Scala. It has its own runtime, but it's not difficult to call Haskell code from C or ATS or whatever.
- paulddraper 7y ago> it was too difficult to embed a JVM-based language into other runtimes and languages In addition to the JVM, Scala has had JS [1] and native (via LLVM) [2] targets for years. (And that's not even mentioning any second-order compilations; e.g. Scala -> JVM bytecode -> native) There's a number of reasons to not choose Scala, but portability is far from one of them. [1] https://www.scala-js.org https://www.scala-js.org [2] http://www.scala-native.org http://www.scala-native.org
- pacala 7y agoThe library ecosystem is different between JVM/JS/Native. Porting across runtimes may require more work than just changing the compilation target.
- paulddraper 7y agoThat's true. If you want it across all platforms, you are restricted to the Scala ecosystem, and can't use Java and JS ecosystems.