3 ms·

Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is

Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is the syntactic sugar. Once there is an API for Kotlin (beyond the preview version), I will be dropping Scala like a bad habit. Kotlin is also concise, has better IntelliJ support, less "implicit magic", and better Java interop. https://github.com/JetBrains/kotlin-spark-api https://github.com/JetBrains/kotlin-spark-api

bjarneh 6y ago
> has better IntelliJ support That's gonna be hard to beat for most languages I guess, the best IntelliJ support will most likely always go to Kotlin :-)
- brabel 6y ago
  That's not actually true, IntelliJ support for Java has always been much better, and still is today! Just try auto-completion in Java VS Kotlin in your IDE... it's much faster in Java files, something I've been complaining about as I've been using more Kotlin than Java, recently. Check also how many refactorings are available in your Java files compared to the Kotlin ones. Jetbrains knows its main source of income is still, by far, Java, so they don't drop that ball.
zmmmmm 6y ago
If only Kotlin would introduce a non-ridiculous way to declare map and list literals it would be a great data science language. As it is, it falls over at the first hurdle where you want to specify some data inline. One reason I've really stuck with Groovy for JVM-based data sciency stuff.
- dbsmith83 6y ago
  val list = listOf('a', 'b', 'c') val map = hashMapOf( "John" to "Doe", "Jane" to "Smith" ) Seems pretty non-ridiculous to me. Maybe you were using an old version? I guess you can switch now :)
lmm 6y ago
Kotlin has lots of syntax sugar but the semantics are crazy. It's essentially an imperative language where the only way to reason about effects (suspend functions) is by reasoning about control flow. That already makes things like error handling incomprehensible, and it'll be a really bad fit for Spark's way of doing things (where values have to be serializable and shipped around the cluster) - imagine trying to use a suspend function from a Spark anything, it's completely impossible to implement in a sane way.
- dbsmith83 6y ago
  I'm curious what your use case is for using a suspend function in spark. I have written several spark jobs with Kotlin and have never had a reason to create one. The only difference from the Scala/Java jobs is that I had to be much more explicit with certain things (encoders in higher order funcs, etc.).
  - lmm 6y ago
    All of the reasons you'd want to use a suspend function at all - e.g. mutually recursive functions, async I/O (e.g. database queries), error handling - suspend functions seem to be kotlin's answer to anything even slightly non-vanilla. Writing in Scala I can, with care, use async and iteratees in spark and be able to reason about what's going to get executed where and when, because these things are represented as values.

bjarneh 6y ago

> has better IntelliJ support That's gonna be hard to beat for most languages I guess, the best IntelliJ support will most likely always go to Kotlin :-)

brabel 6y ago
That's not actually true, IntelliJ support for Java has always been much better, and still is today! Just try auto-completion in Java VS Kotlin in your IDE... it's much faster in Java files, something I've been complaining about as I've been using more Kotlin than Java, recently. Check also how many refactorings are available in your Java files compared to the Kotlin ones. Jetbrains knows its main source of income is still, by far, Java, so they don't drop that ball.

zmmmmm 6y ago

If only Kotlin would introduce a non-ridiculous way to declare map and list literals it would be a great data science language. As it is, it falls over at the first hurdle where you want to specify some data inline. One reason I've really stuck with Groovy for JVM-based data sciency stuff.

dbsmith83 6y ago
val list = listOf('a', 'b', 'c') val map = hashMapOf( "John" to "Doe", "Jane" to "Smith" ) Seems pretty non-ridiculous to me. Maybe you were using an old version? I guess you can switch now :)

lmm 6y ago

Kotlin has lots of syntax sugar but the semantics are crazy. It's essentially an imperative language where the only way to reason about effects (suspend functions) is by reasoning about control flow. That already makes things like error handling incomprehensible, and it'll be a really bad fit for Spark's way of doing things (where values have to be serializable and shipped around the cluster) - imagine trying to use a suspend function from a Spark anything, it's completely impossible to implement in a sane way.

dbsmith83 6y ago
I'm curious what your use case is for using a suspend function in spark. I have written several spark jobs with Kotlin and have never had a reason to create one. The only difference from the Scala/Java jobs is that I had to be much more explicit with certain things (encoders in higher order funcs, etc.).
- lmm 6y ago
  All of the reasons you'd want to use a suspend function at all - e.g. mutually recursive functions, async I/O (e.g. database queries), error handling - suspend functions seem to be kotlin's answer to anything even slightly non-vanilla. Writing in Scala I can, with care, use async and iteratees in spark and be able to reason about what's going to get executed where and when, because these things are represented as values.