6 ms·

Scala is used pretty heavily in the big data world, particularly if you are working with Spark.

by kprybol 8y ago

joshlemer 8y ago
Not to mention, Flink, Akka, Apache Beam (via the Scio library or the Java api).
- kprybol 8y ago
  Kafka too
  - dominotw 8y ago
    Kafka switched to java 100%, afaik.
bitL 8y ago
Python is still king there, even DataBricks market it that way. Scala is for advanced stuff that matters.
- RBerenguel 8y ago
  Python is king in analysis, but for big data engineering, most of the building blocks (as mentioned elsewhere in this thread, Akka, Kafka, Flink, Spark, etc) are written in Scala
  - sandGorgon 8y ago
    It doesn't matter. Because pyspark is still the go-to language. In fact with spark 2.3 python UDF, the performance gap has also reduced. https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-dataframes https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-data...
    - RBerenguel 8y ago
      Being able to pass data through Arrow is a big improvement, but there's also a lot of serialisation going on you pay in Python. Also, if you want to do anything in the fancy areas (like, write your own optimisation rule for the SparkSQL optimiser) it's Scala. Even something simple as writing a custom aggregator is impossible in Python (at least it was in 2.2, haven't checked in 2.3 or "current" 2.4)
    - kprybol 8y ago
      It depends on the use case. Our work primarily revolves around extending Spark with custom pipelines, models, ensembles, etc. to be deployed into our production systems (petabyte scale). Scala was really the only way to go for us.
      - sandGorgon 8y ago
        I can understand performance difference, but I have not generally seen a difference in building custom pipelines and ensembles .. although I grant I'm not at your scale yet. What kind of specific pipelines did you have trouble in pyspark ?
        RBerenguel 8y ago
        Although we decided to start using Scala specifically because PySpark was not as performant (2.0 is not so far ago), a reasonable use case I keep always in mind is aggregation (and in general any API which is still not solid/experimental/under work). Python bindings are always the last to be available (because all groundwork is being done in Scala). We have a relatively large scale process that takes advantage of custom-built aggregation methods on top of groupedDatasets, where we can pack a good deal of logic in the merge and reduce steps of aggregation. We could replicate this in Python using reducers, but aggregating makes more sense semantically, which makes the code is easier to understand. Also, the testing facilities for Spark code under Scala are a bit more advanced than under Python (they are not super-great, but are better), even without considering that being strongly typed makes a whole kind of errors impossible, right out of the compiler. I very, very rarely think of using PySpark (and I have way more experience with Python than with Scala) when working with Spark. In a kitchen setting, it would be like having to prepare a cake and having to choose between a fork and a whisker. I can get it done with the fork, but I'll do a better and faster job with the whisker.
- wenc 8y ago
  Scala is still primarily used for data engineering workloads due to the fact it is a JVM language. (There's Java too, but no one wants to write Java code) PySpark is often used for data science experimentation, but is not as frequently found in production pipelines due to the serialization/deserialization overhead between Python and the JVM. In recent years this problem is less pronounced due to the introduction of Spark dataframes which obviates the performance differences between PySpark and Scala Spark, but for UDFs, Scala Spark is still faster. A newer development that may change all this is the introduction (in Spark 2.3) of Apache Arrow, a in-memory column store engine which lets Python UDFs work with the in-memory object without serializing/deserializing. This is very exciting as this lets Python get closer to the performance of JVM languages. I've played around with it on Spark 2.3 -- the Arrow interface works but still not quite production-ready but I expect it will only get better. Many folks are making strategic bets on Arrow technology due to the AI/GPU craze (and an in-memory standard enables multiple parties to build GPU-based analytics [1]), so there is tremendous momentum there. At some point I expect the relative importance of Scala on Spark will decrease with respect to Python. (even though Spark APIs are Scala native) [1] https://www.nextplatform.com/2017/05/09/goai-keeping-databases-analytics-machine-learning-gpu/ https://www.nextplatform.com/2017/05/09/goai-keeping-databas...

Scala is used pretty heavily in the big data world, particularly if you are working with Spark.

joshlemer 8y ago

Not to mention, Flink, Akka, Apache Beam (via the Scio library or the Java api).

kprybol 8y ago
Kafka too
- dominotw 8y ago
  Kafka switched to java 100%, afaik.

bitL 8y ago

Python is still king there, even DataBricks market it that way. Scala is for advanced stuff that matters.

RBerenguel 8y ago
Python is king in analysis, but for big data engineering, most of the building blocks (as mentioned elsewhere in this thread, Akka, Kafka, Flink, Spark, etc) are written in Scala
- sandGorgon 8y ago
  It doesn't matter. Because pyspark is still the go-to language. In fact with spark 2.3 python UDF, the performance gap has also reduced. https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-dataframes https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-data...
  - RBerenguel 8y ago
    Being able to pass data through Arrow is a big improvement, but there's also a lot of serialisation going on you pay in Python. Also, if you want to do anything in the fancy areas (like, write your own optimisation rule for the SparkSQL optimiser) it's Scala. Even something simple as writing a custom aggregator is impossible in Python (at least it was in 2.2, haven't checked in 2.3 or "current" 2.4)
  - kprybol 8y ago
    It depends on the use case. Our work primarily revolves around extending Spark with custom pipelines, models, ensembles, etc. to be deployed into our production systems (petabyte scale). Scala was really the only way to go for us.
    - sandGorgon 8y ago
      I can understand performance difference, but I have not generally seen a difference in building custom pipelines and ensembles .. although I grant I'm not at your scale yet. What kind of specific pipelines did you have trouble in pyspark ?
      - RBerenguel 8y ago
        Although we decided to start using Scala specifically because PySpark was not as performant (2.0 is not so far ago), a reasonable use case I keep always in mind is aggregation (and in general any API which is still not solid/experimental/under work). Python bindings are always the last to be available (because all groundwork is being done in Scala). We have a relatively large scale process that takes advantage of custom-built aggregation methods on top of groupedDatasets, where we can pack a good deal of logic in the merge and reduce steps of aggregation. We could replicate this in Python using reducers, but aggregating makes more sense semantically, which makes the code is easier to understand. Also, the testing facilities for Spark code under Scala are a bit more advanced than under Python (they are not super-great, but are better), even without considering that being strongly typed makes a whole kind of errors impossible, right out of the compiler. I very, very rarely think of using PySpark (and I have way more experience with Python than with Scala) when working with Spark. In a kitchen setting, it would be like having to prepare a cake and having to choose between a fork and a whisker. I can get it done with the fork, but I'll do a better and faster job with the whisker.
wenc 8y ago
Scala is still primarily used for data engineering workloads due to the fact it is a JVM language. (There's Java too, but no one wants to write Java code) PySpark is often used for data science experimentation, but is not as frequently found in production pipelines due to the serialization/deserialization overhead between Python and the JVM. In recent years this problem is less pronounced due to the introduction of Spark dataframes which obviates the performance differences between PySpark and Scala Spark, but for UDFs, Scala Spark is still faster. A newer development that may change all this is the introduction (in Spark 2.3) of Apache Arrow, a in-memory column store engine which lets Python UDFs work with the in-memory object without serializing/deserializing. This is very exciting as this lets Python get closer to the performance of JVM languages. I've played around with it on Spark 2.3 -- the Arrow interface works but still not quite production-ready but I expect it will only get better. Many folks are making strategic bets on Arrow technology due to the AI/GPU craze (and an in-memory standard enables multiple parties to build GPU-based analytics [1]), so there is tremendous momentum there. At some point I expect the relative importance of Scala on Spark will decrease with respect to Python. (even though Spark APIs are Scala native) [1] https://www.nextplatform.com/2017/05/09/goai-keeping-databases-analytics-machine-learning-gpu/ https://www.nextplatform.com/2017/05/09/goai-keeping-databas...