3 ms·
Is Scala used outside of Spark and Akka? Spark is my company chosen analytic engine, Scala was/is the main language for majority of the production workload. In
by aynyc 6y ago
Is Scala used outside of Spark and Akka? Spark is my company chosen analytic engine, Scala was/is the main language for majority of the production workload. In the last two years or so, more devs are writing PySpark and SparkSQL jobs than Scala.
- dominotw 6y agodef mirrors my experience too. Vast majority of spark jobs were easily ported to sql/dbt and the remaining ones are in pyspark. I used to use a lot of scala spark in backend data processing in 2016 but now its almost down to zero. scala is real big impediment to making data processing accessible to general public in your company. order of preference now at my company is, 1. sql 2. pyspark 3. java spark 4. scala spark eg: shopify found that 70% of their pyspark could be converted to just sql https://shopify.engineering/build-production-grade-workflow-sql-modelling https://shopify.engineering/build-production-grade-workflow-...
- aynyc 6y ago> scala is real big impediment to making data processing accessible to general public in your company Ding Ding Ding! Presto/Athena now is becoming huge in BI ecosystem. We don't really use Spark for ad-hoc BI anymore, we use it for data science and large repetitive workload.
- MrPowers 6y agoI've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features. Here's a more detailed PySpark vs Scala comparison in case folks are interested: https://mungingdata.com/apache-spark/python-pyspark-scala-which-better/ https://mungingdata.com/apache-spark/python-pyspark-scala-wh... I think Scala Spark (using 10% of the language features) is a better technical decision (because it provides huge benefits like fat JARs, shading, better text editor support, etc), but the worse overall choice for most organizations because people are generally terrified of Scala. They'd rather do nothing than write Scala code. I can empathize with their position.
- valenterry 6y agoEven when Scala is used more or less like python?
- deleted 6y ago[deleted]
- UglyGenius 6y agoThere's a blooming ecosystem of pure functional programming libraries. Or even two ecosystems: Typelevel and ZIO. My company uses Scala a lot with with these ecosystems (my team is on Typelevel, another one is on ZIO) and in fact Spark and Akka is what we're trying to avoid as much as possible. Once I got used to the composable nature of FP and static types - both Spark and Akka just feel clumsy and unintuitive, like maintainers of Spark chose Scala because it was a "Java with fancy syntax" back then, ignored most of benefits of static types, HKT, immutability, type classes etc. If there's future behind Scala - it definitely doesn't lie in Apache land. Akka is slightly different and probably has higher potential, but just an overkill for most of use cases it's advertised for.
- sdfhbdf 6y agoIt's used in my workplace for ElasticSearch interactions. It's for search, product recommendations, discovery, sorting in an ecommerce.
- jatcwang 6y agoI've spent the last 5 years working professionally in Scala and I've never touched spark and I know plenty others like me. (I'm mainly a backend developer) I think it's quite location-dependent. From my own surroundings, you can easily find backend Scala jobs in the bigger hubs in Europe, and seems like there are plenty in the US as well.
- the_only_law 6y agoI want to like Scala and Akka, but they just seem to get used very little in the “real world”. I recall some HN comments complaining about how a lot of the Scala they were writing for Spark was more or less a series of procedural API calls rather than using the language to its potential.
- valenterry 6y agoYes, totally. We are using Scala for web development, without using spark. Half of our services run on akka, the other half uses typelevel ecosystem.