4 ms·

Meh, the complaints fall basically in two categories: 1. Open source libraries are sometimes not maintained which is absolutely true but also just a general ha

Meh, the complaints fall basically in two categories: 1. Open source libraries are sometimes not maintained which is absolutely true but also just a general hazard of the open source software ecosystem. The solution in most cases is to just not use open source libraries that are somebody's hobby project (which the author very sensibly recommends). But that is true whether you are using Scala or not. 2. Spark is weird. Which is true but that seems like it is more to do with the idiosyncratic nature of Spark than it is anything to do with Scala. That is, Spark development is sort of its own category and poses certain problems that are pretty specific to Spark and you don't necessarily run into when you are building non-Spark project in Scala.

MrPowers 6y ago
Some responses: 1. Yea, but languages that are backwards compatible have libs that are useful for way longer. I can still use Java JAR files that were built with Java 8, and published many years ago. The open source libs get stale so much quicker in Scala. 2. I'd actually argue that Spark is less weird that some other dependencies. If you look at the cats README (https://github.com/typelevel/cats https://github.com/typelevel/cats) there is this caveat: Cats relies on improved type inference via the fix for SI-2712, which is not enabled by default. For Scala 2.11.9+ or 2.12 you should add the following to your build.sbt: scalacOptions += "-Ypartial-unification" More cats specific stuff here: https://github.com/sbt/sbt/releases/tag/v1.5.0-RC2 https://github.com/sbt/sbt/releases/tag/v1.5.0-RC2 I use Spark a lot so used that for the examples, but think other libs cause even weirder maintenance challenges.
- thinkharderdev 6y ago
  Fair enough, the whole SI-2712 debacle is indeed a source of issues well beyond Spark but in general it's just a matter of adding a compiler flag to sort it out. The other example is of course the whole mess of macros... On the other side though, strict backwards compat creates it's own set of costs (that are often harder to quantify. Java backwards compatibility is great but it seems like it is also at least prat of the reason Java moves at such a glacial pace.
- nafg 6y ago
  You don't have to add it, it just helps reduce boilerplate when using cats

MrPowers 6y ago

Some responses: 1. Yea, but languages that are backwards compatible have libs that are useful for way longer. I can still use Java JAR files that were built with Java 8, and published many years ago. The open source libs get stale so much quicker in Scala. 2. I'd actually argue that Spark is less weird that some other dependencies. If you look at the cats README (https://github.com/typelevel/cats https://github.com/typelevel/cats) there is this caveat: Cats relies on improved type inference via the fix for SI-2712, which is not enabled by default. For Scala 2.11.9+ or 2.12 you should add the following to your build.sbt: scalacOptions += "-Ypartial-unification" More cats specific stuff here: https://github.com/sbt/sbt/releases/tag/v1.5.0-RC2 https://github.com/sbt/sbt/releases/tag/v1.5.0-RC2 I use Spark a lot so used that for the examples, but think other libs cause even weirder maintenance challenges.

thinkharderdev 6y ago
Fair enough, the whole SI-2712 debacle is indeed a source of issues well beyond Spark but in general it's just a matter of adding a compiler flag to sort it out. The other example is of course the whole mess of macros... On the other side though, strict backwards compat creates it's own set of costs (that are often harder to quantify. Java backwards compatibility is great but it seems like it is also at least prat of the reason Java moves at such a glacial pace.
nafg 6y ago
You don't have to add it, it just helps reduce boilerplate when using cats