9 ms·

I don't have a ton to say here, the article says it all really. But this also reflects my experience when developing with Scala and although I love the language

I don't have a ton to say here, the article says it all really. But this also reflects my experience when developing with Scala and although I love the language and its brevity, I went back to Java because I hated the Scala toolchain, sbt, dependency issues, etc. Basically, I like the language in isolation but the overall experience was horrible.

smrtinsert 4y ago
Honestly Java is lately the best it's ever been, at least for backend services. So much power in the frameworks (ok Spring Boot) with so many powerful libraries and tools and available apis. You really feel like you can do anything with it lately.
badpun 4y ago
Java has the same dependency issues? I vividly remember spending days on jars shading in Maven. Basically, a single approach that would solve this, irrespective of the language, would be isolating dependency trees of every library from each other, essentially duplicating the code of many common libraries in RAM (like Docker, but for libraries inside the same process). There's an option to do that in Maven, but it's not the default. And it still would not work for dependencies where there should be only a single instance of it in RAM, like for example loggers...
- mbfg 4y ago
  One thing is Scala plays fast and loose with backwards compatibility. So much so that akka actually prints out messages if they find jars with different versions. It's a pain in the arse.
- zokier 4y ago
  > Basically, a single approach that would solve this, irrespective of the language, would be isolating dependency trees of every library from each other Doesn't solve the problem completely because it is normal that applications need libraries to interoperate; consider case where application is passing objects between two libraries that both use types from a third library.
  - badpun 4y ago
    Then your app would explicitly be doing the impendance mismatch resolution (converting objects from one version of library to the other) when doing the library calls.
    - CrimsonRain 4y ago
      which someone would have to code in as that's not automatically possible all the time.
  - jayd16 4y ago
    You're pretty much screwed in that case but that's kind of by design and the entire point of versioning, isn't it?
  - layer8 4y ago
    A major problem is that Maven (and friends) doesn’t distinguish between API dependencies (the API of a library using types from the API of another library) and implementation dependencies (a library depending on another library as an implementation detail). If that distinction were made consistently, implementation dependencies could indeed be separated into a dedicated classloader for each library, and you’d only have to worry about API dependencies, which would go a long way. The Java 9 Modules architecture would have been an opportunity to introduce something like that, but alas they didn’t.
    - brabel 4y ago
      OSGi allowed you to specify in excruciating detail what dependencies you had, whether API (i.e you wanted to re-export types from another library) or not, and much more. Unfortunately, it became far, far more complex than just implementing a simple module system, so as most things from the era (early 2000's) it became a huge, complex beast no one wanted to touch anymore.
    - clhodapp 4y ago
      I believe that separating interface from implementation doesn't really solve the problem, it just shrinks it. E.g. what if you depend on two libraries that re-export classes from different versions of a third library? I think that if you truly want to solve the problem, you need to extend the definition of the fully qualified classname to include the notion of a breaking version reference. Kind of like what folks now sometimes do the hard way by encoding it into the package name. This would make it possible to directly address the two different versions from the same calling source file.
- valenterry 4y ago
  Jep, this is really an issue of how the JVM and Classloaders are architected. Rust essentially chooses the other side of the sea, which is better in some sense, but worse in others (which we will see once the ecosystem grows). A configurable middleground is needed, but it's not an easy task to get that done.
- teknopaul 4y ago
  Essentially OSGI is that. It is still a ballache tho, and slow to boot, so I would not recommend it. Best bet it to be picky about dependencies that respect backward compatiblity or do name changes (like commons collections4)
  - deleted 4y ago
    [deleted]
- mumblemumble 4y ago
  I have personally spent so much time fussing with runtime ClassDefNotFound and MethodNotFound errors in Java that I am no longer able to respond to assertions that Java is type-safe with anything but an incoherent stream of cursing and rage tears. But Scala somehow seems to take all of that and intensify it. I have heard that Scala 3 is supposed to improve this, but I don't anticipate being able to adopt Scala 3 before the heat death of the universe.
  - jayd16 4y ago
    Not sure what you're doing with Java but that's not a common problem at all. Are you just aggressively stripping code and dealing with the fallout?
    - mumblemumble 4y ago
      Working on projects that involve big frameworks with deep and ever-shifting dependency trees. So upgrading an existing dependency or taking a new one could often be a risky move. Ground zero for most of them was Guava, which I unfortunately could not avoid due to it being a transitive dependency. And, even more unfortunately, Guava types were being exposed on the direct dependency's public API, so package relocation was not an easy option.
      - erik_seaberg 4y ago
        Guava is an outlier. While they do honor semver, they’re already on version 31.1, meaning they have felt they needed to break backward compatibility dozens of times, and it’s likely your dependencies require conflicting versions just by accident of when they were written. I guess it’s because the maintainers live in a monorepo and just force any changes they need?
        mumblemumble 4y ago
        The other interesting corollary to this is: if your package relies on one like Guava, and it is not bumping its major version number every time it updates the version of Guava it depends on, then your package does not itself honor semver.
- kitd 4y ago
  JBang [1] can help a lot here, especially with the new "BOM POM" feature. The experience feels closer to Go than Maven. [1] - https://www.jbang.dev/documentation/guide/latest/dependencies.html https://www.jbang.dev/documentation/guide/latest/dependencie...
- stickfigure 4y ago
  I only very rarely see this in Java, even in huge monoliths. Not that there's a technical reason for this; I think there's a cultural expectation that you don't break backwards compatibility with Java libraries. Perhaps the Scala world is more cutting-edge and fluid so there are different cultural expectations. The OP mentions Jackson. Jackson 1.x -> 2.x changed the package (so they are wholly separate libraries), Jackson 2.x -> 3.x will do it again. I've never had backwards compatibility issues with Jackson. I don't know what someone could be doing in jackson-module-scala that breaks with a minor version bump.
  - dkarl 4y ago
    Jackson doesn't have a stable API for building modules. You have to get elbow deep in their internals, which is already scary enough given that it's an old and highly performant library, but the bad part is that there's no formal commitment to maintaining compatibility. You just have to hope that the modules you use are popular enough that they will test them and make an effort not to break them. I've had an application break on a patch version bump of Jackson.
- zibby8 4y ago
  Java has the same dependency issues. But, from the article, it seems like there might be something cultural within the Scala community that contributes to the problem. Most Java libraries, in my experience, are relatively flat. The article gives an example of HikariCP, which depends on a random JDBC library from Akka. Obviously, that situation is going to end terribly given the popularity of Akka. As soon as Akka updates their library version, HikariCP has to as well or the two dependencies will get hopelessly out of sync. I don't know what `akka-persistence-jdbc ` does, but it can't be so important that it justifies the crazy dependency structure.
  - mbfg 4y ago
    Scala doesn't care about backwards compatibility. They break stuff all the time, And so major libraries are tenuously hanging on to whatever version they were targeted at. Akka prints out messages if it finds jars with even different minor versions.
  - therealdrag0 4y ago
    HikariCP is a Java library I thought, at least I’ve used it in many Java projects, so I’m not sure what that has to do with Scala.
- AtlasBarfed 4y ago
  Dependency conflicts is a fundamental issue of mature ecosystems, linux especially included. The issue is one of granularity of dependency: it can't be calculated from the code and dependency graph easily. Each class or function invocation has an associated metadata version, and something that tracks all the sub-ones, with compatibility matrices across the major versions of the library? As in, a dependency / linked library has a new version. How do you know if that can be safely updated in your code? All you have fundamentally is a coarse version number for the entire library, it's types/classes, it's functions/procedures/methods/signatures. Does anything indicate which signatures changed aside from compilers? Does anything mark which functions have new logic in them? Yikes. That can't be maintained by humans. No language has dared to do something that would resolve to that level of compatibility analysis. Can you imagine the time involved? Maybe there's simpler indicators. Git repo as a service might be able to use the repo history to track what parts of a library actually changed with some metadata file at a sufficiently granular level. Currently, convention dictates that you use a new class name or namespace or something similar for "breaking" changes, or a major version number boost with some assumed "service life" for the previous library major version. You know, if they do that. But there's instances (one of the java bytcode emission libraries IIRC) where the api remainined the same but was a breaking change. Nothing exists to help a library maintainer be aware of what will be a breaking change. At granular levels that is inherent to the language design. At best, you have unit tests as the actually deep introspection of the interface and expected responses. But that is per-project. As mentioned, you can also shade and dynamically renamespace a library. That almost is the solution, but shared libraries are a thing, and a if you have 10 libraries each shaded-using a common util library (ahem, slf4j, first item in the blog post), then that's precious ram wasted, even in modern RAM sizes. With three-deep versions like 4.0.7 or something similar, how would a build system know that it could safely reuse a shaded version? It basically can't. There's no language constructs for saying that it can. To some degree I believe this a failure, in the case of Java, of some sort of overarching organization that tracks commons libraries, and while maybe not placing them in the language standard library, recommending best practices around those libraries across known usage patterns so library import/compiling can be gradually optimized, or common conflicts alleviated. It's almost like security patching to some degree. Maven and its repos aren't up to the task. You have a new language? Likely nothing in it will killer app it, instead what makes modern new languages succeed isn't just novel language features, it's community and herding cats. It's what Zig just learned from one tweet, what Lisp's community never really learned for what otherwise should be the "ultimate" language. It's how Rust has made progress, and maybe Go too. I wonder if in 200 years if there will actually be mature, stable languages with mature, stable libraries that haven't needed to change for a couple decades. Anyway, "compatibility" is a hard hard hard problem.
- oftenwrong 4y ago
  Dependency management is painful almost everywhere. I see a glimmer of hope for the future in Unison, a language in which code exists as content-addressed ASTs. Dependency version conflicts cannot exist; there is not even a need to resolve a set of dependency versions. https://www.unison-lang.org/learn/the-big-idea/ https://www.unison-lang.org/learn/the-big-idea/
cryptos 4y ago
A good middleground is Kotlin. You get some of the nice features / syntax from Java without the problems. While Scala is actually a way more powerful language, Kotlin is more than good enough for most real world tasks and the tooling is so much better.
- jillesvangurp 4y ago
  Exactly, you might even go as far as to say that Kotlin exists because the people at Jetbrains looked at and dismissed Scala as just not what they were looking for in a language. And then they went on to develop Kotlin instead. Kotlin is unique in the sense that the language originated from a company whose primary business it is to provide developer tooling and IDEs (intellij). They know a thing or two about what developers like and appreciate in their products. It shows. This language is very nice to use for programmers. Lot's of clever syntax that minimizes boiler plate. Lot's of clever IDE support. Lot's of convenient features. Good build tooling. And so on. In contrast, Scala was always a bit all over the place. Lot's of different takes on how to do things; not all of them very successful in retrospect, tooling was a bit iffy, and so on. It's the reason Kotlin got popular with Android developers early on because it provided a massive improvement over plain Java and sort of worked as a drop in replacement and was easy to switch to. As soon as Kotlin became available (pre 1.0), the Android community was all over it because at the time they were stuck with a pretty outdated Java language version. It did not take long for Google to make it the language of choice and start providing lot's of Kotlin focused updates to their SDK. For the same reason, it got popular with Java backend developers early on. It doesn't really matter what Java backend framework you use; Kotlin is a drop in replacement for Java and the chances are pretty good that your framework of choice already has extensive Kotlin support in the form of Kotlin DSLs, extension functions, etc. That's certainly true for Spring/Spring Boot, Quarkus, and most other mainstream server side Java frameworks you can name. The comparison with Scala is fair because a lot of Kotlin libraries are inspired by work in the Scala community. Additionally, it seems a lot of former Scala developers are now doing a lot of Kotlin. For better or for worse, it seems to compete pretty effectively with Scala. Either language of course is a pretty big improvement over Java; even with the improvements that Oracle has been trickling in over the last few years.

smrtinsert 4y ago

Honestly Java is lately the best it's ever been, at least for backend services. So much power in the frameworks (ok Spring Boot) with so many powerful libraries and tools and available apis. You really feel like you can do anything with it lately.

badpun 4y ago

Java has the same dependency issues? I vividly remember spending days on jars shading in Maven. Basically, a single approach that would solve this, irrespective of the language, would be isolating dependency trees of every library from each other, essentially duplicating the code of many common libraries in RAM (like Docker, but for libraries inside the same process). There's an option to do that in Maven, but it's not the default. And it still would not work for dependencies where there should be only a single instance of it in RAM, like for example loggers...

mbfg 4y ago
One thing is Scala plays fast and loose with backwards compatibility. So much so that akka actually prints out messages if they find jars with different versions. It's a pain in the arse.
zokier 4y ago
> Basically, a single approach that would solve this, irrespective of the language, would be isolating dependency trees of every library from each other Doesn't solve the problem completely because it is normal that applications need libraries to interoperate; consider case where application is passing objects between two libraries that both use types from a third library.
- badpun 4y ago
  Then your app would explicitly be doing the impendance mismatch resolution (converting objects from one version of library to the other) when doing the library calls.
  - CrimsonRain 4y ago
    which someone would have to code in as that's not automatically possible all the time.
- jayd16 4y ago
  You're pretty much screwed in that case but that's kind of by design and the entire point of versioning, isn't it?
- layer8 4y ago
  A major problem is that Maven (and friends) doesn’t distinguish between API dependencies (the API of a library using types from the API of another library) and implementation dependencies (a library depending on another library as an implementation detail). If that distinction were made consistently, implementation dependencies could indeed be separated into a dedicated classloader for each library, and you’d only have to worry about API dependencies, which would go a long way. The Java 9 Modules architecture would have been an opportunity to introduce something like that, but alas they didn’t.
  - brabel 4y ago
    OSGi allowed you to specify in excruciating detail what dependencies you had, whether API (i.e you wanted to re-export types from another library) or not, and much more. Unfortunately, it became far, far more complex than just implementing a simple module system, so as most things from the era (early 2000's) it became a huge, complex beast no one wanted to touch anymore.
  - clhodapp 4y ago
    I believe that separating interface from implementation doesn't really solve the problem, it just shrinks it. E.g. what if you depend on two libraries that re-export classes from different versions of a third library? I think that if you truly want to solve the problem, you need to extend the definition of the fully qualified classname to include the notion of a breaking version reference. Kind of like what folks now sometimes do the hard way by encoding it into the package name. This would make it possible to directly address the two different versions from the same calling source file.
valenterry 4y ago
Jep, this is really an issue of how the JVM and Classloaders are architected. Rust essentially chooses the other side of the sea, which is better in some sense, but worse in others (which we will see once the ecosystem grows). A configurable middleground is needed, but it's not an easy task to get that done.
teknopaul 4y ago
Essentially OSGI is that. It is still a ballache tho, and slow to boot, so I would not recommend it. Best bet it to be picky about dependencies that respect backward compatiblity or do name changes (like commons collections4)
- deleted 4y ago
  [deleted]
mumblemumble 4y ago
I have personally spent so much time fussing with runtime ClassDefNotFound and MethodNotFound errors in Java that I am no longer able to respond to assertions that Java is type-safe with anything but an incoherent stream of cursing and rage tears. But Scala somehow seems to take all of that and intensify it. I have heard that Scala 3 is supposed to improve this, but I don't anticipate being able to adopt Scala 3 before the heat death of the universe.
- jayd16 4y ago
  Not sure what you're doing with Java but that's not a common problem at all. Are you just aggressively stripping code and dealing with the fallout?
  - mumblemumble 4y ago
    Working on projects that involve big frameworks with deep and ever-shifting dependency trees. So upgrading an existing dependency or taking a new one could often be a risky move. Ground zero for most of them was Guava, which I unfortunately could not avoid due to it being a transitive dependency. And, even more unfortunately, Guava types were being exposed on the direct dependency's public API, so package relocation was not an easy option.
    - erik_seaberg 4y ago
      Guava is an outlier. While they do honor semver, they’re already on version 31.1, meaning they have felt they needed to break backward compatibility dozens of times, and it’s likely your dependencies require conflicting versions just by accident of when they were written. I guess it’s because the maintainers live in a monorepo and just force any changes they need?
      - mumblemumble 4y ago
        The other interesting corollary to this is: if your package relies on one like Guava, and it is not bumping its major version number every time it updates the version of Guava it depends on, then your package does not itself honor semver.
kitd 4y ago
JBang [1] can help a lot here, especially with the new "BOM POM" feature. The experience feels closer to Go than Maven. [1] - https://www.jbang.dev/documentation/guide/latest/dependencies.html https://www.jbang.dev/documentation/guide/latest/dependencie...
stickfigure 4y ago
I only very rarely see this in Java, even in huge monoliths. Not that there's a technical reason for this; I think there's a cultural expectation that you don't break backwards compatibility with Java libraries. Perhaps the Scala world is more cutting-edge and fluid so there are different cultural expectations. The OP mentions Jackson. Jackson 1.x -> 2.x changed the package (so they are wholly separate libraries), Jackson 2.x -> 3.x will do it again. I've never had backwards compatibility issues with Jackson. I don't know what someone could be doing in jackson-module-scala that breaks with a minor version bump.
- dkarl 4y ago
  Jackson doesn't have a stable API for building modules. You have to get elbow deep in their internals, which is already scary enough given that it's an old and highly performant library, but the bad part is that there's no formal commitment to maintaining compatibility. You just have to hope that the modules you use are popular enough that they will test them and make an effort not to break them. I've had an application break on a patch version bump of Jackson.
zibby8 4y ago
Java has the same dependency issues. But, from the article, it seems like there might be something cultural within the Scala community that contributes to the problem. Most Java libraries, in my experience, are relatively flat. The article gives an example of HikariCP, which depends on a random JDBC library from Akka. Obviously, that situation is going to end terribly given the popularity of Akka. As soon as Akka updates their library version, HikariCP has to as well or the two dependencies will get hopelessly out of sync. I don't know what `akka-persistence-jdbc ` does, but it can't be so important that it justifies the crazy dependency structure.
- mbfg 4y ago
  Scala doesn't care about backwards compatibility. They break stuff all the time, And so major libraries are tenuously hanging on to whatever version they were targeted at. Akka prints out messages if it finds jars with even different minor versions.
- therealdrag0 4y ago
  HikariCP is a Java library I thought, at least I’ve used it in many Java projects, so I’m not sure what that has to do with Scala.
AtlasBarfed 4y ago
Dependency conflicts is a fundamental issue of mature ecosystems, linux especially included. The issue is one of granularity of dependency: it can't be calculated from the code and dependency graph easily. Each class or function invocation has an associated metadata version, and something that tracks all the sub-ones, with compatibility matrices across the major versions of the library? As in, a dependency / linked library has a new version. How do you know if that can be safely updated in your code? All you have fundamentally is a coarse version number for the entire library, it's types/classes, it's functions/procedures/methods/signatures. Does anything indicate which signatures changed aside from compilers? Does anything mark which functions have new logic in them? Yikes. That can't be maintained by humans. No language has dared to do something that would resolve to that level of compatibility analysis. Can you imagine the time involved? Maybe there's simpler indicators. Git repo as a service might be able to use the repo history to track what parts of a library actually changed with some metadata file at a sufficiently granular level. Currently, convention dictates that you use a new class name or namespace or something similar for "breaking" changes, or a major version number boost with some assumed "service life" for the previous library major version. You know, if they do that. But there's instances (one of the java bytcode emission libraries IIRC) where the api remainined the same but was a breaking change. Nothing exists to help a library maintainer be aware of what will be a breaking change. At granular levels that is inherent to the language design. At best, you have unit tests as the actually deep introspection of the interface and expected responses. But that is per-project. As mentioned, you can also shade and dynamically renamespace a library. That almost is the solution, but shared libraries are a thing, and a if you have 10 libraries each shaded-using a common util library (ahem, slf4j, first item in the blog post), then that's precious ram wasted, even in modern RAM sizes. With three-deep versions like 4.0.7 or something similar, how would a build system know that it could safely reuse a shaded version? It basically can't. There's no language constructs for saying that it can. To some degree I believe this a failure, in the case of Java, of some sort of overarching organization that tracks commons libraries, and while maybe not placing them in the language standard library, recommending best practices around those libraries across known usage patterns so library import/compiling can be gradually optimized, or common conflicts alleviated. It's almost like security patching to some degree. Maven and its repos aren't up to the task. You have a new language? Likely nothing in it will killer app it, instead what makes modern new languages succeed isn't just novel language features, it's community and herding cats. It's what Zig just learned from one tweet, what Lisp's community never really learned for what otherwise should be the "ultimate" language. It's how Rust has made progress, and maybe Go too. I wonder if in 200 years if there will actually be mature, stable languages with mature, stable libraries that haven't needed to change for a couple decades. Anyway, "compatibility" is a hard hard hard problem.
oftenwrong 4y ago
Dependency management is painful almost everywhere. I see a glimmer of hope for the future in Unison, a language in which code exists as content-addressed ASTs. Dependency version conflicts cannot exist; there is not even a need to resolve a set of dependency versions. https://www.unison-lang.org/learn/the-big-idea/ https://www.unison-lang.org/learn/the-big-idea/

cryptos 4y ago

A good middleground is Kotlin. You get some of the nice features / syntax from Java without the problems. While Scala is actually a way more powerful language, Kotlin is more than good enough for most real world tasks and the tooling is so much better.

jillesvangurp 4y ago
Exactly, you might even go as far as to say that Kotlin exists because the people at Jetbrains looked at and dismissed Scala as just not what they were looking for in a language. And then they went on to develop Kotlin instead. Kotlin is unique in the sense that the language originated from a company whose primary business it is to provide developer tooling and IDEs (intellij). They know a thing or two about what developers like and appreciate in their products. It shows. This language is very nice to use for programmers. Lot's of clever syntax that minimizes boiler plate. Lot's of clever IDE support. Lot's of convenient features. Good build tooling. And so on. In contrast, Scala was always a bit all over the place. Lot's of different takes on how to do things; not all of them very successful in retrospect, tooling was a bit iffy, and so on. It's the reason Kotlin got popular with Android developers early on because it provided a massive improvement over plain Java and sort of worked as a drop in replacement and was easy to switch to. As soon as Kotlin became available (pre 1.0), the Android community was all over it because at the time they were stuck with a pretty outdated Java language version. It did not take long for Google to make it the language of choice and start providing lot's of Kotlin focused updates to their SDK. For the same reason, it got popular with Java backend developers early on. It doesn't really matter what Java backend framework you use; Kotlin is a drop in replacement for Java and the chances are pretty good that your framework of choice already has extensive Kotlin support in the form of Kotlin DSLs, extension functions, etc. That's certainly true for Spring/Spring Boot, Quarkus, and most other mainstream server side Java frameworks you can name. The comparison with Scala is fair because a lot of Kotlin libraries are inspired by work in the Scala community. Additionally, it seems a lot of former Scala developers are now doing a lot of Kotlin. For better or for worse, it seems to compete pretty effectively with Scala. Either language of course is a pretty big improvement over Java; even with the improvements that Oracle has been trickling in over the last few years.