8 ms·

Scala's version fragility makes the enterprise argument near impossible

99 points by LiveTheDream 15y ago

gldalmaso 15y ago
>> An attribute of Scala is that the Scala compiler generates fragile byte-code. Can anybody tell me if this would be true to any JVM targeted language, or is it a specific issue of the Scala compiler? I'm asking because I just recently decided to abort learning Scala in favor of Clojure.
- daniel_solano 15y ago
  As both a Scala and Clojure user, I believe this is primarily a Scala-specific issue. The one caveat to that is that, if you're using Clojure and AOT-compile your code, you can run into some issues if you are trying to use AOT-compiled code from different versions (or using a different version of Clojure than the one used to AOT-compile the code). Now, there have been some source-level incompatibilities, especially going from Clojure 1.2 to 1.3, and this seems to have been a bit of a headache within some parts of the Clojure community.
- cemerick 15y ago
  As long as you're not AOT-compiling Clojure (i.e. producing class files, which is generally entirely unnecessary in Clojure except perhaps as the last step just prior to application delivery / deployment), you'll generally be very happy. The 1.2 -> 1.3 transition has had a couple of bumps, but nothing compared to the pain I recall from my time across Scala versions years ago. FWIW, I have code I wrote years ago against pre-1.0 Clojure that still sits in jars in my Maven repository that is loaded and run in 1.4 alphas today. Backwards compatibility is mostly quite good.
  - CountHackulus 15y ago
    In the JIT world, going .java->.class is not AOT. You may call it that, but it's essentially just source->source conversion. What AOT really is, is when you first run a java program with certain flags, and some compilation starts, a special AOT version, which is usually a low optimization level which is saved to disk with some relocation markers. This helps with startup time in subsequent runs as the compiler can just load its baseline pre-compiled classes straight into memory instead of having to do a bunch of grunt work during startup. Once things get going, the compiler can much more quickly work its way up to the higher optimization levels on the sections of code that need it. THAT is what AOT means from a jitted language perspective.
    - gtrak 15y ago
      Java bytecode is what we mean when we say 'compile'. AOT is 'ahead of time' with regard to clojure, since it compiles to bytecode at run-time normally.
    - ataggart 15y ago
      "AOT" in the clojure world means compiling your clojure source down to JVM bytecode, which also means compiled-in calls against a particular state of clojure's internal java api. By shipping clojure source, one avoids any concerns over incidental changes to internal code.
- taeric 15y ago
  Pretty sure this isn't even language specific. If you are using a library that drastically modifies how they implement something between versions, then you will run into this. Even in pure Java. Now, the ecosystem in Java is to usually not make large changes. At least not at the public API level. I think Scala is slowing down in this area, but have not stopped.
- allertonm 15y ago
  It's not true of Java - the bytecode generated by javac is very robust in the face of new JVM versions. This is one of the reasons that Java has had to move so slowly in terms of language features. Neither Scala or Clojure's compilers generate byte code that is expected to work with that from another version of the respective compiler or with a different version of the language runtime. In the case of Clojure this is mitigated by the fact that (since it is a dynamic language) that applications and libraries are generally delivered in source form and "ahead of time" compilation is seen as an optimization rather than a code delivery mechanism.
nobodysbusiness 15y ago
A recent Typesafe blog post was written in response to the last Scala-complaint furor: http://blog.typesafe.com/getting-down-to-work http://blog.typesafe.com/getting-down-to-work One of the points mentioned there is binary compatibility. The Typesafe people are working on it, so hopefully we'll see some improvement over the next year or so.
jzoidberg 15y ago
David carries quite a lot of weight in my book (we use Lift in production). Despite the negative title he makes a very good point and points out that this is entirely solvable. This falls under the "growing pains" category for Scala. I hope we get this right soon.
donjigweed 15y ago
Really, it is just becoming increasingly difficult to see how proponents of Scala can credibly claim any kind of developer productivity gains. I can only imagine how much time your average Scala developer spends solving problems of accidental complexity that simply do not exist in more mature languages.
- noelwelsh 15y ago
  As a Scala developer, let me shed some light on this. In the last few months I have spent exactly zero time on this problem. It was a minor issue when 2.9 came out. No one forces you to upgrade, and we didn't till all our dependencies worked on 2.9.
  - rickmode 15y ago
    And your organization has much more than 3 teams? The article specifically says this is a large multi-team coordination issue.
    - lliiffee 15y ago
      noelwelsh is responding to donjigweed, not the article.
- SWengineer 15y ago
  Time spent: zilch, perhaps because most of my dependencies are java libs not scala ones. Like many Java devs starting out using Scala as a 'better java' which means adding more scala libs over time. It's definitely very pleasant to code in Scala vs Java, def more productive, to a surprising extent even. The compatibility issue needs to be addressed though.
  - brightsize 15y ago
    Same here, zero time spent on this. Scala is a joy to program in. It's good to see these kinds of postings with complaints and suggestions. They'll move the language and the ecosystem forward.
stephenjudkins 15y ago
This is the largest practical issue I've had with developing a Scala project. One major cause of the problem seems to be that Scala is stuck with the "precompiled bytecode in JARs" model of software distribution; while this may be appropriate for Java (where core developers go through great pains to maintain compatibility between versions) it seems like a poor fit for Scala. Not only do the core library APIs change between versions, but the conventions used by the compiler to "link" Scala code together are also altered, for good reason. An alternative distribution mechanism that's more resistant to these kinds of issues might make sense. Even just distributing source code and build files (like, say, Ruby) could be a better solution. Building dependencies might be slow at first, but centralized caching could address this. Another solution might be an intermediate representation (after type-checking and other compiler phases, but before "linking") that gives the Scala compiler the knowledge that's missing from compiled class files. Even namespace-mangling or class-loader magic could be helpful here. Loading multiple versions of a single dependency should in theory be possible, since OSGi manages it. Sadly, there seems to be a pretty large investment by the Scala community in the whole Java/Maven/JAR ecosystem. It'd take a concerted effort by many people to create a more robust solution.
- carterschonwald 15y ago
  I think you've hit the nail on the head, that having a centralized package manager tool/host, a la the haskell hackage / cabal-install or the like, would substantially simplify the problem, but because of desired JVM library compatibility this path isn't explored as much as might be ideal
- keturn 15y ago
  Thanks for this comment, as not yet having got into Scala myself, I had to read between the lines to figure out what the problem was here. I thought "Ok, you don't have binary compatibility, so recompile your dependencies. Sure, it may extend your build time somewhat, but it shouldn't make it impossible to use dependencies or test pre-release builds..." Oh, you don't _have_ the source? Well, there's your problem. A good reminder of what some of us take for granted in our development environments, I guess.
jbarham 15y ago
Go is also moving very fast and they often make backwards-incompatible changes to the libraries (and sometimes the core language) but upgrading code can be done automatically 95% of the time using gofix, which programmatically rewrites code to conform to the latest language and library updates. http://blog.golang.org/2011/04/introducing-gofix.html http://blog.golang.org/2011/04/introducing-gofix.html http://golang.org/cmd/gofix/ http://golang.org/cmd/gofix/
- angelbob 15y ago
  Wow. So it just rewrites it to the latest version, with (according to what you just said) about 95% accuracy? Does that make other people feel queasy, or just me? Like, that feels like a really bad idea to me. What happens if the result only mostly works?
  - dmm 15y ago
    It's a language undergoing rapid development. The developers are planning sometime in early 2012, to release a "Version 1" of go, to which no backwards incompatible changes will be made. http://blog.golang.org/2011/10/preview-of-go-version-1.html http://blog.golang.org/2011/10/preview-of-go-version-1.html
  - timclark 15y ago
    Typically gofix just works, well that is my experience of running gofix three times over the six last months. The release notes document when the language has changed and when gofix can help you migrate, and sometimes it can't help and it is documented. Additionally, go exposes the complete AST for the language and a pretty printer as libraries, so you can write your own fixes if you don't trust gofix.
  - sausagefeet 15y ago
    Python has been doing this for awhile, hasn't it?
    - DasIch 15y ago
Roboprog 15y ago
I expect this to be exactly the sort of thing that a commercial vendor will support: here is the language version and library versions that we support. Don't upgrade any of these until we tell you that you can, or you're on your own. I'm just starting to experiment with Scala for scripting on top of existing Java, so the current state is not a hardship for me. The article is helpful in pointing out the task (technical as well as P/R) to be faced by any would-be "golden standard" vendor, though.
latchkey 15y ago
How about taking some time off from Lift development to implement the ideas that you and other worked on?
- soc88 15y ago
  That's pretty much what I think every time he spends time on his anti-Scala rants. Maybe he should switch the language? Looking at Lift's source code/API this is pretty much a abuse of the language.
  - billmcneale 15y ago
    He's one of the few true Scala experts around (how many can claim they've been using Scala for more than five years?). The code he writes is probably as idiomatic Scala as it can get. Yet another reason why people should be wary.
    - dkhenry 15y ago
      No he is a scala early adopter. There is a big difference. I have less and less respect for the man the more I look at Lift. In fact after I started working with Play and their scala support I have and even better sense that he is just a java developer who found a new way to manipulate XML files.
      - billmcneale 15y ago
        We're not discussing Lift nor his design skills. We're talking about his level of expertise in Scala, which is probably one of the highest in the community.
eweise 15y ago
A response by J. Suereth to this post http://suereth.blogspot.com/2011/12/scala-fresh-is-alive.html http://suereth.blogspot.com/2011/12/scala-fresh-is-alive.htm...
Uchikoma 15y ago
I've raised this issue one year ago http://codemonkeyism.com/scala-unfit-development/ http://codemonkeyism.com/scala-unfit-development/ There are two forces at play here: a.) Binary fragility of Scala b.) A community that upgrades libraries (your dependencies) to every new RC candidate - even when only fixing bugs. So if you have a show stopper bug you need to upgrade to the newest RC breaking all your other dependencies. Either a. and b. would be no major problem, combined it throws you into a dependency hell for days hunting for dependencies that work. Hope they resolve this issue. People promised a solution in the blog comments, then promised a solution a year later as the issue surfaced again on the #typesafe blog.

99 points by LiveTheDream 15y ago

>> An attribute of Scala is that the Scala compiler generates fragile byte-code. Can anybody tell me if this would be true to any JVM targeted language, or is it a specific issue of the Scala compiler? I'm asking because I just recently decided to abort learning Scala in favor of Clojure.

daniel_solano 15y ago
As both a Scala and Clojure user, I believe this is primarily a Scala-specific issue. The one caveat to that is that, if you're using Clojure and AOT-compile your code, you can run into some issues if you are trying to use AOT-compiled code from different versions (or using a different version of Clojure than the one used to AOT-compile the code). Now, there have been some source-level incompatibilities, especially going from Clojure 1.2 to 1.3, and this seems to have been a bit of a headache within some parts of the Clojure community.
cemerick 15y ago
As long as you're not AOT-compiling Clojure (i.e. producing class files, which is generally entirely unnecessary in Clojure except perhaps as the last step just prior to application delivery / deployment), you'll generally be very happy. The 1.2 -> 1.3 transition has had a couple of bumps, but nothing compared to the pain I recall from my time across Scala versions years ago. FWIW, I have code I wrote years ago against pre-1.0 Clojure that still sits in jars in my Maven repository that is loaded and run in 1.4 alphas today. Backwards compatibility is mostly quite good.
- CountHackulus 15y ago
  In the JIT world, going .java->.class is not AOT. You may call it that, but it's essentially just source->source conversion. What AOT really is, is when you first run a java program with certain flags, and some compilation starts, a special AOT version, which is usually a low optimization level which is saved to disk with some relocation markers. This helps with startup time in subsequent runs as the compiler can just load its baseline pre-compiled classes straight into memory instead of having to do a bunch of grunt work during startup. Once things get going, the compiler can much more quickly work its way up to the higher optimization levels on the sections of code that need it. THAT is what AOT means from a jitted language perspective.
  - gtrak 15y ago
    Java bytecode is what we mean when we say 'compile'. AOT is 'ahead of time' with regard to clojure, since it compiles to bytecode at run-time normally.
  - ataggart 15y ago
    "AOT" in the clojure world means compiling your clojure source down to JVM bytecode, which also means compiled-in calls against a particular state of clojure's internal java api. By shipping clojure source, one avoids any concerns over incidental changes to internal code.
taeric 15y ago
Pretty sure this isn't even language specific. If you are using a library that drastically modifies how they implement something between versions, then you will run into this. Even in pure Java. Now, the ecosystem in Java is to usually not make large changes. At least not at the public API level. I think Scala is slowing down in this area, but have not stopped.
allertonm 15y ago
It's not true of Java - the bytecode generated by javac is very robust in the face of new JVM versions. This is one of the reasons that Java has had to move so slowly in terms of language features. Neither Scala or Clojure's compilers generate byte code that is expected to work with that from another version of the respective compiler or with a different version of the language runtime. In the case of Clojure this is mitigated by the fact that (since it is a dynamic language) that applications and libraries are generally delivered in source form and "ahead of time" compilation is seen as an optimization rather than a code delivery mechanism.

nobodysbusiness 15y ago

A recent Typesafe blog post was written in response to the last Scala-complaint furor: http://blog.typesafe.com/getting-down-to-work http://blog.typesafe.com/getting-down-to-work One of the points mentioned there is binary compatibility. The Typesafe people are working on it, so hopefully we'll see some improvement over the next year or so.

jzoidberg 15y ago

David carries quite a lot of weight in my book (we use Lift in production). Despite the negative title he makes a very good point and points out that this is entirely solvable. This falls under the "growing pains" category for Scala. I hope we get this right soon.

donjigweed 15y ago

Really, it is just becoming increasingly difficult to see how proponents of Scala can credibly claim any kind of developer productivity gains. I can only imagine how much time your average Scala developer spends solving problems of accidental complexity that simply do not exist in more mature languages.

noelwelsh 15y ago
As a Scala developer, let me shed some light on this. In the last few months I have spent exactly zero time on this problem. It was a minor issue when 2.9 came out. No one forces you to upgrade, and we didn't till all our dependencies worked on 2.9.
- rickmode 15y ago
  And your organization has much more than 3 teams? The article specifically says this is a large multi-team coordination issue.
  - lliiffee 15y ago
    noelwelsh is responding to donjigweed, not the article.
SWengineer 15y ago
Time spent: zilch, perhaps because most of my dependencies are java libs not scala ones. Like many Java devs starting out using Scala as a 'better java' which means adding more scala libs over time. It's definitely very pleasant to code in Scala vs Java, def more productive, to a surprising extent even. The compatibility issue needs to be addressed though.
- brightsize 15y ago
  Same here, zero time spent on this. Scala is a joy to program in. It's good to see these kinds of postings with complaints and suggestions. They'll move the language and the ecosystem forward.

stephenjudkins 15y ago

This is the largest practical issue I've had with developing a Scala project. One major cause of the problem seems to be that Scala is stuck with the "precompiled bytecode in JARs" model of software distribution; while this may be appropriate for Java (where core developers go through great pains to maintain compatibility between versions) it seems like a poor fit for Scala. Not only do the core library APIs change between versions, but the conventions used by the compiler to "link" Scala code together are also altered, for good reason. An alternative distribution mechanism that's more resistant to these kinds of issues might make sense. Even just distributing source code and build files (like, say, Ruby) could be a better solution. Building dependencies might be slow at first, but centralized caching could address this. Another solution might be an intermediate representation (after type-checking and other compiler phases, but before "linking") that gives the Scala compiler the knowledge that's missing from compiled class files. Even namespace-mangling or class-loader magic could be helpful here. Loading multiple versions of a single dependency should in theory be possible, since OSGi manages it. Sadly, there seems to be a pretty large investment by the Scala community in the whole Java/Maven/JAR ecosystem. It'd take a concerted effort by many people to create a more robust solution.

carterschonwald 15y ago
I think you've hit the nail on the head, that having a centralized package manager tool/host, a la the haskell hackage / cabal-install or the like, would substantially simplify the problem, but because of desired JVM library compatibility this path isn't explored as much as might be ideal
keturn 15y ago
Thanks for this comment, as not yet having got into Scala myself, I had to read between the lines to figure out what the problem was here. I thought "Ok, you don't have binary compatibility, so recompile your dependencies. Sure, it may extend your build time somewhat, but it shouldn't make it impossible to use dependencies or test pre-release builds..." Oh, you don't _have_ the source? Well, there's your problem. A good reminder of what some of us take for granted in our development environments, I guess.

jbarham 15y ago

Go is also moving very fast and they often make backwards-incompatible changes to the libraries (and sometimes the core language) but upgrading code can be done automatically 95% of the time using gofix, which programmatically rewrites code to conform to the latest language and library updates. http://blog.golang.org/2011/04/introducing-gofix.html http://blog.golang.org/2011/04/introducing-gofix.html http://golang.org/cmd/gofix/ http://golang.org/cmd/gofix/

angelbob 15y ago
Wow. So it just rewrites it to the latest version, with (according to what you just said) about 95% accuracy? Does that make other people feel queasy, or just me? Like, that feels like a really bad idea to me. What happens if the result only mostly works?
- dmm 15y ago
  It's a language undergoing rapid development. The developers are planning sometime in early 2012, to release a "Version 1" of go, to which no backwards incompatible changes will be made. http://blog.golang.org/2011/10/preview-of-go-version-1.html http://blog.golang.org/2011/10/preview-of-go-version-1.html
- timclark 15y ago
  Typically gofix just works, well that is my experience of running gofix three times over the six last months. The release notes document when the language has changed and when gofix can help you migrate, and sometimes it can't help and it is documented. Additionally, go exposes the complete AST for the language and a pretty printer as libraries, so you can write your own fixes if you don't trust gofix.
- sausagefeet 15y ago
  Python has been doing this for awhile, hasn't it?
  - DasIch 15y ago

Roboprog 15y ago

I expect this to be exactly the sort of thing that a commercial vendor will support: here is the language version and library versions that we support. Don't upgrade any of these until we tell you that you can, or you're on your own. I'm just starting to experiment with Scala for scripting on top of existing Java, so the current state is not a hardship for me. The article is helpful in pointing out the task (technical as well as P/R) to be faced by any would-be "golden standard" vendor, though.

latchkey 15y ago

How about taking some time off from Lift development to implement the ideas that you and other worked on?

soc88 15y ago
That's pretty much what I think every time he spends time on his anti-Scala rants. Maybe he should switch the language? Looking at Lift's source code/API this is pretty much a abuse of the language.
- billmcneale 15y ago
  He's one of the few true Scala experts around (how many can claim they've been using Scala for more than five years?). The code he writes is probably as idiomatic Scala as it can get. Yet another reason why people should be wary.
  - dkhenry 15y ago
    No he is a scala early adopter. There is a big difference. I have less and less respect for the man the more I look at Lift. In fact after I started working with Play and their scala support I have and even better sense that he is just a java developer who found a new way to manipulate XML files.
    - billmcneale 15y ago
      We're not discussing Lift nor his design skills. We're talking about his level of expertise in Scala, which is probably one of the highest in the community.

eweise 15y ago

A response by J. Suereth to this post http://suereth.blogspot.com/2011/12/scala-fresh-is-alive.html http://suereth.blogspot.com/2011/12/scala-fresh-is-alive.htm...

Uchikoma 15y ago

I've raised this issue one year ago http://codemonkeyism.com/scala-unfit-development/ http://codemonkeyism.com/scala-unfit-development/ There are two forces at play here: a.) Binary fragility of Scala b.) A community that upgrades libraries (your dependencies) to every new RC candidate - even when only fixing bugs. So if you have a show stopper bug you need to upgrade to the newest RC breaking all your other dependencies. Either a. and b. would be no major problem, combined it throws you into a dependency hell for days hunting for dependencies that work. Hope they resolve this issue. People promised a solution in the blog comments, then promised a solution a year later as the issue surfaced again on the #typesafe blog.