6 ms·

Currently upgrading a ~1M SLoC Java enterprise app (with regular Spring, Jetty in there, scheduled processes, PrimeFaces for web UI, REST API services, SOAP ser

Currently upgrading a ~1M SLoC Java enterprise app (with regular Spring, Jetty in there, scheduled processes, PrimeFaces for web UI, REST API services, SOAP services, the whole shebang) from Java 8 to Java 11 (since 17 wasn't out when that change was approved) and it's largely proving to be a pain. Since the version of Spring is ancient and changes over to Spring Boot were also approved, now have to rewrite parts of it and also get rid of the XML configuration and other old approaches which are simply no longer compatible with this upgraded tech stack. It feels like something that perhaps a team should do, instead of one dev over a month or so, but scope creep and estimates for something like that are nigh impossible, so we'll see. My point is that migrating to new releases isn't always trivial, especially the more complicated and complex a project gets. If i knew that i'll run if compiled successfully, then it wouldn't be too bad, but with the amount of reflection, dynamic class loading etc. that frameworks like Spring favor, my workflow to date has been fixing a bug, building and running, something else breaking, fixing that bug, building and running, something else breaking, realizing that i cannot fix this because upgrades to the logic would be inherently "lossy" due to a mismatch of what the new framework versions provide, making it so that breakages that aren't covered by tests will also be created and so on ad infinitum. Sometimes it makes me wonder why JDK 9 onward just didn't have a compatibility module: "Here, install this to have all of the old Java classes that were removed from the standard library after JDK 8 and retain that old functionality for software projects that would otherwise be stuck in development hell short of a full rewrite." Or something like that for Spring Boot, that lets you use web.xml instead of having to use hacks to load its contents and register all of the servlets, with half of them not working anyways, because some class names were changed along the way. Software doesn't always age beautifully. Furthermore, it feels like new runtime and package versions are made in ways that accidentally break backwards compatibility with no clear ways to avoid this.

virtue3 5y ago
I think this is one of the few benefits of micro service architectures. You end up with much smaller pieces to migrate over. And if the service is well defined and you have ample logging for endpoints you actually could re-write it. Mind you; you usually end up with the services all in different states of node/jdk/etc versioning(at least the companies I worked at).
chrisseaton 5y ago
Can you give a specific technical problem with updating from 8 to 11? Everyone says it's hard - nobody's ever able to give a practical example of a problem.
- KronisLV 5y ago
  I sadly didn't write everything down, or didn't keep references to the GitHub issues, but here's a few of the things that i ran into. Here's a list of the things that were available in JDK 8 and no longer are: https://advancedweb.hu/a-categorized-list-of-all-java-and-jvm-features-since-jdk-8-to-17/#deprecation-and-removal https://advancedweb.hu/a-categorized-list-of-all-java-and-jv... When you stumble upon something like that, at best you can import a Maven dependency with a version that has what you need, at worst you need to rewrite the code to use another library or set of libraries. If you have any low level logic like custom class loaders written against older JDK versions (think before 8), then they'll be forwards compatible until 8 for the most part, but will break afterwards. Coincidentally, reading code that deals with low level logic is also not easy to do, especially if it's not commented well. If you rely upon reflection, or use advanced language features (like the JasperReports framework for generating PDFs, which also has a build step for building the reports), in some cases things might compile but not work at runtime due to class mismatches. Many frameworks need new major versions to support newer releases than JDK 8, for example Spring Boot 1.5 needs to be upgraded, so you're also dealing with all the changes that are encapsulated by your dependencies. In another project that i also migrated, needed to rewrite a lot of web initialization code for Spring Boot 2.X. Not only that, but with those framework changes, certain things can break in the actual environments. For example, if you package your app as a far .jar, then you'll no longer be able to serve JSP files out of it. It makes no sense, but packaging it as a .war which can be executed with "java -jar your-app.war" will work for some reason. I some other libraries, method names remain the same, but signatures change, or sometimes things just get deprecated and removed - you have to deal with all of that, which is especially unfun in code that isn't commented but that the business depends on to work correctly. Throw in external factors such as insufficient coverage of tests and you're in for an interesting time. I'm not saying that it's a type of environment that should be condoned, but it's the objective reality in many software projects. Oh and i hope that you're also okay with updating all of your app servers (like Tomcat) or JDK installs on the server as well, especially if you depend on some of the Tomcat libraries to be provided and for your frameworks/libraries that depend on them being present at runtime to accept those new versions effortlessly. It all feels very brittle at times. This is especially a nightmare if your servers were configured manually - personally i'm introducing Ansible and containers to at least isolate the damage of Ops rot, but it's been an uphill battle since day 1. Here's an exceedingly stupid one: sometimes there are checks in code to make sure that you're using the right version of JDK (or even the Oracle JDK), which get confused with the larger versions. It's easy to fix when it's your code, but really annoying when it's external tools - personal nitpick. Addendum: here's something that i expected to break, but didn't. We use myBatis as an ORM. It has XML mapper files that define how to map entities and construct queries against the DB based on that. Also, it uses Java interfaces and dynamically calls the underlying XML code as necessary. So essentially you have an interface, which has a corresponding XML file in which you have quasi-Java code (e.g. checking what parameters are passed in to the query) that's used alongside a number of tags to dynamically build SQL queries. Here's an example: https://mybatis.org/mybatis-3/dynamic-sql.html https://mybatis.org/mybatis-3/dynamic-sql.html Instead of something breaking in myBatis, what broke actually was Hibernate in the newer versions of Spring. Oh, and Flyway for DB migrations simply used to work with a particular Oracle DB version as well.
krzyk 5y ago
JDK 8 -> 9 migration was painful, but all other upgrades should be painless (well, maybe the 16-17, but I assume that most libraries will be fixed by the time you upgrade - either add `add-opens` instructions or use appropriate new classes from JDK).
__jem 5y ago
I got stuck in this conundrum at my previous gig as well, and think reached a very different conclusion than you. In short, while I’m skeptical of some of the degree of churn across modern software development as a whole, I’m not sure that this is actually an issue with the Java ecosystem, which has been, and I think continues to be, way more sensitive to backwards compatibility than almost any other ecosystem. In some ways, this is just a consequence of a kind of evolutionary leap forward for many applications. We don’t deploy our WARs into application servers anymore, we try to pack a fat JAR, or maybe even a statically linked runtime, into a container. We rebuild the whole world on CI all the time. I mean, Java 8 has issues just respecting cgroup limits, which many a despondent ops engineer has discovered. I think there’s still room for the 1M SLoC monolith even in this new world, but there are real benefits on the horizon for upgrading to Java 17 and beyond. We’ve reached a point where it’s just more expensive not to make upgrades a regular part of you development cycle. And, no one is stopping you from just staying on 1.8. I mean, I’m sure there are still decaying enterprises stranded on 1.3.
- KronisLV 5y ago
  Personally, i'd never let a single monolith grow that far. Having a problem component or two that could be left on JDK 8 until they die and are rewritten (which would actually be possible with systems that have clearly defined boundaries and smaller scope), while migrating everything else. Sadly, i don't get that choice, nor do i get the choice to make the judgement call to leave the monolith on JDK 8 in the name of stability. But hey, at least i'm paid a bit of money for it, so i have some motivation to work on all of it to the best of my abilities, learn a bit more about the JDK internals, have a word or two to tell others about my experience before i inevitably burn out from the churn. Personally, i do still think that Java is pretty good when it comes to stability and backwards and forwards compatibility, especially since tooling like Maven is actually pretty good when compared to the alternatives (well, the parent POM functionality is confusing sometimes, but oh well, it has its use cases). That said, JDK 8 to newer releases was indeed a generational shift (probably one for the better) and you see similar things with Spring Boot 1.5 to Spring Boot 2.X, those breaking releases are inevitable. I only wish that the things blocking people from writing more modular systems would disappear over time, so that huge monoliths that are incredibly hard to work with wouldn't be such an issue. I'm not necessarily advocating for microservices here, since people tend to go straight from one ditch into the opposite one (multiple services per person, nightmarish service mesh, needlessly large % of the code being for shuffling data around, suddenly building a distributed system even within a single domain), but at the very least it would be really nice to have someone look at it and say: "Hey, this PDF generation logic looks really brittle and perhaps should be a service of its own." or maybe: "Oh, hey, we're serving our RESTful API calls and front end resources from the same application, maybe we should have a separate front end app, served through a regular web server?" Then again, if nothing else, i always have the choice to choose where i'm employed, even though i don't feel like letting my coworkers down at the moment either.
swsieber 5y ago
I hope you move to spring voot first vefore upgrading to JDK11, or vice versa. I'm in the middle of a migration to 11 and my general approach has been to get it compiling on 11 while targeting 8 with (--release 8). That seems to catch a great many things that would otherwise fail. The hardest part thus far was figuring out the appropriate libs to replace Java 8 ee stuff, mostly wrt soap and xml. I'm hoping moving to 11 goes smoothly but we'll see. I'm hoping to have the artifacts to a point where they can run on either 8 or 11 without issues.
stickfigure 5y ago
It sounds like the problem is the scope creep. Do the JDK upgrade or the spring boot migration first. Yes it is tempting to do both at once but you are adding a lot of risk.
- KronisLV 5y ago
  Agreed! Sadly, it wasn't my choice to make. Then again, if it's an older version of Spring, chances are that a lot of changes would be needed regardless.
AmpsterMan 5y ago
We had that about a year ago in our product as well, though from Java 7 to Java 11. That JDK 9+ hump was definitely bad, but since then each incremental JDK has been smooth. The idea of using "the latest JDK" is fairly modern in the Java world, so people are still getting used to it. I'd say it's good now though, with perhaps 16 -> 17 being a minor hump since things that were deprecated but allowed in 9+ are now removed.
zz865 5y ago
Just do the 8-11 upgrade first. The XML part has a lot of life left in it, still lots of advantages over attributes.
achoice 5y ago
Upgrading Spring and especially major Boot versions is usually much more work than bumping Java version. Boot change how it does config and such which .. can be confusing.
grumpyprole 5y ago
Isn't your problem above essentially with Spring rather than the new JDK? Presumably Spring is using internal APIs that have changed in JDK11. And yes, the reflection and dynamic behaviour in Spring are not type safe (i.e. runtime failures). Spring XML is a manifestation of "Greenspun's tenth rule" - it gives people the flexibility of Lisp/Python by unsurprisingly removing static type safety - not a trade-off that many will agree with. You should tell management that your job had been made harder by the choices made by the original architects of the system (who I assume are long gone),
nomercy400 5y ago
Aa somebody who has had to do a Spring upgrade in the past, from 2.x to 4.x, I suggest you do Spring first, then the JDK after. Spring still supports 8 well after it starts supporting 11. Once you reach a certain version of Spring, upgrading the JDK is as simple as changing a 1.8 into an 11. I don't know PrimeFaces, but I have worked with JSP, which does have a flag to set the compiler version (in case you were ever thinking about putting java lambda's in JSP files, which is possible but not recommended due to mixing code). The hardest part of upgrading the JDK is going from 8 to 9. After 9 it is much more forgivable. As for Spring, major versions are a pain, as is Hibernate. The other parts of Spring are generally well upgradable if you read the Spring changelogs.

virtue3 5y ago

I think this is one of the few benefits of micro service architectures. You end up with much smaller pieces to migrate over. And if the service is well defined and you have ample logging for endpoints you actually could re-write it. Mind you; you usually end up with the services all in different states of node/jdk/etc versioning(at least the companies I worked at).

chrisseaton 5y ago

Can you give a specific technical problem with updating from 8 to 11? Everyone says it's hard - nobody's ever able to give a practical example of a problem.

KronisLV 5y ago
I sadly didn't write everything down, or didn't keep references to the GitHub issues, but here's a few of the things that i ran into. Here's a list of the things that were available in JDK 8 and no longer are: https://advancedweb.hu/a-categorized-list-of-all-java-and-jvm-features-since-jdk-8-to-17/#deprecation-and-removal https://advancedweb.hu/a-categorized-list-of-all-java-and-jv... When you stumble upon something like that, at best you can import a Maven dependency with a version that has what you need, at worst you need to rewrite the code to use another library or set of libraries. If you have any low level logic like custom class loaders written against older JDK versions (think before 8), then they'll be forwards compatible until 8 for the most part, but will break afterwards. Coincidentally, reading code that deals with low level logic is also not easy to do, especially if it's not commented well. If you rely upon reflection, or use advanced language features (like the JasperReports framework for generating PDFs, which also has a build step for building the reports), in some cases things might compile but not work at runtime due to class mismatches. Many frameworks need new major versions to support newer releases than JDK 8, for example Spring Boot 1.5 needs to be upgraded, so you're also dealing with all the changes that are encapsulated by your dependencies. In another project that i also migrated, needed to rewrite a lot of web initialization code for Spring Boot 2.X. Not only that, but with those framework changes, certain things can break in the actual environments. For example, if you package your app as a far .jar, then you'll no longer be able to serve JSP files out of it. It makes no sense, but packaging it as a .war which can be executed with "java -jar your-app.war" will work for some reason. I some other libraries, method names remain the same, but signatures change, or sometimes things just get deprecated and removed - you have to deal with all of that, which is especially unfun in code that isn't commented but that the business depends on to work correctly. Throw in external factors such as insufficient coverage of tests and you're in for an interesting time. I'm not saying that it's a type of environment that should be condoned, but it's the objective reality in many software projects. Oh and i hope that you're also okay with updating all of your app servers (like Tomcat) or JDK installs on the server as well, especially if you depend on some of the Tomcat libraries to be provided and for your frameworks/libraries that depend on them being present at runtime to accept those new versions effortlessly. It all feels very brittle at times. This is especially a nightmare if your servers were configured manually - personally i'm introducing Ansible and containers to at least isolate the damage of Ops rot, but it's been an uphill battle since day 1. Here's an exceedingly stupid one: sometimes there are checks in code to make sure that you're using the right version of JDK (or even the Oracle JDK), which get confused with the larger versions. It's easy to fix when it's your code, but really annoying when it's external tools - personal nitpick. Addendum: here's something that i expected to break, but didn't. We use myBatis as an ORM. It has XML mapper files that define how to map entities and construct queries against the DB based on that. Also, it uses Java interfaces and dynamically calls the underlying XML code as necessary. So essentially you have an interface, which has a corresponding XML file in which you have quasi-Java code (e.g. checking what parameters are passed in to the query) that's used alongside a number of tags to dynamically build SQL queries. Here's an example: https://mybatis.org/mybatis-3/dynamic-sql.html https://mybatis.org/mybatis-3/dynamic-sql.html Instead of something breaking in myBatis, what broke actually was Hibernate in the newer versions of Spring. Oh, and Flyway for DB migrations simply used to work with a particular Oracle DB version as well.

krzyk 5y ago

JDK 8 -> 9 migration was painful, but all other upgrades should be painless (well, maybe the 16-17, but I assume that most libraries will be fixed by the time you upgrade - either add `add-opens` instructions or use appropriate new classes from JDK).

__jem 5y ago

I got stuck in this conundrum at my previous gig as well, and think reached a very different conclusion than you. In short, while I’m skeptical of some of the degree of churn across modern software development as a whole, I’m not sure that this is actually an issue with the Java ecosystem, which has been, and I think continues to be, way more sensitive to backwards compatibility than almost any other ecosystem. In some ways, this is just a consequence of a kind of evolutionary leap forward for many applications. We don’t deploy our WARs into application servers anymore, we try to pack a fat JAR, or maybe even a statically linked runtime, into a container. We rebuild the whole world on CI all the time. I mean, Java 8 has issues just respecting cgroup limits, which many a despondent ops engineer has discovered. I think there’s still room for the 1M SLoC monolith even in this new world, but there are real benefits on the horizon for upgrading to Java 17 and beyond. We’ve reached a point where it’s just more expensive not to make upgrades a regular part of you development cycle. And, no one is stopping you from just staying on 1.8. I mean, I’m sure there are still decaying enterprises stranded on 1.3.

KronisLV 5y ago
Personally, i'd never let a single monolith grow that far. Having a problem component or two that could be left on JDK 8 until they die and are rewritten (which would actually be possible with systems that have clearly defined boundaries and smaller scope), while migrating everything else. Sadly, i don't get that choice, nor do i get the choice to make the judgement call to leave the monolith on JDK 8 in the name of stability. But hey, at least i'm paid a bit of money for it, so i have some motivation to work on all of it to the best of my abilities, learn a bit more about the JDK internals, have a word or two to tell others about my experience before i inevitably burn out from the churn. Personally, i do still think that Java is pretty good when it comes to stability and backwards and forwards compatibility, especially since tooling like Maven is actually pretty good when compared to the alternatives (well, the parent POM functionality is confusing sometimes, but oh well, it has its use cases). That said, JDK 8 to newer releases was indeed a generational shift (probably one for the better) and you see similar things with Spring Boot 1.5 to Spring Boot 2.X, those breaking releases are inevitable. I only wish that the things blocking people from writing more modular systems would disappear over time, so that huge monoliths that are incredibly hard to work with wouldn't be such an issue. I'm not necessarily advocating for microservices here, since people tend to go straight from one ditch into the opposite one (multiple services per person, nightmarish service mesh, needlessly large % of the code being for shuffling data around, suddenly building a distributed system even within a single domain), but at the very least it would be really nice to have someone look at it and say: "Hey, this PDF generation logic looks really brittle and perhaps should be a service of its own." or maybe: "Oh, hey, we're serving our RESTful API calls and front end resources from the same application, maybe we should have a separate front end app, served through a regular web server?" Then again, if nothing else, i always have the choice to choose where i'm employed, even though i don't feel like letting my coworkers down at the moment either.

swsieber 5y ago

I hope you move to spring voot first vefore upgrading to JDK11, or vice versa. I'm in the middle of a migration to 11 and my general approach has been to get it compiling on 11 while targeting 8 with (--release 8). That seems to catch a great many things that would otherwise fail. The hardest part thus far was figuring out the appropriate libs to replace Java 8 ee stuff, mostly wrt soap and xml. I'm hoping moving to 11 goes smoothly but we'll see. I'm hoping to have the artifacts to a point where they can run on either 8 or 11 without issues.

stickfigure 5y ago

It sounds like the problem is the scope creep. Do the JDK upgrade or the spring boot migration first. Yes it is tempting to do both at once but you are adding a lot of risk.

KronisLV 5y ago
Agreed! Sadly, it wasn't my choice to make. Then again, if it's an older version of Spring, chances are that a lot of changes would be needed regardless.

AmpsterMan 5y ago

We had that about a year ago in our product as well, though from Java 7 to Java 11. That JDK 9+ hump was definitely bad, but since then each incremental JDK has been smooth. The idea of using "the latest JDK" is fairly modern in the Java world, so people are still getting used to it. I'd say it's good now though, with perhaps 16 -> 17 being a minor hump since things that were deprecated but allowed in 9+ are now removed.

zz865 5y ago

Just do the 8-11 upgrade first. The XML part has a lot of life left in it, still lots of advantages over attributes.

achoice 5y ago

Upgrading Spring and especially major Boot versions is usually much more work than bumping Java version. Boot change how it does config and such which .. can be confusing.

grumpyprole 5y ago

Isn't your problem above essentially with Spring rather than the new JDK? Presumably Spring is using internal APIs that have changed in JDK11. And yes, the reflection and dynamic behaviour in Spring are not type safe (i.e. runtime failures). Spring XML is a manifestation of "Greenspun's tenth rule" - it gives people the flexibility of Lisp/Python by unsurprisingly removing static type safety - not a trade-off that many will agree with. You should tell management that your job had been made harder by the choices made by the original architects of the system (who I assume are long gone),

nomercy400 5y ago

Aa somebody who has had to do a Spring upgrade in the past, from 2.x to 4.x, I suggest you do Spring first, then the JDK after. Spring still supports 8 well after it starts supporting 11. Once you reach a certain version of Spring, upgrading the JDK is as simple as changing a 1.8 into an 11. I don't know PrimeFaces, but I have worked with JSP, which does have a flag to set the compiler version (in case you were ever thinking about putting java lambda's in JSP files, which is possible but not recommended due to mixing code). The hardest part of upgrading the JDK is going from 8 to 9. After 9 it is much more forgivable. As for Spring, major versions are a pain, as is Hibernate. The other parts of Spring are generally well upgradable if you read the Spring changelogs.