5 ms·

Faster Maven Builds

74 points by based2 5y ago

willvarfar 5y ago
For my projects, building the jar-with-dependencies takes forever. And as there is a zip file from the previous build just sitting there, it’s a real shame the system isn’t clever enough to examine and reuse vast swathes of it.
- lmm 5y ago
  A jar-with-dependencies is not really working with the grain of maven - you're meant to use the "exploded" version and use maven to run with the right classpath. Caching and reusing build outputs sounds clever, but my experience with Gradle is that the cache wastes far more time than it ever saves, either by taking longer to search the cache than to rebuild the thing, or by causing nondeterministic misbuilds that take days to debug.
- emmelaich 5y ago
  Can't you just 'install' it then make it a dependency itself?
mey 5y ago
Laughs in horrible Gradle Kotlin DSL build times.
- lmm 5y ago
  Yeah, don't do that.
- kaba0 5y ago
  Gradle is one of the fastest build systems out there (certainly the fastest for Java projects because the tight integration with the compiler), given people know what they are doing. Most often than not they will write build logic as configuration which runs each time. But when configured properly, it actually has parallel builds, with fine-grained build dependencies, so only the actually changed classes will be recompiled.
- pjmlp 5y ago
  If it wasn't for Google's sponsorship it would have been gone by now. Android is probably the only platform whose conferences to this day have a sessions about improving build times, thanks Gradle. When doing Windows CE, Pocket PC, Windows Phone 7, WinRT, Symbian, iOS, I never needed a gaming rig as development workstation.
  - kaba0 5y ago
    It is android’s fault, not gradle’s.
    - pjmlp 5y ago
      When Gradle requires a background daemon consuming at least 2GB and SSD disks to achieve usuable builds, that lies on Gradle not Android.
      - kaba0 5y ago
        Why exactly? It stores caches, that gets used for the build process. How else should it operate? The problem is the amount of dependencies/build tasks, not the tool managing those.
        pjmlp 5y ago
        The excuses with Gradle are always the kind of "you're holding it wrong". Thankfully outside Android, I never have to put up with it on regular Java projects.
        kaba0 5y ago
        Which would show you how is it a very fast build tool, but if you prefer not even learning about it, do stay in your ignorance.
        deleted 5y ago
        [deleted]
vbezhenar 5y ago
Java build systems are too archaic and complex at this moment. They should be purged and replaced with new simple tiny tool, compiled with GraalVM. Java could be compiled extremely fast, javac does not do any fancy optimizations, it just spews bytecode directly derived from AST. 99% of maven/gradle run is mindless abstraction overhead.
- bitcharmer 5y ago
  Could you provide examples of better solutions for other platforms? I've been doing software dev professionally in 5+ languages for the past 20 years and speaking from experience java has one of the best build/dep management systems to choose from.
- tehbeard 5y ago
  Don't. Just don't. It's already a clusterfuck in JS land with grunt/gulp/webpack/TSC/rollup/parcel/snowpack/esbuild/rome etc, do not bring that madness to java as well.
  - chii 5y ago
    that java madness has already happened...10 yrs ago! maven is at the level of maturity where you don't need to even refer to it by it's version - it's just maven now (there's been two previous incarnations of maven, called maven 1 and maven 2). Then there's ANT, and there's a bunch of bespoke tools as well that i'm not too familiar with - ANT is still useful, but i would say that it plugins into the maven toolchain acceptably OK.
    - frant-hartm 5y ago
      Yeah we had this stable period, but now get ready for Maven 4. There will be breaking changes.
- raducu 5y ago
  Are there other build tools for other languages that are much faster for the same codebase/complexity?
jillesvangurp 5y ago
Parallelizing builds and tests is definitely useful. Not only does it make things run faster but it can force concurrency issues to the surface that you would otherwise not find until after you deploy the code. This works especially well with integration tests that each might take a few seconds to run. Most of that is IO so you can actually utilize more threads than CPU cores here and still expect a speedup. However, it requires designing your tests such you can do this. One key trick is using randomized data and test fixtures. Another one is avoiding the overhead of initializing databases, etc. over and over again. Of course, all that stuff is useful in just about any language. But some languages make this easier than others.
- chenxiaolong 5y ago
  100% agreed re. bringing concurrency issues to the surface. Another nasty one I encountered a couple years ago: After switching the build servers to using ramdisks, the app started breaking in subtle ways. Turns out there were two conflicting dependencies, but building on a server using ext4 and with the specific number of concurrent threads caused the "right" one to appear first the .jar's central directory. Gotta add filesystem directory entry ordering to the list too.
the_alchemist 5y ago
Nice write-up, and good to see that maven also evolves. The use of the daemon is really significant for the build time. Is there any clue what would take 90 seconds everytime you execute the maven process? Is it the creating of the build graph and its dependencies? What would happen when I change my dependencies or module structure, does the daemon still stay on 30 seconds?
shrx 5y ago
It would be great if the mvnd authors provided some instructions about how to set up mvnd to be used in an IDE like Eclipse or IntelliJ on the github repo [0] [0] https://github.com/mvndaemon/mvnd https://github.com/mvndaemon/mvnd
dikei 5y ago
Imho, mvnd is a step in the right direction. Next is how to make maven cache work more reliably and aggressively, as maven is still too slow at incremental build: A zero-change build can take seconds, while Bazel finishes almost instantly.
- brown9-2 5y ago
  I think it’s more of the fault of needing to invoke a number of plugins bound to different cycles, rather than the compiler deciding if it needs to recompile files or not. I don’t believe Maven’s domain model has any concept of incrementalism or caching - only the compiler does.
skinkestek 5y ago
Step 1: if you use Windows, just use anything else. Even a Linux VM running on top of the same Windows will probably be faster. (I have not tested exactly that but I have tested in dual boot configuration and I have tested .Net cli apps and seen them running faster under a Linux VM on top of Windows than it did natively on Windows.)
tomohawk 5y ago
We switched to buildr. Build scripts massively simplified and build times drastically reduced.

74 points by based2 5y ago

For my projects, building the jar-with-dependencies takes forever. And as there is a zip file from the previous build just sitting there, it’s a real shame the system isn’t clever enough to examine and reuse vast swathes of it.

lmm 5y ago
A jar-with-dependencies is not really working with the grain of maven - you're meant to use the "exploded" version and use maven to run with the right classpath. Caching and reusing build outputs sounds clever, but my experience with Gradle is that the cache wastes far more time than it ever saves, either by taking longer to search the cache than to rebuild the thing, or by causing nondeterministic misbuilds that take days to debug.
emmelaich 5y ago
Can't you just 'install' it then make it a dependency itself?

mey 5y ago

Laughs in horrible Gradle Kotlin DSL build times.

lmm 5y ago
Yeah, don't do that.
kaba0 5y ago
Gradle is one of the fastest build systems out there (certainly the fastest for Java projects because the tight integration with the compiler), given people know what they are doing. Most often than not they will write build logic as configuration which runs each time. But when configured properly, it actually has parallel builds, with fine-grained build dependencies, so only the actually changed classes will be recompiled.
pjmlp 5y ago
If it wasn't for Google's sponsorship it would have been gone by now. Android is probably the only platform whose conferences to this day have a sessions about improving build times, thanks Gradle. When doing Windows CE, Pocket PC, Windows Phone 7, WinRT, Symbian, iOS, I never needed a gaming rig as development workstation.
- kaba0 5y ago
  It is android’s fault, not gradle’s.
  - pjmlp 5y ago
    When Gradle requires a background daemon consuming at least 2GB and SSD disks to achieve usuable builds, that lies on Gradle not Android.
    - kaba0 5y ago
      Why exactly? It stores caches, that gets used for the build process. How else should it operate? The problem is the amount of dependencies/build tasks, not the tool managing those.
      - pjmlp 5y ago
        The excuses with Gradle are always the kind of "you're holding it wrong". Thankfully outside Android, I never have to put up with it on regular Java projects.
        kaba0 5y ago
        Which would show you how is it a very fast build tool, but if you prefer not even learning about it, do stay in your ignorance.
        deleted 5y ago
        [deleted]

vbezhenar 5y ago

Java build systems are too archaic and complex at this moment. They should be purged and replaced with new simple tiny tool, compiled with GraalVM. Java could be compiled extremely fast, javac does not do any fancy optimizations, it just spews bytecode directly derived from AST. 99% of maven/gradle run is mindless abstraction overhead.

bitcharmer 5y ago
Could you provide examples of better solutions for other platforms? I've been doing software dev professionally in 5+ languages for the past 20 years and speaking from experience java has one of the best build/dep management systems to choose from.
tehbeard 5y ago
Don't. Just don't. It's already a clusterfuck in JS land with grunt/gulp/webpack/TSC/rollup/parcel/snowpack/esbuild/rome etc, do not bring that madness to java as well.
- chii 5y ago
  that java madness has already happened...10 yrs ago! maven is at the level of maturity where you don't need to even refer to it by it's version - it's just maven now (there's been two previous incarnations of maven, called maven 1 and maven 2). Then there's ANT, and there's a bunch of bespoke tools as well that i'm not too familiar with - ANT is still useful, but i would say that it plugins into the maven toolchain acceptably OK.
  - frant-hartm 5y ago
    Yeah we had this stable period, but now get ready for Maven 4. There will be breaking changes.
raducu 5y ago
Are there other build tools for other languages that are much faster for the same codebase/complexity?

jillesvangurp 5y ago

Parallelizing builds and tests is definitely useful. Not only does it make things run faster but it can force concurrency issues to the surface that you would otherwise not find until after you deploy the code. This works especially well with integration tests that each might take a few seconds to run. Most of that is IO so you can actually utilize more threads than CPU cores here and still expect a speedup. However, it requires designing your tests such you can do this. One key trick is using randomized data and test fixtures. Another one is avoiding the overhead of initializing databases, etc. over and over again. Of course, all that stuff is useful in just about any language. But some languages make this easier than others.

chenxiaolong 5y ago
100% agreed re. bringing concurrency issues to the surface. Another nasty one I encountered a couple years ago: After switching the build servers to using ramdisks, the app started breaking in subtle ways. Turns out there were two conflicting dependencies, but building on a server using ext4 and with the specific number of concurrent threads caused the "right" one to appear first the .jar's central directory. Gotta add filesystem directory entry ordering to the list too.

the_alchemist 5y ago

Nice write-up, and good to see that maven also evolves. The use of the daemon is really significant for the build time. Is there any clue what would take 90 seconds everytime you execute the maven process? Is it the creating of the build graph and its dependencies? What would happen when I change my dependencies or module structure, does the daemon still stay on 30 seconds?

shrx 5y ago

It would be great if the mvnd authors provided some instructions about how to set up mvnd to be used in an IDE like Eclipse or IntelliJ on the github repo [0] [0] https://github.com/mvndaemon/mvnd https://github.com/mvndaemon/mvnd

dikei 5y ago

Imho, mvnd is a step in the right direction. Next is how to make maven cache work more reliably and aggressively, as maven is still too slow at incremental build: A zero-change build can take seconds, while Bazel finishes almost instantly.

brown9-2 5y ago
I think it’s more of the fault of needing to invoke a number of plugins bound to different cycles, rather than the compiler deciding if it needs to recompile files or not. I don’t believe Maven’s domain model has any concept of incrementalism or caching - only the compiler does.

skinkestek 5y ago

Step 1: if you use Windows, just use anything else. Even a Linux VM running on top of the same Windows will probably be faster. (I have not tested exactly that but I have tested in dual boot configuration and I have tested .Net cli apps and seen them running faster under a Linux VM on top of Windows than it did natively on Windows.)

tomohawk 5y ago

We switched to buildr. Build scripts massively simplified and build times drastically reduced.