8 ms·

What tends to be complex about build requirements that necessitates special purpose tools? Golang seems to be doing fine with just go build and go test. What el

What tends to be complex about build requirements that necessitates special purpose tools? Golang seems to be doing fine with just go build and go test. What else are people doing with gradle/maven that requires static typing, DAGs, plugins etc.?

iamcalledrob 2y ago
I'm working on a project that encompasses both JVM (Gradle, Kotlin) and Golang. My hot take: JVM build tools, especially Gradle, are a soup of unnecessary complexity, and people working in that ecosystem have Stockholm Syndrome. In Golang, I spend about 99% of my time dealing with code. In JVM land, I'm spending 30% just dealing with the build system. It's actually insane, and the community at large thinks this is normal. The amount of time it takes to publish a multi-platform Kotlin library for the first time can be measured in days. I published my first Golang library in minutes, by comparison.
- threeseed 2y ago
  > the community at large thinks this is normal Half are ignorant. Other half are like me and just stuck with no options. But the tooling ecosystem on the JVM truly is horrific.
  - iamcalledrob 2y ago
    I think there are a lot of "JVM Lifers" who are so deep in the ecosystem they are unaware how much better things can be. Anecdote: I wanted to publish a ~100LoC multiplatform Kotlin library -- just some bindings. I publish these sorts of things for Go with just a "git push". Steps were: 1. Spend a few hours trying to understand Maven Central/Sonotype, register and get "verified". They're in the middle of some kind of transition so everything is deprecated or unstable. 2. Figure out signing, because of course published packages must be signed. Now I have a secret to keep track of too, great. 3. Discover that there is no stable Gradle plugin for publishing to the "new" Maven Central, it's coming soon... Choose one of the handful of community plugins with a handful of stars on GitHub. 4. Spend a few hours troubleshooting a "Gradle build failed to end" error, which ended up being due to signing not finding a signing key. 3rd party plugin didn't handle errors properly, and a bug in Gradle meant that my secret wasn't picked up from local.properties. 4. Eventually discover that because Kotlin Multiplatform can't be cross-compiled, there is no way to actually publish a multiplatform library without spinning up a bunch of CI runners. And you can't just publish code -- JVM packages have to contain compiled artifacts. 5. Realise this now involves maintaining GitHub Actions and Gradle, which is an ongoing cost. 6. Give up. The harm that this kind of complexity must be causing to the ecosystem is immeasurable.
    - iainmerrick 2y ago
      I’d just like to add, NPM gets a lot of flak (mostly deservedly) but it too is still vastly easier than anything in the JVM ecosystem. Even with all the headaches around modules versus CJS, and JS versus TypeScript, NPM is a lot easier than Gradle. Notably, you have a choice of alternate tools (eg pnpm, yarn, bun) that interoperate pretty well. I guess my point is, Gradle and Maven are specifically and outstandingly bad.
      - lihaoyi 2y ago
        If you think gradle and maven are bad, you should try Mill! There is more to build tooling than gradle or maven, the field has evolved significantly since those tools launched 15-20 years ago, and Mill tries to do things better
      - ackfoobar 2y ago
        I must be missing something here. Don't the tools you mentioned do a lot less than Gradle? Gradle knows test depends on compile, which depends on code generation (say protobuf) - with caching and change detection. Compare that to chaining up the commands in the `scripts` section of `package.json`. EDIT: another comment making this point: https://news.ycombinator.com/item?id=41969847 https://news.ycombinator.com/item?id=41969847
        iainmerrick 2y ago
        I could be convinced if those features of Gradle actually worked well, or even worked properly, like dependency management does in e.g. Bazel. In practice, Gradle really seems to fall down on the basic task of just being able to build stuff in the first place. It feels like you’re constantly fighting version hell just to find a Gradle version and plugins that work together, let alone your actual code dependencies. And if you actually do need to do something slightly more complicated, like code generation, it’s very difficult to work with and the docs are really bad.
        ackfoobar 2y ago
- user1241320 2y ago
  One note from having worked with both that I don’t see mentioned: Golang dependencies are sources you basically pull and compile with your own code. In JVM-land dependencies are precompliled packages (jars). This adds one little step.
  - iamcalledrob 2y ago
    ...or a big step, if cross-compiling is required (e.g. Kotlin Multiplatform) I'm surprised there is no source-only dependency solution for JVM -- it'd solve this issue. Pull down the source and build on the fly. Perhaps there is and I'm unaware?
    - jitl 2y ago
      Well since the builds tend to be monstrously complicated for some reason, and there’s no standard build tool, maybe it’s more impossible than possible to consider source based distribution. Or it would be like JavaScript where you still need a build and publish step to turn “developer Java / other languages” into “vanilla source distributable Java”.
    - pkolaczk 2y ago
      I'm afraid Java/Scala/Kotlin compilers are too slow to make that convenient. Even currently building pure Java projects can take minutes when it's compiling just like 300k lines. What if it had to compile millions of lines from all the dependencies?
      - kaba0 2y ago
        The actual compilation step is 100% not the bottleneck - it can go as fast as 10k-50k lines per second! (According to the Mill benchmark, but that’s the Mill-independent part). Comparatively, Go does “only” 16k lines per second based on some HN comments.
        pkolaczk 2y ago
        But you’re likely comparing on different hardware though. Go compiling only 16k lines per second is hard to believe for me. Maybe they meant on single CPU core. Rustc compiles over 50k lines per second on my MBP in debug mode and Go must be definitely faster, as everyone knows rust is very slow to compile. But anyway, you may be right. I just ran mvn install for the second time with no source change on my current project. It took 57 seconds.
- cryptos 2y ago
  You speak from my soul! I'm in the Java world for a really long time now and I'm wondering for years why the build tools need to be so complicated an annoying. I know Go, Node.js and bit of Rust and all have more pleasant easier to use build tools! The JVM (or GraalVM) as an ecosystem is just fine and probably one of the best, but build tools might be achille's heel. Maybe it would be a good idea for Oracle to invest into that area ...
  - mike_hearn 2y ago
    My experience of JS projects is that build tools are frequently ad-hoc. That is, there simply isn't a general build tool at all, but just a large pile of scripts calling under-documented libraries. Parallelization, caching and quite often even portability are just missing. To justify this statement consider this blog post I wrote a while ago about porting GitHub Desktop (an Electron app) from its prior build/deployment system to Conveyor [1]. Conveyor is a tool for shipping desktop apps and is implemented as a single-purpose build system. The relevant part is this commit: https://github.com/hydraulic-software/github-desktop/commit/5247d87871cf0ddc1d51e486c77ccc9896831bec https://github.com/hydraulic-software/github-desktop/commit/... The amount of code that can be deleted is huge! Some of it is in-process code that isn't needed with Conveyor (setting up Squirrel etc), but a lot is just shell scripts that happen to be written in JS. Replacing that with a real build system not only simplifies the codebase but means the build steps are fully parallelized, fully incremental, easier to debug, portable (the build can run on any platform), progress is reported in a uniform way and so on. So whilst the JS ecosystem's approach to build tools may be "simple" in some way, in the sense that there's no dominant build tool like Maven or Gradle, that simplicity does cost you in other ways. [1] https://hydraulic.dev/blog/8-packaging-electron-apps.html https://hydraulic.dev/blog/8-packaging-electron-apps.html (Disclosure: Conveyor is a commercial product made by my company)
- dikei 2y ago
  > The amount of time it takes to publish a multi-platform Kotlin library for the first time can be measured in days. I published my first Golang library in minutes, by comparison. It's a bit Apple & Orange comparison: publishing a JVM only Kotlin library is quite easy, it's the multiplatform part that takes time.
  - jitl 2y ago
    Last time I published a JVM library I had to Open A Jira Ticket to request the rights to publish a package on the main package registry. Then I had to verify I owned the DNS name prefix for my package by fiddling the DNS records at my hosting provider. It took days just to get authorized! Not including the time needed to like, figure out how to make JARs happen. In go: `git push` to a public repo In js: `npm publish` after making an NPM account
    - pkolaczk 2y ago
      In rust: `cargo publish` after making an account on crates.io
    - mdaniel 2y ago
      Merely as a "for your consideration," GitLab ships with its own Maven repository (along with npm, docker, Nuget, and a bazillion others)[1] so you have total sovereignty over the publishing auth story. I can appreciate going with Central can be a DX win if you're distributing a library, since having folks add <repository> lines to their pom.xml or settings.xml is a hassle, but at least you get to decide which hassle you prefer :-D In fairness, GitHub also finally got on board the train, too: https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-java-packages-with-maven#publishing-packages-to-github-packages https://docs.github.com/en/actions/use-cases-and-examples/pu... 1: https://docs.gitlab.com/ee/user/packages/maven_repository/ https://docs.gitlab.com/ee/user/packages/maven_repository/
    - vips7L 2y ago
      Sometimes barrier to entry is good. For example, both npm and cargo struggle with package name squatting and malicious packages that are miss spellings of common packages.
- neeleshs 2y ago
  Interesting. I spend nearly zero time with my maven setup and almost all the time is in coding. I am genuinely curious to know where that 30% time goes? Is it waiting for builds?
- philipwhiuk 2y ago
  > I published my first Golang library in minutes, by comparison. For what platform(s)? Or did you really just push the source code?
  - pkolaczk 2y ago
    That's the trick. You publish the source code. And it's still faster to build all dependencies from source than maven / gradle manages to resolve and download the binary dependencies ;)
    - brabel 2y ago
      That's true, Maven is ridiculously slow to resolve dependencies while Gradle only really works with reasonable speed if you allow it to hog your system with a deamon. I myself wrote a dependency resolver that matches Maven in functionality, and even a large project that uses Spring Boot and its dozens of dependencies can be resolved in a couple of seconds. About 10x faster than Maven or something like that. If you look at Maven's source code you'll see why. It's the worst kind of Java Enterprise overengineering you can imagine, complete with its own dependency injection framework, everything is pluggable (for no reason, really, do you really need to replace HTTPS for your protocols?? In Plexus you can), to the point that all the de-coupling results in lots of things duplicating functionality everywhere. I am not sure but I would bet Maven parses your POM at least 10 times to do anything due to the de-coupled nature of it.
      - lihaoyi 2y ago
        Maven is actually pretty behind in terms of JVM dependency resolution. Mill uses Coursier, same as my last company did, and when my last company switched from Maven to Coursier we saw a 2 order of magnitude speedup, with resolution commands that used to take 30min finish in a few seconds to give the exact same artifacts and versions. I actually have no idea why these other resolvers are so slow, or why Coursier is so fast, but this slowness is very much a "maven" or "gradle" thing rather than a "jvm" thing. And Mill using coursier does significantly better!
- pkolaczk 2y ago
  I've been working on Java-based systems for about 20 years now, and I fully relate to that. Same experience. This is so annoying that I prefer to use Rust over Java even in areas where things like better performance or better type system don't matter. But being able to start a fresh project with one `cargo init` and a few `cargo add` invocations to add any dependencies... well, this is priceless.
  - mdaniel 2y ago
    Are you aware of Maven Archetypes[1]? I believe they were the "cookiecutter" before cookiecutter existed, although I am 10000000% on-board that their discovery story is total garbage :-( 1: https://maven.apache.org/archetype/index.html https://maven.apache.org/archetype/index.html and https://maven.apache.org/archetype/maven-archetype-plugin/usage.html https://maven.apache.org/archetype/maven-archetype-plugin/us...
    - pkolaczk 2y ago
      But I don’t want to copy a full project with prepopulated list of dependencies chosen by someone else. I want to start small and add dependencies I need. It’s like LEGO vs Playmobil. I want LEGO. ;)
  - brabel 2y ago
    Interesting that you ended up going all the way to Rust land instead of just using one of the multiple tools that have been created to help with this, like: * Spring Boot (it has a UI to create projects where you pick Java version, DB, build tool, some libs etc): https://spring.io/guides/gs/spring-boot https://spring.io/guides/gs/spring-boot * JHipster - the nuclear option, pick what you want a la carte: https://www.jhipster.tech/ https://www.jhipster.tech/ * JBang - a cute CLI for this: https://www.jbang.dev/ https://www.jbang.dev/ * Maven Archetypes - the old fashioned way (existed before "create-app" kind of tools appeared): https://maven.apache.org/guides/introduction/introduction-to-archetypes.html https://maven.apache.org/guides/introduction/introduction-to... And most IDEs also have "new project" wizzards.
  - rileymichael 2y ago
- imtringued 2y ago
  I'm in JVM land. I spend very little time dealing with the build system. It is actually insane how well it works. Also, why does it matter how long it takes to publish a library for the first time? It sounds like a non-issue to me. I have written dozens of libraries and published them to a local artifactory instance because it simply doesn't matter if your company specific code is accessible to the world or not.
rykad 2y ago
To be blunt, nothing. The issue is that most people in jvm land are on a closed bubble and haven't seen anything else. This is true for build systems as is for non OO design for example. Most simply don't know better and the rest of us are simply stuck. Ant and then Maven started simple enough but people always find a way to justify adding more stuff. Gradle already started complex enough and they keep adding more stuff...
- cryptos 2y ago
  You have to keep in mind that Gradle Inc. earns money by providing consulting for complex builds. An easy build tool would destroy this business model ;-)
  - mrudolph22 2y ago
    They transitioned to a product company 5+ years ago. https://gradle.com https://gradle.com Gradle's complexity comes from at least two places: 1. The original vision of solving complex multi-technology/language Enterprise builds. 2. Poor early design decisions that they never recovered from.
- kaba0 2y ago
  To be blunt, if anything it might be you who live in a closed bubble. Builds that require ad-hoc functionality is the default. It’s extremely rare that everything fits nicely into “cargo build” or other single language build tools’ model. And while these often have escape hatches, at that point you have to write imperative code with no caching and parallelization that is literally the job of a build tool.
  - rykad 2y ago
    I've been doing jvm apps for almost 20yrs... What builds need to do and what people made the builds do are completely different things. I don't remember a single project I was involved in that could not had had a simpler build...
    - 2y ago
dikei 2y ago
In JVM world, the de factor equivalent to `go build` and `go test` are `mvn compile` and `mvn test`, which works 99% percent of the time. Other build tools and plugins just compete/fill in for: * improved build speed / test speed: using background daemon to reduce strtup speed, intelligent caching / task reordering to avoid redoing, etc.. * extra functionalities like code generation, publishing or deployments. As code generation is really big in JVM world, and there are many ways to deploy an application: jar + libs in a zip file, uber jars, container image, etc...
- dcminter 2y ago
  Curmudgeon here: this was true for a relatively brief period of time. Nowadays I'd say that gradle has (inexplicably to me) taken the lead - and everyone adds custom crap to their gradle build making them far less predictable than maven builds used to be. I guess it's better than the nightmare over in the front-enders' world...
  - switchbak 2y ago
    I rarely see Maven files in non-trivial projects that are anything but a confusing mess of XML. Granted, the constrained abilities do tend to keep folks from writing one-off snowflake build customizations, which is nice. But it still leave a hell of a lot to be desired. It was however, leagues ahead of Ant, which wasn't a high bar.
    - dcminter 2y ago
      I saw a lot of that too - I just see more and worse with gradle. Maven gave us two things; good dependency management and convention driven builds (removing the horrible scripted build stuff in Ant). Gradle from my point of view took the second one away again and it feels like it was just because people didn't like XML and couldn't be bothered to learn how Maven's build lifecycle actually worked! Like I say, I'm a curmudgeon...
- pkolaczk 2y ago
  My typical mvn session after a month of not touching maven: % mvn [ERROR] No goals have been specified for this build. Uh. % mvn build [ERROR] Unknown lifecycle phase "build". Uh, nope, that wasn't it.... % mvn compile BUILD SUCCESS Now trying to run the app... Error: app jar not found. Reading the README.md. Aha! So I need to install it! % mvn install 16:59:35,075 [INFO] Building <blahblahblah> [1/32] ... 16:59:38,483 [INFO] ------------------------------------------------------- 16:59:38,483 [INFO] T E S T S 16:59:38,483 [INFO] ------------------------------------------------------- ^C // me searching on google how to not run tests % mvn install -DskipTests=true // now good... Ok, let's run some tests. The FlakyTest broke again in CI. Let me run it locally: % mvn test FlakyTest 17:02:19,481 [ERROR] Unknown lifecycle phase "FlakyTest". Aaargh, ok, googling it again: % mvn test -Dtest=FlakyTest 17:03:16,102 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.3.0:test (default-test) on project blah blah: No tests matching pattern "FlakyTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.) The original idea behind maven is nice. But the defaults are so bad, that the whole experience of convention over configuration has been ruined. Cargo and go build systems took the original maven philosophy and implemented them right, with good UX. Gradle, SBT and friends took a step back to the times before maven, and went fully the Ant way, doubling down on "configuration over convention". Where "configuration" is actually "programming in a DSL on top of another language on top of Java".
mike_hearn 2y ago
Build tools sit in an unhappy corner of the design space where they provide features not found in the core of regular programming languages, and which are so generally useful that there's a temptation to make them very abstract, but then they often lack some of the features that let regular programs scale well. The key feature that justifies their existence is parallel and incremental execution of DAGs of world-mutating tasks. This is an awkward fit with most programming languages, hence the prevalence of DSLs. But people don't want their build system to become a general purpose programming language, because they don't want to think about build systems at all and because programmers don't buy programming languages anymore, so, this causes a big design tension between generality (people want to use build systems to automate many things) and deliberately limiting the expressive power to try and constrain the design space and thus tooling investment required. Java is in an awkward place because the JDK was born in the 90s on UNIX, by people who thought make is a sufficiently good solution. You still see remnants of this belief in the official Java tutorials, in JEPs, and in the fact that OpenJDK itself is compiled using an autotools based build system! (fortunately it's one of the nice make based build systems out there). The problem with make is twofold: 1. It assumes a CLI that's both powerful and standardized provided by the host OS. Windows violates this assumption but Java is meant to be portable to Windows. 2. "Plugins" are CLI tools or scripts and so make implicitly assumes that subprocess creation is cheap. But process creation on Windows is expensive, and starting up JVM programs is also expensive due to the JIT compiling. Therefore make just doesn't work well in the JVM ecosystem. At the same time, the Java project wasn't providing any competing solution, so the wider open source community was left to fill in the gaps. These days language developers provide build tooling out of the box as part of the base toolset along with the compiler, but Java still doesn't. So - you ask, what are people doing with Gradle/Maven that requires all those features. The answer is: everything! Gradle builds frequently orchestrate dozens of different tools as part of a build pipeline, build documentation websites, do upload and deployment, download and manage dependencies, run security scanners and license compliance checkers, analyze dependency graphs, modify compiler behaviors, and so on. Additionally Gradle isn't specific to Java, or even JVM apps. It can also be used to compile C/C++ programs, run native code compilers like Kotlin/Native, and it abstracts the underlying platform so Gradle builds aren't tied to UNIX. That's why it's so complicated.
- 2y ago
lihaoyi 2y ago
Author here! `go build` and `go test` do work, at limited scale and complexity. In Scala there's Scala-CLI which is excellent. If they work for you, you probably aren't the target market for these build tools. Once you start layering on bash scripts, layering on make, layering on Python scripts, layering on manual steps written down in a readme.md somewhere, that's the time when you should consider a proper build tool. And if that has never happened to you in your career, count your blessings :) Why not just write some boring, pure code to handle the build? Why not write my build system in vanilla LYAH Haskell? It turns out that builds do have some specific requirements that most programs do not need to care about: caching, parallelization, introspection, and so on. Check out the following blog/talk for more details: * Blog Post: Build Tools as Pure Functional Programs https://www.lihaoyi.com/post/BuildToolsasPureFunctionalPrograms.html https://www.lihaoyi.com/post/BuildToolsasPureFunctionalProgr... * Video: Mill: a Build Tool based on Pure Functional Programming https://www.youtube.com/watch?v=j6uThGxx-18&list=PLBqWQH1MiwBSThMSFV4k4dGTZkS0mfbxo&index=16&t=27s https://www.youtube.com/watch?v=j6uThGxx-18&list=PLBqWQH1Miw... Thus "naively" building your project "directly with code" ends up not working, so you do need some additional support. While most build tools end up constructing a complete bespoke programming environment from scratch, Mill tries to leverage the Scala language and JVM as much as possible, so you can re-use all your expertise and tooling (e.g. IntelliJ, VSCode, Maven Central, etc.) almost verbatim while getting all the necessary build-tool stuff (parallelism, caching, introspection) for free. Check out those two links if you want to learn more!
- potamic 2y ago
  Thanks, the post actually answers my question on build requirements. It's a very good write up overall! If your USP is to solve those layering cases, then why not target other ecosystems like golang/rust as well? Your design philosophy certainly seems to be language agnostic. By calling yourself a build tool for Java and Scala, it gives an impression that this is solving problems specific to those environments, and your adoption also indicates as such. Is it that these communities do not like to adopt such tools or is there something about the JVM ecosystem that tends towards having complex build requirements?
  - 2y ago
kaba0 2y ago
There are quite a few cases: the moment you touch another language, resources that require a compile-step (e.g. xml schemas to code dtos, protonbuf, all that kind of stuff), sometimes even the code itself requires generation.
zem 2y ago
I recommend the "build systems a la carte" paper for a good overview of the various problems build systems address
cyberax 2y ago
Try to build a Go project that uses Cgo and non-trivial C/C++ libraries. Throw in cross-compilation for more fun. You'll end up with an external build system that invokes Go build as one of the steps. Go projects normally just tend to be self-contained server-like software that doesn't need a lot of external libraries. But once you step away from that, you're on your own. I guess my problem with Gradle is that app building should be way simpler than it is. Apps are not something niche anymore, but the tooling is still similar to the embedded software for microcontrollers.

iamcalledrob 2y ago

I'm working on a project that encompasses both JVM (Gradle, Kotlin) and Golang. My hot take: JVM build tools, especially Gradle, are a soup of unnecessary complexity, and people working in that ecosystem have Stockholm Syndrome. In Golang, I spend about 99% of my time dealing with code. In JVM land, I'm spending 30% just dealing with the build system. It's actually insane, and the community at large thinks this is normal. The amount of time it takes to publish a multi-platform Kotlin library for the first time can be measured in days. I published my first Golang library in minutes, by comparison.

threeseed 2y ago
> the community at large thinks this is normal Half are ignorant. Other half are like me and just stuck with no options. But the tooling ecosystem on the JVM truly is horrific.
- iamcalledrob 2y ago
  I think there are a lot of "JVM Lifers" who are so deep in the ecosystem they are unaware how much better things can be. Anecdote: I wanted to publish a ~100LoC multiplatform Kotlin library -- just some bindings. I publish these sorts of things for Go with just a "git push". Steps were: 1. Spend a few hours trying to understand Maven Central/Sonotype, register and get "verified". They're in the middle of some kind of transition so everything is deprecated or unstable. 2. Figure out signing, because of course published packages must be signed. Now I have a secret to keep track of too, great. 3. Discover that there is no stable Gradle plugin for publishing to the "new" Maven Central, it's coming soon... Choose one of the handful of community plugins with a handful of stars on GitHub. 4. Spend a few hours troubleshooting a "Gradle build failed to end" error, which ended up being due to signing not finding a signing key. 3rd party plugin didn't handle errors properly, and a bug in Gradle meant that my secret wasn't picked up from local.properties. 4. Eventually discover that because Kotlin Multiplatform can't be cross-compiled, there is no way to actually publish a multiplatform library without spinning up a bunch of CI runners. And you can't just publish code -- JVM packages have to contain compiled artifacts. 5. Realise this now involves maintaining GitHub Actions and Gradle, which is an ongoing cost. 6. Give up. The harm that this kind of complexity must be causing to the ecosystem is immeasurable.
  - iainmerrick 2y ago
    I’d just like to add, NPM gets a lot of flak (mostly deservedly) but it too is still vastly easier than anything in the JVM ecosystem. Even with all the headaches around modules versus CJS, and JS versus TypeScript, NPM is a lot easier than Gradle. Notably, you have a choice of alternate tools (eg pnpm, yarn, bun) that interoperate pretty well. I guess my point is, Gradle and Maven are specifically and outstandingly bad.
    - lihaoyi 2y ago
      If you think gradle and maven are bad, you should try Mill! There is more to build tooling than gradle or maven, the field has evolved significantly since those tools launched 15-20 years ago, and Mill tries to do things better
    - ackfoobar 2y ago
      I must be missing something here. Don't the tools you mentioned do a lot less than Gradle? Gradle knows test depends on compile, which depends on code generation (say protobuf) - with caching and change detection. Compare that to chaining up the commands in the `scripts` section of `package.json`. EDIT: another comment making this point: https://news.ycombinator.com/item?id=41969847 https://news.ycombinator.com/item?id=41969847
      - iainmerrick 2y ago
        I could be convinced if those features of Gradle actually worked well, or even worked properly, like dependency management does in e.g. Bazel. In practice, Gradle really seems to fall down on the basic task of just being able to build stuff in the first place. It feels like you’re constantly fighting version hell just to find a Gradle version and plugins that work together, let alone your actual code dependencies. And if you actually do need to do something slightly more complicated, like code generation, it’s very difficult to work with and the docs are really bad.
        ackfoobar 2y ago
user1241320 2y ago
One note from having worked with both that I don’t see mentioned: Golang dependencies are sources you basically pull and compile with your own code. In JVM-land dependencies are precompliled packages (jars). This adds one little step.
- iamcalledrob 2y ago
  ...or a big step, if cross-compiling is required (e.g. Kotlin Multiplatform) I'm surprised there is no source-only dependency solution for JVM -- it'd solve this issue. Pull down the source and build on the fly. Perhaps there is and I'm unaware?
  - jitl 2y ago
    Well since the builds tend to be monstrously complicated for some reason, and there’s no standard build tool, maybe it’s more impossible than possible to consider source based distribution. Or it would be like JavaScript where you still need a build and publish step to turn “developer Java / other languages” into “vanilla source distributable Java”.
  - pkolaczk 2y ago
    I'm afraid Java/Scala/Kotlin compilers are too slow to make that convenient. Even currently building pure Java projects can take minutes when it's compiling just like 300k lines. What if it had to compile millions of lines from all the dependencies?
    - kaba0 2y ago
      The actual compilation step is 100% not the bottleneck - it can go as fast as 10k-50k lines per second! (According to the Mill benchmark, but that’s the Mill-independent part). Comparatively, Go does “only” 16k lines per second based on some HN comments.
      - pkolaczk 2y ago
        But you’re likely comparing on different hardware though. Go compiling only 16k lines per second is hard to believe for me. Maybe they meant on single CPU core. Rustc compiles over 50k lines per second on my MBP in debug mode and Go must be definitely faster, as everyone knows rust is very slow to compile. But anyway, you may be right. I just ran mvn install for the second time with no source change on my current project. It took 57 seconds.
cryptos 2y ago
You speak from my soul! I'm in the Java world for a really long time now and I'm wondering for years why the build tools need to be so complicated an annoying. I know Go, Node.js and bit of Rust and all have more pleasant easier to use build tools! The JVM (or GraalVM) as an ecosystem is just fine and probably one of the best, but build tools might be achille's heel. Maybe it would be a good idea for Oracle to invest into that area ...
- mike_hearn 2y ago
  My experience of JS projects is that build tools are frequently ad-hoc. That is, there simply isn't a general build tool at all, but just a large pile of scripts calling under-documented libraries. Parallelization, caching and quite often even portability are just missing. To justify this statement consider this blog post I wrote a while ago about porting GitHub Desktop (an Electron app) from its prior build/deployment system to Conveyor [1]. Conveyor is a tool for shipping desktop apps and is implemented as a single-purpose build system. The relevant part is this commit: https://github.com/hydraulic-software/github-desktop/commit/5247d87871cf0ddc1d51e486c77ccc9896831bec https://github.com/hydraulic-software/github-desktop/commit/... The amount of code that can be deleted is huge! Some of it is in-process code that isn't needed with Conveyor (setting up Squirrel etc), but a lot is just shell scripts that happen to be written in JS. Replacing that with a real build system not only simplifies the codebase but means the build steps are fully parallelized, fully incremental, easier to debug, portable (the build can run on any platform), progress is reported in a uniform way and so on. So whilst the JS ecosystem's approach to build tools may be "simple" in some way, in the sense that there's no dominant build tool like Maven or Gradle, that simplicity does cost you in other ways. [1] https://hydraulic.dev/blog/8-packaging-electron-apps.html https://hydraulic.dev/blog/8-packaging-electron-apps.html (Disclosure: Conveyor is a commercial product made by my company)
dikei 2y ago
> The amount of time it takes to publish a multi-platform Kotlin library for the first time can be measured in days. I published my first Golang library in minutes, by comparison. It's a bit Apple & Orange comparison: publishing a JVM only Kotlin library is quite easy, it's the multiplatform part that takes time.
- jitl 2y ago
  Last time I published a JVM library I had to Open A Jira Ticket to request the rights to publish a package on the main package registry. Then I had to verify I owned the DNS name prefix for my package by fiddling the DNS records at my hosting provider. It took days just to get authorized! Not including the time needed to like, figure out how to make JARs happen. In go: `git push` to a public repo In js: `npm publish` after making an NPM account
  - pkolaczk 2y ago
    In rust: `cargo publish` after making an account on crates.io
  - mdaniel 2y ago
    Merely as a "for your consideration," GitLab ships with its own Maven repository (along with npm, docker, Nuget, and a bazillion others)[1] so you have total sovereignty over the publishing auth story. I can appreciate going with Central can be a DX win if you're distributing a library, since having folks add <repository> lines to their pom.xml or settings.xml is a hassle, but at least you get to decide which hassle you prefer :-D In fairness, GitHub also finally got on board the train, too: https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-java-packages-with-maven#publishing-packages-to-github-packages https://docs.github.com/en/actions/use-cases-and-examples/pu... 1: https://docs.gitlab.com/ee/user/packages/maven_repository/ https://docs.gitlab.com/ee/user/packages/maven_repository/
  - vips7L 2y ago
    Sometimes barrier to entry is good. For example, both npm and cargo struggle with package name squatting and malicious packages that are miss spellings of common packages.
neeleshs 2y ago
Interesting. I spend nearly zero time with my maven setup and almost all the time is in coding. I am genuinely curious to know where that 30% time goes? Is it waiting for builds?
philipwhiuk 2y ago
> I published my first Golang library in minutes, by comparison. For what platform(s)? Or did you really just push the source code?
- pkolaczk 2y ago
  That's the trick. You publish the source code. And it's still faster to build all dependencies from source than maven / gradle manages to resolve and download the binary dependencies ;)
  - brabel 2y ago
    That's true, Maven is ridiculously slow to resolve dependencies while Gradle only really works with reasonable speed if you allow it to hog your system with a deamon. I myself wrote a dependency resolver that matches Maven in functionality, and even a large project that uses Spring Boot and its dozens of dependencies can be resolved in a couple of seconds. About 10x faster than Maven or something like that. If you look at Maven's source code you'll see why. It's the worst kind of Java Enterprise overengineering you can imagine, complete with its own dependency injection framework, everything is pluggable (for no reason, really, do you really need to replace HTTPS for your protocols?? In Plexus you can), to the point that all the de-coupling results in lots of things duplicating functionality everywhere. I am not sure but I would bet Maven parses your POM at least 10 times to do anything due to the de-coupled nature of it.
    - lihaoyi 2y ago
      Maven is actually pretty behind in terms of JVM dependency resolution. Mill uses Coursier, same as my last company did, and when my last company switched from Maven to Coursier we saw a 2 order of magnitude speedup, with resolution commands that used to take 30min finish in a few seconds to give the exact same artifacts and versions. I actually have no idea why these other resolvers are so slow, or why Coursier is so fast, but this slowness is very much a "maven" or "gradle" thing rather than a "jvm" thing. And Mill using coursier does significantly better!
pkolaczk 2y ago
I've been working on Java-based systems for about 20 years now, and I fully relate to that. Same experience. This is so annoying that I prefer to use Rust over Java even in areas where things like better performance or better type system don't matter. But being able to start a fresh project with one `cargo init` and a few `cargo add` invocations to add any dependencies... well, this is priceless.
- mdaniel 2y ago
  Are you aware of Maven Archetypes[1]? I believe they were the "cookiecutter" before cookiecutter existed, although I am 10000000% on-board that their discovery story is total garbage :-( 1: https://maven.apache.org/archetype/index.html https://maven.apache.org/archetype/index.html and https://maven.apache.org/archetype/maven-archetype-plugin/usage.html https://maven.apache.org/archetype/maven-archetype-plugin/us...
  - pkolaczk 2y ago
    But I don’t want to copy a full project with prepopulated list of dependencies chosen by someone else. I want to start small and add dependencies I need. It’s like LEGO vs Playmobil. I want LEGO. ;)
- brabel 2y ago
  Interesting that you ended up going all the way to Rust land instead of just using one of the multiple tools that have been created to help with this, like: * Spring Boot (it has a UI to create projects where you pick Java version, DB, build tool, some libs etc): https://spring.io/guides/gs/spring-boot https://spring.io/guides/gs/spring-boot * JHipster - the nuclear option, pick what you want a la carte: https://www.jhipster.tech/ https://www.jhipster.tech/ * JBang - a cute CLI for this: https://www.jbang.dev/ https://www.jbang.dev/ * Maven Archetypes - the old fashioned way (existed before "create-app" kind of tools appeared): https://maven.apache.org/guides/introduction/introduction-to-archetypes.html https://maven.apache.org/guides/introduction/introduction-to... And most IDEs also have "new project" wizzards.
- rileymichael 2y ago
imtringued 2y ago
I'm in JVM land. I spend very little time dealing with the build system. It is actually insane how well it works. Also, why does it matter how long it takes to publish a library for the first time? It sounds like a non-issue to me. I have written dozens of libraries and published them to a local artifactory instance because it simply doesn't matter if your company specific code is accessible to the world or not.

rykad 2y ago

To be blunt, nothing. The issue is that most people in jvm land are on a closed bubble and haven't seen anything else. This is true for build systems as is for non OO design for example. Most simply don't know better and the rest of us are simply stuck. Ant and then Maven started simple enough but people always find a way to justify adding more stuff. Gradle already started complex enough and they keep adding more stuff...

cryptos 2y ago
You have to keep in mind that Gradle Inc. earns money by providing consulting for complex builds. An easy build tool would destroy this business model ;-)
- mrudolph22 2y ago
  They transitioned to a product company 5+ years ago. https://gradle.com https://gradle.com Gradle's complexity comes from at least two places: 1. The original vision of solving complex multi-technology/language Enterprise builds. 2. Poor early design decisions that they never recovered from.
kaba0 2y ago
To be blunt, if anything it might be you who live in a closed bubble. Builds that require ad-hoc functionality is the default. It’s extremely rare that everything fits nicely into “cargo build” or other single language build tools’ model. And while these often have escape hatches, at that point you have to write imperative code with no caching and parallelization that is literally the job of a build tool.
- rykad 2y ago
  I've been doing jvm apps for almost 20yrs... What builds need to do and what people made the builds do are completely different things. I don't remember a single project I was involved in that could not had had a simpler build...
  - 2y ago

dikei 2y ago

In JVM world, the de factor equivalent to `go build` and `go test` are `mvn compile` and `mvn test`, which works 99% percent of the time. Other build tools and plugins just compete/fill in for: * improved build speed / test speed: using background daemon to reduce strtup speed, intelligent caching / task reordering to avoid redoing, etc.. * extra functionalities like code generation, publishing or deployments. As code generation is really big in JVM world, and there are many ways to deploy an application: jar + libs in a zip file, uber jars, container image, etc...

dcminter 2y ago
Curmudgeon here: this was true for a relatively brief period of time. Nowadays I'd say that gradle has (inexplicably to me) taken the lead - and everyone adds custom crap to their gradle build making them far less predictable than maven builds used to be. I guess it's better than the nightmare over in the front-enders' world...
- switchbak 2y ago
  I rarely see Maven files in non-trivial projects that are anything but a confusing mess of XML. Granted, the constrained abilities do tend to keep folks from writing one-off snowflake build customizations, which is nice. But it still leave a hell of a lot to be desired. It was however, leagues ahead of Ant, which wasn't a high bar.
  - dcminter 2y ago
    I saw a lot of that too - I just see more and worse with gradle. Maven gave us two things; good dependency management and convention driven builds (removing the horrible scripted build stuff in Ant). Gradle from my point of view took the second one away again and it feels like it was just because people didn't like XML and couldn't be bothered to learn how Maven's build lifecycle actually worked! Like I say, I'm a curmudgeon...
pkolaczk 2y ago
My typical mvn session after a month of not touching maven: % mvn [ERROR] No goals have been specified for this build. Uh. % mvn build [ERROR] Unknown lifecycle phase "build". Uh, nope, that wasn't it.... % mvn compile BUILD SUCCESS Now trying to run the app... Error: app jar not found. Reading the README.md. Aha! So I need to install it! % mvn install 16:59:35,075 [INFO] Building <blahblahblah> [1/32] ... 16:59:38,483 [INFO] ------------------------------------------------------- 16:59:38,483 [INFO] T E S T S 16:59:38,483 [INFO] ------------------------------------------------------- ^C // me searching on google how to not run tests % mvn install -DskipTests=true // now good... Ok, let's run some tests. The FlakyTest broke again in CI. Let me run it locally: % mvn test FlakyTest 17:02:19,481 [ERROR] Unknown lifecycle phase "FlakyTest". Aaargh, ok, googling it again: % mvn test -Dtest=FlakyTest 17:03:16,102 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.3.0:test (default-test) on project blah blah: No tests matching pattern "FlakyTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.) The original idea behind maven is nice. But the defaults are so bad, that the whole experience of convention over configuration has been ruined. Cargo and go build systems took the original maven philosophy and implemented them right, with good UX. Gradle, SBT and friends took a step back to the times before maven, and went fully the Ant way, doubling down on "configuration over convention". Where "configuration" is actually "programming in a DSL on top of another language on top of Java".

mike_hearn 2y ago

Build tools sit in an unhappy corner of the design space where they provide features not found in the core of regular programming languages, and which are so generally useful that there's a temptation to make them very abstract, but then they often lack some of the features that let regular programs scale well. The key feature that justifies their existence is parallel and incremental execution of DAGs of world-mutating tasks. This is an awkward fit with most programming languages, hence the prevalence of DSLs. But people don't want their build system to become a general purpose programming language, because they don't want to think about build systems at all and because programmers don't buy programming languages anymore, so, this causes a big design tension between generality (people want to use build systems to automate many things) and deliberately limiting the expressive power to try and constrain the design space and thus tooling investment required. Java is in an awkward place because the JDK was born in the 90s on UNIX, by people who thought make is a sufficiently good solution. You still see remnants of this belief in the official Java tutorials, in JEPs, and in the fact that OpenJDK itself is compiled using an autotools based build system! (fortunately it's one of the nice make based build systems out there). The problem with make is twofold: 1. It assumes a CLI that's both powerful and standardized provided by the host OS. Windows violates this assumption but Java is meant to be portable to Windows. 2. "Plugins" are CLI tools or scripts and so make implicitly assumes that subprocess creation is cheap. But process creation on Windows is expensive, and starting up JVM programs is also expensive due to the JIT compiling. Therefore make just doesn't work well in the JVM ecosystem. At the same time, the Java project wasn't providing any competing solution, so the wider open source community was left to fill in the gaps. These days language developers provide build tooling out of the box as part of the base toolset along with the compiler, but Java still doesn't. So - you ask, what are people doing with Gradle/Maven that requires all those features. The answer is: everything! Gradle builds frequently orchestrate dozens of different tools as part of a build pipeline, build documentation websites, do upload and deployment, download and manage dependencies, run security scanners and license compliance checkers, analyze dependency graphs, modify compiler behaviors, and so on. Additionally Gradle isn't specific to Java, or even JVM apps. It can also be used to compile C/C++ programs, run native code compilers like Kotlin/Native, and it abstracts the underlying platform so Gradle builds aren't tied to UNIX. That's why it's so complicated.

2y ago

lihaoyi 2y ago

Author here! `go build` and `go test` do work, at limited scale and complexity. In Scala there's Scala-CLI which is excellent. If they work for you, you probably aren't the target market for these build tools. Once you start layering on bash scripts, layering on make, layering on Python scripts, layering on manual steps written down in a readme.md somewhere, that's the time when you should consider a proper build tool. And if that has never happened to you in your career, count your blessings :) Why not just write some boring, pure code to handle the build? Why not write my build system in vanilla LYAH Haskell? It turns out that builds do have some specific requirements that most programs do not need to care about: caching, parallelization, introspection, and so on. Check out the following blog/talk for more details: * Blog Post: Build Tools as Pure Functional Programs https://www.lihaoyi.com/post/BuildToolsasPureFunctionalPrograms.html https://www.lihaoyi.com/post/BuildToolsasPureFunctionalProgr... * Video: Mill: a Build Tool based on Pure Functional Programming https://www.youtube.com/watch?v=j6uThGxx-18&list=PLBqWQH1MiwBSThMSFV4k4dGTZkS0mfbxo&index=16&t=27s https://www.youtube.com/watch?v=j6uThGxx-18&list=PLBqWQH1Miw... Thus "naively" building your project "directly with code" ends up not working, so you do need some additional support. While most build tools end up constructing a complete bespoke programming environment from scratch, Mill tries to leverage the Scala language and JVM as much as possible, so you can re-use all your expertise and tooling (e.g. IntelliJ, VSCode, Maven Central, etc.) almost verbatim while getting all the necessary build-tool stuff (parallelism, caching, introspection) for free. Check out those two links if you want to learn more!

potamic 2y ago
Thanks, the post actually answers my question on build requirements. It's a very good write up overall! If your USP is to solve those layering cases, then why not target other ecosystems like golang/rust as well? Your design philosophy certainly seems to be language agnostic. By calling yourself a build tool for Java and Scala, it gives an impression that this is solving problems specific to those environments, and your adoption also indicates as such. Is it that these communities do not like to adopt such tools or is there something about the JVM ecosystem that tends towards having complex build requirements?
- 2y ago

kaba0 2y ago

There are quite a few cases: the moment you touch another language, resources that require a compile-step (e.g. xml schemas to code dtos, protonbuf, all that kind of stuff), sometimes even the code itself requires generation.

zem 2y ago

I recommend the "build systems a la carte" paper for a good overview of the various problems build systems address

cyberax 2y ago

Try to build a Go project that uses Cgo and non-trivial C/C++ libraries. Throw in cross-compilation for more fun. You'll end up with an external build system that invokes Go build as one of the steps. Go projects normally just tend to be self-contained server-like software that doesn't need a lot of external libraries. But once you step away from that, you're on your own. I guess my problem with Gradle is that app building should be way simpler than it is. Apps are not something niche anymore, but the tooling is still similar to the embedded software for microcontrollers.