6 ms·

Depends – the missing linker for the JVM

124 points by voodootrucker 8y ago

rad_gruchalski 8y ago
Great stuff. Never got to building this myself but this is awesome.
setheron 8y ago
There will always be a case where you can't solve the diamond dependency problem.
- voodootrucker 8y ago
  Two responses to the diamond problem: - I see this often with Guava. A wants 18 and B wants 21, neither works with the other, but they both with with 20. Because this tool looks at the actual methods being invoked, even if 100% of the methods are present, it's fine as long as 100% of the ones being invoked are. - Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list.
  - sleepychu 8y ago
    Isn't choosing to sub in 20 just helping you build more on an unstable foundation? 1 - how can you be sure the method implementation is the same? That there aren't side effects you don't know about. (I don't think you can!) 2 - when it emerges that B2 really does need 25 what are you going to do? IMO it would be better to resolve this earlier than later.
  - Slartie 8y ago
    > Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list. Not only technically, this also works practically. We've done this for years in a POS application in order to be able to load different incompatible versions of vendor-provided hardware drivers at the same time, with the decision on which to use (one, the other, or maybe even both at the same time) being done at runtime. It's based on OSGi for modularization, which also provides other benefits besides classloader isolation, like allowing to bundle multiple native libs for different OSes and bit-nesses within a single jar and automatically choosing the right one to dynamically link into your JVM process that is needed for a particular platform that the application finds itself running on.
    - voodootrucker 8y ago
      Taken to a logical extent, I think this tool could be used to generate conflict-aware ClassLoaders at application start time, and automatically give each module the version it wants. I guess it would take generating a new jar with a bootloader main() to wrap the existing main()?
latchkey 8y ago
Apache Gump [1] is a similar very old project, but meant to be more realtime. [1] https://gump.apache.org/ https://gump.apache.org/
cortesoft 8y ago
That name makes me think of the adult diapers. Might have a branding issue
- irascible 8y ago
  I dunno.. java is pretty old..
- brian_herman 8y ago
  I think that is the joke.
- deeg 8y ago
  If I recall correctly 'depends' is the name of a similar tool in Windows for debugging DLLs.
- bunderbunder 8y ago
  Given that this tool is for solving problems that stem from the Java platform's incontinent dependency management, that branding seems spot-on to me.
mcnichol 8y ago
This seems less linking and more dependency resolution. What was wrong with > ./gradlew dependencies and then sorting out your duplicate dependencies with improper version issues. I have spent a lot of time down this road myself and most of it was not fully understanding the framework dependencies, munging of stackoverflow solutions, or apache-commons hell...sometimes a little bit of it all. Not trying to crap on your hardwork but interested in your motivations if its something bigger which I am not seeing. Long term I don't think you can properly suggest the appropriate dependency, this is why most build tools provide default selection and let you override with exclude patterns. Additionally there is a linting tool that will parse through duplicates and offer to clean up your gradle files if that is interesting to you alongside your work.
- bassman9000 8y ago
  Couple this to the fact that you can search for the class at https://search.maven.org/#browse https://search.maven.org/#browse to find the artifact hosting it, and it tends to be a couple minute job Bonus tracks: gradle dependencyInsight is great for a similar task: finding out which artifact is bringing the dep.
  - village-idiot 8y ago
    That’s all fine and well if you have one or two broken dependencies from big popular libraries. This technique does not scale if you have a few dozen internal libraries and a few hundred internally deployed artifacts, or more. At that scale you need a more automated solution.
    - bassman9000 8y ago
      Anecdote: we have a large codebase with probably 100+ 3rd party dependencies, and dozens of internal modules, and that technique works pretty well. Besides, where's the automated part on the OP solution? I agree the automation, either with this tool, or vanilla project management ones like gradle, could be beneficial.
ilovecaching 8y ago
That's not a linker, a linker produces an executable from object files.
exabrial 8y ago
I can't imagine what sort of ill managed corp crapshoot frameworks you're importing in order to necessitate this tool :( but you might also try mvn dependency:analyze to help identify problem areas
- kpil 8y ago
  Given a huge codebase: Spring. Blue-coloured j2ee containers. Old non-mavenized applications found as source jars, etc.
  - mrherder 8y ago
    Spring has for some time solved this by using a bon with set versions of each library. Works great in Maven, and with Gradle via a plugin: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#using-boot-dependency-management https://docs.spring.io/spring-boot/docs/current/reference/ht...
    - eecc 8y ago
      Incredibly, there might be non-spring applications that need to be maintained, or even non spring-boot ones /s
      - mrherder 8y ago
        I've heard some rumours about that, but wrote them off as too implausible.
tofflos 8y ago
Nice! I have two suggestions: I'm probably too lazy to download and run a JAR but I'd probably run a Maven plugin from the command-line. Try packaging it as a plugin. I often run into ClassNotFoundException when using an API but not providing an implementation, when using an optional feature that is provided in a separate dependency (Jersey HK2), and when using a feature that has been removed or broken into a separate dependency (JAXB). I believe these scenarios are all fairly common. Will your program help developers resolve those kinds of problems?
- voodootrucker 8y ago
  Thanks! This is great advice. I will look into it. I would definitely want it to be available for both Maven and Gradle, but I think this should be achievable.

124 points by voodootrucker 8y ago

rad_gruchalski 8y ago

Great stuff. Never got to building this myself but this is awesome.

setheron 8y ago

There will always be a case where you can't solve the diamond dependency problem.

voodootrucker 8y ago
Two responses to the diamond problem: - I see this often with Guava. A wants 18 and B wants 21, neither works with the other, but they both with with 20. Because this tool looks at the actual methods being invoked, even if 100% of the methods are present, it's fine as long as 100% of the ones being invoked are. - Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list.
- sleepychu 8y ago
  Isn't choosing to sub in 20 just helping you build more on an unstable foundation? 1 - how can you be sure the method implementation is the same? That there aren't side effects you don't know about. (I don't think you can!) 2 - when it emerges that B2 really does need 25 what are you going to do? IMO it would be better to resolve this earlier than later.
- Slartie 8y ago
  > Technically, as long as objects aren't exchanged between modules, it would be possible to use ClassLoader isolation to have two libraries each load their own incompatible dependency at run time. It's on my blue-sky TODO list. Not only technically, this also works practically. We've done this for years in a POS application in order to be able to load different incompatible versions of vendor-provided hardware drivers at the same time, with the decision on which to use (one, the other, or maybe even both at the same time) being done at runtime. It's based on OSGi for modularization, which also provides other benefits besides classloader isolation, like allowing to bundle multiple native libs for different OSes and bit-nesses within a single jar and automatically choosing the right one to dynamically link into your JVM process that is needed for a particular platform that the application finds itself running on.
  - voodootrucker 8y ago
    Taken to a logical extent, I think this tool could be used to generate conflict-aware ClassLoaders at application start time, and automatically give each module the version it wants. I guess it would take generating a new jar with a bootloader main() to wrap the existing main()?

latchkey 8y ago

Apache Gump [1] is a similar very old project, but meant to be more realtime. [1] https://gump.apache.org/ https://gump.apache.org/

cortesoft 8y ago

That name makes me think of the adult diapers. Might have a branding issue

irascible 8y ago
I dunno.. java is pretty old..
brian_herman 8y ago
I think that is the joke.
deeg 8y ago
If I recall correctly 'depends' is the name of a similar tool in Windows for debugging DLLs.
bunderbunder 8y ago
Given that this tool is for solving problems that stem from the Java platform's incontinent dependency management, that branding seems spot-on to me.

mcnichol 8y ago

This seems less linking and more dependency resolution. What was wrong with > ./gradlew dependencies and then sorting out your duplicate dependencies with improper version issues. I have spent a lot of time down this road myself and most of it was not fully understanding the framework dependencies, munging of stackoverflow solutions, or apache-commons hell...sometimes a little bit of it all. Not trying to crap on your hardwork but interested in your motivations if its something bigger which I am not seeing. Long term I don't think you can properly suggest the appropriate dependency, this is why most build tools provide default selection and let you override with exclude patterns. Additionally there is a linting tool that will parse through duplicates and offer to clean up your gradle files if that is interesting to you alongside your work.

bassman9000 8y ago
Couple this to the fact that you can search for the class at https://search.maven.org/#browse https://search.maven.org/#browse to find the artifact hosting it, and it tends to be a couple minute job Bonus tracks: gradle dependencyInsight is great for a similar task: finding out which artifact is bringing the dep.
- village-idiot 8y ago
  That’s all fine and well if you have one or two broken dependencies from big popular libraries. This technique does not scale if you have a few dozen internal libraries and a few hundred internally deployed artifacts, or more. At that scale you need a more automated solution.
  - bassman9000 8y ago
    Anecdote: we have a large codebase with probably 100+ 3rd party dependencies, and dozens of internal modules, and that technique works pretty well. Besides, where's the automated part on the OP solution? I agree the automation, either with this tool, or vanilla project management ones like gradle, could be beneficial.

ilovecaching 8y ago

That's not a linker, a linker produces an executable from object files.

exabrial 8y ago

I can't imagine what sort of ill managed corp crapshoot frameworks you're importing in order to necessitate this tool :( but you might also try mvn dependency:analyze to help identify problem areas

kpil 8y ago
Given a huge codebase: Spring. Blue-coloured j2ee containers. Old non-mavenized applications found as source jars, etc.
- mrherder 8y ago
  Spring has for some time solved this by using a bon with set versions of each library. Works great in Maven, and with Gradle via a plugin: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#using-boot-dependency-management https://docs.spring.io/spring-boot/docs/current/reference/ht...
  - eecc 8y ago
    Incredibly, there might be non-spring applications that need to be maintained, or even non spring-boot ones /s
    - mrherder 8y ago
      I've heard some rumours about that, but wrote them off as too implausible.

tofflos 8y ago

Nice! I have two suggestions: I'm probably too lazy to download and run a JAR but I'd probably run a Maven plugin from the command-line. Try packaging it as a plugin. I often run into ClassNotFoundException when using an API but not providing an implementation, when using an optional feature that is provided in a separate dependency (Jersey HK2), and when using a feature that has been removed or broken into a separate dependency (JAXB). I believe these scenarios are all fairly common. Will your program help developers resolve those kinds of problems?

voodootrucker 8y ago
Thanks! This is great advice. I will look into it. I would definitely want it to be available for both Maven and Gradle, but I think this should be achievable.