4 ms·

The moment the article mentioned "Fat jar" I knew that'd be the problem. I don't recommend using any type of fat jar plugin (like OneJar) or even Google Guice

The moment the article mentioned "Fat jar" I knew that'd be the problem. I don't recommend using any type of fat jar plugin (like OneJar) or even Google Guice for that matter. Custom class loaders are a nightmare. Thanks to Docker containers, you should never really need a far jar again. Just find a decent Docker packager for your build system (sbt, gradle, etc.) and it can plop all your dependencies in there in a nice, isolated container that uses the standard class loader.

Quekid5 8y ago
Fat jars are evil, but there's no need for Docker: Just collect the dependency jars in a lib/ (or whatever) folder and explicitly give them on the classpath when running the 'java' executable. We use the sbt 'pack' plugin where I work and it works a treat. (Docker has its place, but it's massive overkill just to avoid fat jars.)
thecatspaw 8y ago
What is problematic about Fat jars? The problem seemed to be Flink's implementation to unload the FatJar's classloader when erroring. This would have happened with slim jars as well, wouldnt it? I also dont see how docker relates exactly, you can have hundreds of library jars in a classpath with standard classloaders, no docker required
- Quekid5 8y ago
  I'm not sure about whether this particular problem has anything to do with fat jars, but there a couple of really big annoyances with fat jars which have bitten me occasionally: * If you flatten the classpath completely when building the fat jar then you need to somehow be able reconcile duplicate classpath entries[1]. (Duplicate class path entries are perfectly within spec as far as I can tell -- at least as long as they are from distinct jars. Not sure if they're allowed in a single jar.) * If you use nested jars (+ custom classloader perhaps) then it becomes impossible to refer to classpath entry resources in nested jars in a standard way (via java.net.URL, that is). Sometimes you can use foo.jar!bar.jar/blah, but even with that non-standard syntax I've encountered at least one case where it was impossible to refer to a doubly-nested classpath resource (don't ask). The lack of standard support for nesting jars seems like an oversight in the class loading/resource API, but there it is. There's probably more, but that's at least a couple of the ones that come to mind. [1] For example the plugin data file used by log4j 2.x where you have to have custom merge logic to handle that specific (binary!) file. You might blame this on log4j, but as a practical matter it's hard to avoid it. This is a pretty rare scenario, but any custom build logic or special-case plugins can be a huge pain for maintenance.
- tjoener 8y ago
  Also Scala's lambdas create new anonymous classes, but Java's lambdas are kind of bootstrapped static methods in the existing class file. It does get converted to a class at runtime, but then it's long past the classloader anyway.
  - yawaramin 8y ago
    Scala 2.12 lambdas compile to Java 8 lambdas.
justinhj 8y ago
I was thinking of doing this. We have a Scala app that we deploy as a fat jar in a docker container. But if we instead shipped the individual files, the diffs between docker builds would be much smaller. Still, the time to build and deploy the docker container is so small it hasn't been worth the effort yet.

Quekid5 8y ago

Fat jars are evil, but there's no need for Docker: Just collect the dependency jars in a lib/ (or whatever) folder and explicitly give them on the classpath when running the 'java' executable. We use the sbt 'pack' plugin where I work and it works a treat. (Docker has its place, but it's massive overkill just to avoid fat jars.)

thecatspaw 8y ago

What is problematic about Fat jars? The problem seemed to be Flink's implementation to unload the FatJar's classloader when erroring. This would have happened with slim jars as well, wouldnt it? I also dont see how docker relates exactly, you can have hundreds of library jars in a classpath with standard classloaders, no docker required

Quekid5 8y ago
I'm not sure about whether this particular problem has anything to do with fat jars, but there a couple of really big annoyances with fat jars which have bitten me occasionally: * If you flatten the classpath completely when building the fat jar then you need to somehow be able reconcile duplicate classpath entries[1]. (Duplicate class path entries are perfectly within spec as far as I can tell -- at least as long as they are from distinct jars. Not sure if they're allowed in a single jar.) * If you use nested jars (+ custom classloader perhaps) then it becomes impossible to refer to classpath entry resources in nested jars in a standard way (via java.net.URL, that is). Sometimes you can use foo.jar!bar.jar/blah, but even with that non-standard syntax I've encountered at least one case where it was impossible to refer to a doubly-nested classpath resource (don't ask). The lack of standard support for nesting jars seems like an oversight in the class loading/resource API, but there it is. There's probably more, but that's at least a couple of the ones that come to mind. [1] For example the plugin data file used by log4j 2.x where you have to have custom merge logic to handle that specific (binary!) file. You might blame this on log4j, but as a practical matter it's hard to avoid it. This is a pretty rare scenario, but any custom build logic or special-case plugins can be a huge pain for maintenance.
tjoener 8y ago
Also Scala's lambdas create new anonymous classes, but Java's lambdas are kind of bootstrapped static methods in the existing class file. It does get converted to a class at runtime, but then it's long past the classloader anyway.
- yawaramin 8y ago
  Scala 2.12 lambdas compile to Java 8 lambdas.

justinhj 8y ago

I was thinking of doing this. We have a Scala app that we deploy as a fat jar in a docker container. But if we instead shipped the individual files, the diffs between docker builds would be much smaller. Still, the time to build and deploy the docker container is so small it hasn't been worth the effort yet.