3 ms·

I've spent some time in the past few weeks trying to rebuild a bunch of uberjars as capsules, and have found it not suitable for production server use. The main

by necubi 11y ago

I've spent some time in the past few weeks trying to rebuild a bunch of uberjars as capsules, and have found it not suitable for production server use. The main promise for me is that rather than shipping 100M+ uberjars, we can ship just our code (~5MB) and have the capsule download its dependencies on startup. This sounds pretty nice in theory: your boxes can cache your dependencies (and transitive dependencies) and new deploys only need to update the new stuff (generally just your code). But there are a bunch of ways capsule makes this difficult, particularly in the context of a mono-repo with a bunch of internal libraries. Capsule lacks a real transitive dependency management system, so you're stuck bypassing it entirely and having your build system compute the complete graph of transitive dependencies and write it in the jar manifest. Even worse, on every start up capsule will open all the jars that have already been cached to validate checksums, which can take many minutes if you have a lot of dependencies. The idea is really solid. I just wish the implementation were better.

pron 11y ago
> Capsule lacks a real transitive dependency management system The Maven caplet uses Aether which is the same dependency-resolution library used by Maven. If you've found a problem, file an issue and we'll solve it. > so you're stuck bypassing it entirely and having your build system compute the complete graph of transitive dependencies and write it in the jar manifest That sure sounds like a bug. The Maven capsule should resolve transitive dependencies.
- necubi 11y ago
  I'm referring to the issue in this thread [0]. I haven't seen any issues with how capsule (or rather, the maven caplet) does dependency resolution; it's that you don't have much control over transitive dependencies. This is particularly a problem when trying to create a capsule that contains the output of your internal dependencies but not your external dependencies. In other words, we've spent a lot of time getting gradle to package the correct set of transitive dependencies so that our code works; I have no desires to re-implement that in capsule's limited dependency system. See also this bug on maven-capsule-plugin [1]. Having tried to actually implement that strategy (albeit in gradle), it's clear that capsule isn't well-adapted to this sort of thing. [0] https://groups.google.com/forum/#!topic/capsule-user/MjtnvwtzSmI https://groups.google.com/forum/#!topic/capsule-user/Mjtnvwt... [1] https://github.com/chrischristo/capsule-maven-plugin/issues/28 https://github.com/chrischristo/capsule-maven-plugin/issues/...
  - pron 11y ago
    Ah, well, that's a design issue we've been grappling with, and haven't yet come up with a definitive solution. The opposing forces are convenience vs. repeatable results, and hopefully we'd like to provide the former without harming the latter. The way I see it, there are three options: 1) Allow a more elaborate description of dependencies 2) Improve the build-tool plugins to make generating the full dependency tree in the build tool and passing that on to Capsule more convenient. I favor option 2, b/c the precise resolution already happens in the build tool, and seems like an easy and complete solution. We can continue this discussion on the mailing list if you like. In any event, as you can see, we are always open to suggestions and happy to accept PRs.
mwcampbell 11y ago
I think fetching dependencies at run time is a bad idea. The more dynamism and complexity you have at run time, the more ways there are for something to go wrong. For maximum robustness, I think a good rule is to never do at run time what you can do at build time. So in that respect, a fat jar is better. Also, in princple a fat-jar deployment should start faster, because (given an appropriately optimized JVM implementation) the system can just map one file into memory and go. So why not just use a fat jar, as you were before?
- necubi 11y ago
  I agree, in principle. The issue is that as the number of dependencies (and, in particular, transitive dependencies) grows, fat jars become less and less wieldy. Our uberjars are in the 150-200MB range, which means they take up a lot of space (we produce and store artifacts from every merge to master) and take a while to build and deploy.
- pron 11y ago
  Capsule by itself does not fetch dependencies at runtime. But because Capsule is so flexible, we have a caplet[1] that you can choose to use that does. And a capsule is a fat jar. Non-capsule fat jars frequently just don't work: they can create resolution conflicts which requires shadowing and even that can fail, they don't support native dependencies, and most importantly they require a startup script as they don't set up JVM options. [1]: http://www.capsule.io/caplets/ http://www.capsule.io/caplets/

by necubi 11y ago

pron 11y ago

> Capsule lacks a real transitive dependency management system The Maven caplet uses Aether which is the same dependency-resolution library used by Maven. If you've found a problem, file an issue and we'll solve it. > so you're stuck bypassing it entirely and having your build system compute the complete graph of transitive dependencies and write it in the jar manifest That sure sounds like a bug. The Maven capsule should resolve transitive dependencies.

necubi 11y ago
I'm referring to the issue in this thread [0]. I haven't seen any issues with how capsule (or rather, the maven caplet) does dependency resolution; it's that you don't have much control over transitive dependencies. This is particularly a problem when trying to create a capsule that contains the output of your internal dependencies but not your external dependencies. In other words, we've spent a lot of time getting gradle to package the correct set of transitive dependencies so that our code works; I have no desires to re-implement that in capsule's limited dependency system. See also this bug on maven-capsule-plugin [1]. Having tried to actually implement that strategy (albeit in gradle), it's clear that capsule isn't well-adapted to this sort of thing. [0] https://groups.google.com/forum/#!topic/capsule-user/MjtnvwtzSmI https://groups.google.com/forum/#!topic/capsule-user/Mjtnvwt... [1] https://github.com/chrischristo/capsule-maven-plugin/issues/28 https://github.com/chrischristo/capsule-maven-plugin/issues/...
- pron 11y ago
  Ah, well, that's a design issue we've been grappling with, and haven't yet come up with a definitive solution. The opposing forces are convenience vs. repeatable results, and hopefully we'd like to provide the former without harming the latter. The way I see it, there are three options: 1) Allow a more elaborate description of dependencies 2) Improve the build-tool plugins to make generating the full dependency tree in the build tool and passing that on to Capsule more convenient. I favor option 2, b/c the precise resolution already happens in the build tool, and seems like an easy and complete solution. We can continue this discussion on the mailing list if you like. In any event, as you can see, we are always open to suggestions and happy to accept PRs.

mwcampbell 11y ago

I think fetching dependencies at run time is a bad idea. The more dynamism and complexity you have at run time, the more ways there are for something to go wrong. For maximum robustness, I think a good rule is to never do at run time what you can do at build time. So in that respect, a fat jar is better. Also, in princple a fat-jar deployment should start faster, because (given an appropriately optimized JVM implementation) the system can just map one file into memory and go. So why not just use a fat jar, as you were before?

necubi 11y ago
I agree, in principle. The issue is that as the number of dependencies (and, in particular, transitive dependencies) grows, fat jars become less and less wieldy. Our uberjars are in the 150-200MB range, which means they take up a lot of space (we produce and store artifacts from every merge to master) and take a while to build and deploy.
pron 11y ago
Capsule by itself does not fetch dependencies at runtime. But because Capsule is so flexible, we have a caplet[1] that you can choose to use that does. And a capsule is a fat jar. Non-capsule fat jars frequently just don't work: they can create resolution conflicts which requires shadowing and even that can fail, they don't support native dependencies, and most importantly they require a startup script as they don't set up JVM options. [1]: http://www.capsule.io/caplets/ http://www.capsule.io/caplets/