9 ms·

An alternative to fat JARs

73 points by javinpaul 10y ago

pcl 10y ago
Cool! We've been doing something similar with public Maven repositories as the backing store. Jonathan: if you're listening, have you guys looked at Maven as a source instead of an S3 upload? What percentage of your dependencies are internal vs. external? We get 10-20x space savings from public-Maven-only elision. EDIT: we've been working on a similar technique for executable JARs as well, which shares the same general optimization numbers and approach. It's a bit trickier, since we want to preserve the same 'java -jar' syntax, so requires a bit of classloader tomfoolery, and corresponding classloader awareness in code.
- LoneWolf 10y ago
  I really like your approach, is there any code you can share on that?
  - pcl 10y ago
    Getting it open-sourced has been a bit of a back-burner thing; I'll see if I can stoke the fires a bit.
    - LoneWolf 10y ago
      If it is not open sourced yet, can you at least elaborate a little more on the inner workings?
      - pcl 10y ago
        Going from memory, we do something like this: For WARs: In a mvn build step: - identify candidate dependencies to exclude from the WAR. This includes a filter step to keep internal dependencies in the WEB-INF dir. - generate a JSON manifest of all the excluded dependencies, including their local SHAs, and store the manifest in WEB-INF. - copy a shell script into the new Maven Archive (MAR) that knows how to reverse the process above and output a WAR Then, in our build process, we pass around the MAR files until the moment of deploy, at which time we expand back into a WAR and deploy. We looked at just deploying the MARs themselves, but the way our containers manage file cache space, it was easier to rehydrate them pre-deploy. For executable JARs: again, we start in a Maven build phase, and again identify the dependencies to include vs. the dependencies to exclude and fetch later. We create a totally-new JAR that contains all the classes in the thin Maven-built jar (I believe as a referenced JAR, not as expanded classes). We then bundle all the included JARs directly into the target JAR, and write a CSV manifest identifying which external dependencies are needed at runtime. We then copy a raw .class file with no dependencies [1] into the target JAR and write a manifest that identifies that class as the Main-Class. That class is responsible for invoking 'mvn' via shell execution to load dependencies, building a classloader with the right classpath, etc., and then handing off to the original Main-Class. [1] hence CSV instead of JSON
- jpollock 10y ago
  I would recommend against depending on public repositories (of any form) as a backing store for repeatable builds. We've repeatedly seen public repositories remove packages, disappear entirely or otherwise make third party projects a release depends on unavailable. If bringing up a production service requires the presence of Maven, it will suffer an outage. Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.
  - niftich 10y ago
    You can run Nexus or Archiva or some other Maven artifact mirror within your network to get the best of both worlds -- maven builds and none of your stuff will break if central is down.
    - jpollock 10y ago
      Since those are proxy caches, they don't provide repeatable builds. The referred to release can be changed without your knowledge and Nexus will happily serve that to you as correct.
      - niftich 10y ago
        I don't follow. I shouldn't have mentioned Nexus as I'm a lot more familiar with Archiva. But, - if the release of the same version was changed upstream, and you want this change to be picked up, that's not a repeatable build anyway. - if the release was changed upstream and you don't want to pick it up, configure Archiva to fail on checksum mismatch [1]. - or as an alternative to enforcing the checksum, configure your local repo as internal instead of proxied, and handle it all yourself [2]. [1] https://archiva.apache.org/docs/2.2.1/adminguide/proxy-connectors.html https://archiva.apache.org/docs/2.2.1/adminguide/proxy-conne... [2] https://archiva.apache.org/docs/2.2.1/adminguide/repositories.html https://archiva.apache.org/docs/2.2.1/adminguide/repositorie...
      - qw 10y ago
        If Nexus doesn't have that functionality, you can probably put an Nginx reverse proxy behind it that will always serve local files if available.
- makmanalp 10y ago
  The iffy thing here (if I understand you correctly) is creating a dependency in your deployments that requires the maven servers to be alive, which isn't an uncommon kind of thing, but just saying - larger companies usually run their own package manager mirrors.
- HiJon89 10y ago
  We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc. We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.
- pron 10y ago
  Capsule.io supports Maven as a source, and requires no classloader tomfoolery. It also supports native libraries, either embedded or in a Maven repo, java and native agents, and automatically applies JVM flags embedded in the manifest. It's got Maven, Gradle and Leiningen plugins to assist in capsule creation.
virmundi 10y ago
It's neat. But is it worth it? I thought inbound for S3 was free so the 150 Mb doesn't cost much. You're paying $0.25 a month to store up to a gig of fat jars so really nkthing. It's 20-ish seconds faster. Assuming a CI build, that doesn't matter. It is a more complex solution. This makes it more brittle. Which makes it more bad. Finally you're losing the single deployable. When something goes bad, you can't just download the jar. You have to fish through the maven dependencies.
- swsieber 10y ago
  I will respectfully disagree and say it's worth it in this case. Here's the reasons I see that makes it worth it: * It reduces the build time to less than 50% * They are doing this for 1000+ java applications * The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.
  - brianwawok 10y ago
    So the real question is.. do they NEED 1000-2000 java applications? Or could it be much more sanely organized with say 15 or 20 applications? I know nothing about the codebase but it sounds like they may have nanoservices...
    - HiJon89 10y ago
      It's roughly 300 REST APIs and the rest are cron jobs, kafka workers, Hadoop or Spark jobs, etc. etc. Each part of the product gets its own API which is where most of them come from as our product is quite broad
- jacques_chester 10y ago
  > It's neat. But is it worth it? Saving traffic is good in general, insofar as it speeds deploys and allows higher container density. But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect. I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it. Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.
matt_wulfeck 10y ago
It's hard for me to understand why in 2016 200MB files are still any kind of real problem.
- pjc50 10y ago
  From the article: "Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."
  - matt_wulfeck 10y ago
    I wasn't being critical of the OPs build process. I agree that uploading 200 MB to S3 is slow, but it shouldn't be.
devsatish 10y ago
Back in the day , used to have all common dependency jars in server/lib folder and only deploy slim application war to the server . Reminds me of this
- sytringy05 10y ago
  and remember how bad it was trying to get application server X to deploy your war file from a command line without exploding for some bizarre reason? It still actually makes me sad to think of the hours of my life wasted writing WLST jython scripts trying to get WebLogic to behave properly.
- HiJon89 10y ago
  What was the process for adding or removing a dependency? Changing a version? If two apps needed different versions?
  - niftich 10y ago
    You added the dependency to server/lib. You standardized on versions, and applied only security fixes, testing them in a separate environment first. You only upgraded versions of libs if you had a critical feature you couldn't live without, of if your existing libs were about to be deprecated and out-of-support. I'm not saying it wasn't a dark time, but it had its merits and drawbacks just like everything else.
_pmf_ 10y ago
Reinventing OSGi configurations.
- mindcrime 10y ago
  We're using ServiceMix as a container for services (micro or otherwise) and while OSGI isn't without its own annoyances, it really is nice once you get everything working. The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. Recently I specifically had an issue with Spring in this regard (yeah, yeah, don't use Spring, I know... but there was a specific reason in that case, at least initially) that would have been easy to resolve if the Spring guys still distributed OSGI'ified versions of their jars. Still, all in all, I'm pretty happy with this approach.
  - _pmf_ 10y ago
    > The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. It's a mess, and I fear Java 9/10's modularization approach will even make it worse (so that instead of P2 (Eclipse's Equinox' OSGi bundles) and Maven, we'll have another set ... sigh. I'm still looking for a solution that transparently translates Maven metainformation to OSGi bundle information at some higher level (for example by hooking into the OSGi container's dependency resolution logic). Edit: apparently, such a mechanism exists (ResolverHook)
    - sytringy05 10y ago
      isn't that exactly what wrap is for?
      - mindcrime 10y ago
        That has mostly worked well for me when dealing with non-OSGI jars. Once I figured out how to set things up using the features.xml file and specify wrap: for the generic jars, things worked nicely. All of this said, I'm fairly new to OSGI, so I may still run into some dark corners that will turn me off. But right now, it seems to be serving the purpose well.
chadmaughan 10y ago
First thought (to the article title "Why we stopped building fatjars") was that the build person had "graduated" and took their newly "acquired superpowers elsewhere" [0] Joking aside, fantastic idea =) Have you received a cease and desist from SlimFast yet for violating their trademark? [0] - http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyons/ http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyo...
- gurubavan 10y ago
  Don't believe everything you read -- especially when there's a single source.
sytringy05 10y ago
I did my first fat jar project in 2010 with embedded jersey and it was about 10mb. We use dropwizard now and most of our microservices sit at around 20mb which I think is still slim enough to avoid the extra overhead of managing the dependencies on deployment. IMHO Its hard to overstate how much benefit you get from java -jar
- pron 10y ago
  You can have both. Capsule.io (I'm a developer) gives you the option of either embedding all dependencies in the JAR (including native libraries, if you need them), or just listing Maven dependencies and having them downloaded, cached, and shared on the first launch. It also lets you supply all the JVM flags directly in the JAR manifest.
  - sytringy05 10y ago
    I'll have to check it out, thanks!
- HiJon89 10y ago
  We still use java -jar with this setup. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR with no runtime or ClassLoader funny business. Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.
bradleybuda 10y ago
> We have over 1,000 microservices constantly being built and deployed. Wait until they find out how much faster in-process function calls are than RPCs.
- heartsucker 10y ago
  Wait until they realized microservices can be deployed independently and have strict APIs without people using classes they shouldn't or creating spaghetti code because they have 800k LOC.
  - msbarnett 10y ago
    > without people using classes they shouldn't or creating spaghetti code because they have 800k LOC. Honestly, if you can't trust your devs not to make a mess in a monolith, you can't trust them not to make a distributed mess with microservices. Both architectural patterns have their uses. Neither will save you from bad developers.
    - heartsucker 10y ago
      I think monoliths lend themselves to hacks more often than microservices do. It's too easy to just import something you shouldn't, where as breaking the rest API is a lot harder. You can't just do a bad database access because there's usually a network firewall that will keep you from doing your tomfoolery on the DB. I also could be wrong about this, but I like the microservice after projects reach a certain size.
      - eropple 10y ago
        "Microservices" often lead to the degenerate fragmentary case (look at people bragging about how many microservices they have--that's a weird mindset to have). I am a big fan of a well-designed SOA, but that's not properly bloggable here in 2016.
      - sievebrain 10y ago
        That's what module systems and access modifiers are for. Now, OK, the JVM ecosystem lacks a good module system at the moment. But Java 9 will fix that. There is also OSGi though I don't know much about it.
pacoverdi 10y ago
To deploy a process in my current project, I use Apache Ivy to resolve its dependencies (starting from a small set of top-level modules). Then I cache the results because Ivy resolution tends to take a while when you have a lot of dependencies and a lot of repositories. Then I send a message to a remote agent with the command to run (classpath, main class, jvm args, args, configuration, etc). The agent then downloads the jars that are not already present in its cache with a valid SHA-1 checksum then starts. Works quite well.
exelius 10y ago
Everyone uses fat JARs because it removes a lot of things that can go wrong operationally (especially when standing up new servers). The approach described in this article is the way Java was designed to work; but then everyone said you needed a "self-executing" service binary for portability. Enter the fat JAR, containers, orchestration layers, etc. JVM application servers are certainly the way to go if you run a bunch of Java. But then you're committing to maintaining a bunch of dependencies on your application servers, so really you're pushing the tech debt down the line to your ops folks who now have to ensure dependencies are synced, maven servers running and updated properly, proxies punched for new external dependencies, etc.
1138 10y ago
On limited information, this sounds like a work around on a root problem of runaway dependencies. Maybe something like proguard could reduce deployable jar size to only the classes used. I assume these are daily dev deploys and not production deploys.
- HiJon89 10y ago
  We definitely have some dependency cruft that could be trimmed (lots of relocated copies of Guava due to incompatibilities, for example). We do frequent production deploys (of individual services, there's no such thing as deploying our entire application). To give an idea, it's a little before 1pm here and across our team there have been 180 production deploys already today.
setheron 10y ago
Here is something I mentioned on the reddit post: You can go just as far with http://www.capsule.io/ http://www.capsule.io/ and the capulet to import dependencies with Maven (from a Nexus proxy that also fronts S3) You are now deploying thin Jars and having the artifacts cached on the local host through maven~!
mjt0229 10y ago
It doesn't use S3, but you can do something similar with Coursier: https://github.com/alexarchambault/coursier#launch https://github.com/alexarchambault/coursier#launch
twic 10y ago
This sounds pretty sensible. I wonder if there's a simple way to reproduce it. Maybe: I. Build your app as a thin JAR, with: a. a Main-Class entry in the manifest b. Class-Path entries in the manifest c. a file containing a list of dependency coordinates (group, artifact, version) in META-INF II. Write a shell script (for whatever value of 'shell' you like) which: 1. takes a Maven repo URL and a Maven coordinate (group, artifact, version) 2. downloads the JAR from the repo, extracts its dependency list, then pulls its dependencies too 3. (optionally) somehow records which JAR is the main one, say by writing a tiny shell script or a symlink pointing at it You might be able to use a standard embedded POM instead of a dependency list, which might reduce the work a bit, but that would then require doing transitive dependency resolution at deploy time, which is probably a bad idea. Phase I is something like 10 - 20 lines of Gradle, tops. You could pack it up into a plugin easily enough. Phase II is a similar amount of shell script, maybe more. For extra safety, add SHA3 hashes to the dependency list file, and check them when you download the dependencies. I've described this as using a Maven repo, but that doesn't mean it has to hit Nexus or whatever; you can just put JARs in S3 in the right layout. A while ago, i was doing this by maintaining a repository locally, and just pushing the whole thing to a Bitbucket website: https://bitbucket.org/twic/twic.bitbucket.org/src/14ac48d4c4a9/repository/?at=default https://bitbucket.org/twic/twic.bitbucket.org/src/14ac48d4c4... You could do something similar, perhaps going from Nexus to S3.
- HiJon89 10y ago
  This similar to our setup with the slimfast-plugin. We use the maven-jar-plugin to add the Main-Class and Class-Path entries to the manifest. At build time, we use the upload goal of the slimfast-plugin to upload the dependencies. It automatically reads the configuration of the maven-jar-plugin to make sure the paths it generates match the classpath in the manifest. Then it spits out a JSON file that has info on each dependency, including its location in S3, file size, checksum, and relative path where it needs to be copied to match the Class-Path in the manifest (the JSON entries look like this https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e1bc https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e...). On deploy, the simplest way to get up and running is to use the download goal of the slimfast-plugin. It reads this JSON file, downloads each dependency (using an optional cache folder), verifies the file size and checksum, and copies it to the correct relative path. The application will then start up happily with java -jar. At HubSpot, we instead integrated this download step more transparently into our deploy process. At build time we read this JSON file and store the dependency information in our build database. Our deploy infrastructure already accepts S3 URLs and handles downloading, caching, verifying checksums, and copying to arbitrary directories so we just piggybacked on this existing functionality to have it download the application plus all of its dependencies for us.
cmcginty 10y ago
Sounds like an interesting idea. I would be even more excited to see someone leverage Gradle for this. Assuming you publish your custom jar artifacts from your build server, your "deployment artifact" could be a single Gradle file that acts a your JAR dependency definition and execution wrapper. For deployment, sync the Gradle file and execute the wrapper to launch the service with Gradle handling all dependency updates and caching.

73 points by javinpaul 10y ago

pcl 10y ago

Cool! We've been doing something similar with public Maven repositories as the backing store. Jonathan: if you're listening, have you guys looked at Maven as a source instead of an S3 upload? What percentage of your dependencies are internal vs. external? We get 10-20x space savings from public-Maven-only elision. EDIT: we've been working on a similar technique for executable JARs as well, which shares the same general optimization numbers and approach. It's a bit trickier, since we want to preserve the same 'java -jar' syntax, so requires a bit of classloader tomfoolery, and corresponding classloader awareness in code.

LoneWolf 10y ago
I really like your approach, is there any code you can share on that?
- pcl 10y ago
  Getting it open-sourced has been a bit of a back-burner thing; I'll see if I can stoke the fires a bit.
  - LoneWolf 10y ago
    If it is not open sourced yet, can you at least elaborate a little more on the inner workings?
    - pcl 10y ago
      Going from memory, we do something like this: For WARs: In a mvn build step: - identify candidate dependencies to exclude from the WAR. This includes a filter step to keep internal dependencies in the WEB-INF dir. - generate a JSON manifest of all the excluded dependencies, including their local SHAs, and store the manifest in WEB-INF. - copy a shell script into the new Maven Archive (MAR) that knows how to reverse the process above and output a WAR Then, in our build process, we pass around the MAR files until the moment of deploy, at which time we expand back into a WAR and deploy. We looked at just deploying the MARs themselves, but the way our containers manage file cache space, it was easier to rehydrate them pre-deploy. For executable JARs: again, we start in a Maven build phase, and again identify the dependencies to include vs. the dependencies to exclude and fetch later. We create a totally-new JAR that contains all the classes in the thin Maven-built jar (I believe as a referenced JAR, not as expanded classes). We then bundle all the included JARs directly into the target JAR, and write a CSV manifest identifying which external dependencies are needed at runtime. We then copy a raw .class file with no dependencies [1] into the target JAR and write a manifest that identifies that class as the Main-Class. That class is responsible for invoking 'mvn' via shell execution to load dependencies, building a classloader with the right classpath, etc., and then handing off to the original Main-Class. [1] hence CSV instead of JSON
jpollock 10y ago
I would recommend against depending on public repositories (of any form) as a backing store for repeatable builds. We've repeatedly seen public repositories remove packages, disappear entirely or otherwise make third party projects a release depends on unavailable. If bringing up a production service requires the presence of Maven, it will suffer an outage. Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.
- niftich 10y ago
  You can run Nexus or Archiva or some other Maven artifact mirror within your network to get the best of both worlds -- maven builds and none of your stuff will break if central is down.
  - jpollock 10y ago
    Since those are proxy caches, they don't provide repeatable builds. The referred to release can be changed without your knowledge and Nexus will happily serve that to you as correct.
    - niftich 10y ago
      I don't follow. I shouldn't have mentioned Nexus as I'm a lot more familiar with Archiva. But, - if the release of the same version was changed upstream, and you want this change to be picked up, that's not a repeatable build anyway. - if the release was changed upstream and you don't want to pick it up, configure Archiva to fail on checksum mismatch [1]. - or as an alternative to enforcing the checksum, configure your local repo as internal instead of proxied, and handle it all yourself [2]. [1] https://archiva.apache.org/docs/2.2.1/adminguide/proxy-connectors.html https://archiva.apache.org/docs/2.2.1/adminguide/proxy-conne... [2] https://archiva.apache.org/docs/2.2.1/adminguide/repositories.html https://archiva.apache.org/docs/2.2.1/adminguide/repositorie...
    - qw 10y ago
      If Nexus doesn't have that functionality, you can probably put an Nginx reverse proxy behind it that will always serve local files if available.
makmanalp 10y ago
The iffy thing here (if I understand you correctly) is creating a dependency in your deployments that requires the maven servers to be alive, which isn't an uncommon kind of thing, but just saying - larger companies usually run their own package manager mirrors.
HiJon89 10y ago
We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc. We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.
pron 10y ago
Capsule.io supports Maven as a source, and requires no classloader tomfoolery. It also supports native libraries, either embedded or in a Maven repo, java and native agents, and automatically applies JVM flags embedded in the manifest. It's got Maven, Gradle and Leiningen plugins to assist in capsule creation.

virmundi 10y ago

It's neat. But is it worth it? I thought inbound for S3 was free so the 150 Mb doesn't cost much. You're paying $0.25 a month to store up to a gig of fat jars so really nkthing. It's 20-ish seconds faster. Assuming a CI build, that doesn't matter. It is a more complex solution. This makes it more brittle. Which makes it more bad. Finally you're losing the single deployable. When something goes bad, you can't just download the jar. You have to fish through the maven dependencies.

swsieber 10y ago
I will respectfully disagree and say it's worth it in this case. Here's the reasons I see that makes it worth it: * It reduces the build time to less than 50% * They are doing this for 1000+ java applications * The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.
- brianwawok 10y ago
  So the real question is.. do they NEED 1000-2000 java applications? Or could it be much more sanely organized with say 15 or 20 applications? I know nothing about the codebase but it sounds like they may have nanoservices...
  - HiJon89 10y ago
    It's roughly 300 REST APIs and the rest are cron jobs, kafka workers, Hadoop or Spark jobs, etc. etc. Each part of the product gets its own API which is where most of them come from as our product is quite broad
jacques_chester 10y ago
> It's neat. But is it worth it? Saving traffic is good in general, insofar as it speeds deploys and allows higher container density. But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect. I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it. Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.

matt_wulfeck 10y ago

It's hard for me to understand why in 2016 200MB files are still any kind of real problem.

pjc50 10y ago
From the article: "Combining 100,000 tiny files into a single archive is slow. Uploading this JAR to S3 at the end of the build is slow. Downloading this JAR at deploy time is slow (and can saturate the network cards on our application servers if we have a lot of concurrent deploys)."
- matt_wulfeck 10y ago
  I wasn't being critical of the OPs build process. I agree that uploading 200 MB to S3 is slow, but it shouldn't be.

devsatish 10y ago

Back in the day , used to have all common dependency jars in server/lib folder and only deploy slim application war to the server . Reminds me of this

sytringy05 10y ago
and remember how bad it was trying to get application server X to deploy your war file from a command line without exploding for some bizarre reason? It still actually makes me sad to think of the hours of my life wasted writing WLST jython scripts trying to get WebLogic to behave properly.
HiJon89 10y ago
What was the process for adding or removing a dependency? Changing a version? If two apps needed different versions?
- niftich 10y ago
  You added the dependency to server/lib. You standardized on versions, and applied only security fixes, testing them in a separate environment first. You only upgraded versions of libs if you had a critical feature you couldn't live without, of if your existing libs were about to be deprecated and out-of-support. I'm not saying it wasn't a dark time, but it had its merits and drawbacks just like everything else.

_pmf_ 10y ago

Reinventing OSGi configurations.

mindcrime 10y ago
We're using ServiceMix as a container for services (micro or otherwise) and while OSGI isn't without its own annoyances, it really is nice once you get everything working. The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. Recently I specifically had an issue with Spring in this regard (yeah, yeah, don't use Spring, I know... but there was a specific reason in that case, at least initially) that would have been easy to resolve if the Spring guys still distributed OSGI'ified versions of their jars. Still, all in all, I'm pretty happy with this approach.
- _pmf_ 10y ago
  > The biggest problem I've found so far is simply 3rd party dependencies that aren't packaged with OSGI manifest information. It's a mess, and I fear Java 9/10's modularization approach will even make it worse (so that instead of P2 (Eclipse's Equinox' OSGi bundles) and Maven, we'll have another set ... sigh. I'm still looking for a solution that transparently translates Maven metainformation to OSGi bundle information at some higher level (for example by hooking into the OSGi container's dependency resolution logic). Edit: apparently, such a mechanism exists (ResolverHook)
  - sytringy05 10y ago
    isn't that exactly what wrap is for?
    - mindcrime 10y ago
      That has mostly worked well for me when dealing with non-OSGI jars. Once I figured out how to set things up using the features.xml file and specify wrap: for the generic jars, things worked nicely. All of this said, I'm fairly new to OSGI, so I may still run into some dark corners that will turn me off. But right now, it seems to be serving the purpose well.

chadmaughan 10y ago

First thought (to the article title "Why we stopped building fatjars") was that the build person had "graduated" and took their newly "acquired superpowers elsewhere" [0] Joking aside, fantastic idea =) Have you received a cease and desist from SlimFast yet for violating their trademark? [0] - http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyons/ http://fortune.com/disrupted-excerpt-hubspot-startup-dan-lyo...

gurubavan 10y ago
Don't believe everything you read -- especially when there's a single source.

sytringy05 10y ago

I did my first fat jar project in 2010 with embedded jersey and it was about 10mb. We use dropwizard now and most of our microservices sit at around 20mb which I think is still slim enough to avoid the extra overhead of managing the dependencies on deployment. IMHO Its hard to overstate how much benefit you get from java -jar

pron 10y ago
You can have both. Capsule.io (I'm a developer) gives you the option of either embedding all dependencies in the JAR (including native libraries, if you need them), or just listing Maven dependencies and having them downloaded, cached, and shared on the first launch. It also lets you supply all the JVM flags directly in the JAR manifest.
- sytringy05 10y ago
  I'll have to check it out, thanks!
HiJon89 10y ago
We still use java -jar with this setup. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR with no runtime or ClassLoader funny business. Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.

bradleybuda 10y ago

> We have over 1,000 microservices constantly being built and deployed. Wait until they find out how much faster in-process function calls are than RPCs.

heartsucker 10y ago
Wait until they realized microservices can be deployed independently and have strict APIs without people using classes they shouldn't or creating spaghetti code because they have 800k LOC.
- msbarnett 10y ago
  > without people using classes they shouldn't or creating spaghetti code because they have 800k LOC. Honestly, if you can't trust your devs not to make a mess in a monolith, you can't trust them not to make a distributed mess with microservices. Both architectural patterns have their uses. Neither will save you from bad developers.
  - heartsucker 10y ago
    I think monoliths lend themselves to hacks more often than microservices do. It's too easy to just import something you shouldn't, where as breaking the rest API is a lot harder. You can't just do a bad database access because there's usually a network firewall that will keep you from doing your tomfoolery on the DB. I also could be wrong about this, but I like the microservice after projects reach a certain size.
    - eropple 10y ago
      "Microservices" often lead to the degenerate fragmentary case (look at people bragging about how many microservices they have--that's a weird mindset to have). I am a big fan of a well-designed SOA, but that's not properly bloggable here in 2016.
    - sievebrain 10y ago
      That's what module systems and access modifiers are for. Now, OK, the JVM ecosystem lacks a good module system at the moment. But Java 9 will fix that. There is also OSGi though I don't know much about it.

pacoverdi 10y ago

To deploy a process in my current project, I use Apache Ivy to resolve its dependencies (starting from a small set of top-level modules). Then I cache the results because Ivy resolution tends to take a while when you have a lot of dependencies and a lot of repositories. Then I send a message to a remote agent with the command to run (classpath, main class, jvm args, args, configuration, etc). The agent then downloads the jars that are not already present in its cache with a valid SHA-1 checksum then starts. Works quite well.

exelius 10y ago

Everyone uses fat JARs because it removes a lot of things that can go wrong operationally (especially when standing up new servers). The approach described in this article is the way Java was designed to work; but then everyone said you needed a "self-executing" service binary for portability. Enter the fat JAR, containers, orchestration layers, etc. JVM application servers are certainly the way to go if you run a bunch of Java. But then you're committing to maintaining a bunch of dependencies on your application servers, so really you're pushing the tech debt down the line to your ops folks who now have to ensure dependencies are synced, maven servers running and updated properly, proxies punched for new external dependencies, etc.

1138 10y ago

On limited information, this sounds like a work around on a root problem of runaway dependencies. Maybe something like proguard could reduce deployable jar size to only the classes used. I assume these are daily dev deploys and not production deploys.

HiJon89 10y ago
We definitely have some dependency cruft that could be trimmed (lots of relocated copies of Guava due to incompatibilities, for example). We do frequent production deploys (of individual services, there's no such thing as deploying our entire application). To give an idea, it's a little before 1pm here and across our team there have been 180 production deploys already today.

setheron 10y ago

Here is something I mentioned on the reddit post: You can go just as far with http://www.capsule.io/ http://www.capsule.io/ and the capulet to import dependencies with Maven (from a Nexus proxy that also fronts S3) You are now deploying thin Jars and having the artifacts cached on the local host through maven~!

mjt0229 10y ago

It doesn't use S3, but you can do something similar with Coursier: https://github.com/alexarchambault/coursier#launch https://github.com/alexarchambault/coursier#launch

twic 10y ago

This sounds pretty sensible. I wonder if there's a simple way to reproduce it. Maybe: I. Build your app as a thin JAR, with: a. a Main-Class entry in the manifest b. Class-Path entries in the manifest c. a file containing a list of dependency coordinates (group, artifact, version) in META-INF II. Write a shell script (for whatever value of 'shell' you like) which: 1. takes a Maven repo URL and a Maven coordinate (group, artifact, version) 2. downloads the JAR from the repo, extracts its dependency list, then pulls its dependencies too 3. (optionally) somehow records which JAR is the main one, say by writing a tiny shell script or a symlink pointing at it You might be able to use a standard embedded POM instead of a dependency list, which might reduce the work a bit, but that would then require doing transitive dependency resolution at deploy time, which is probably a bad idea. Phase I is something like 10 - 20 lines of Gradle, tops. You could pack it up into a plugin easily enough. Phase II is a similar amount of shell script, maybe more. For extra safety, add SHA3 hashes to the dependency list file, and check them when you download the dependencies. I've described this as using a Maven repo, but that doesn't mean it has to hit Nexus or whatever; you can just put JARs in S3 in the right layout. A while ago, i was doing this by maintaining a repository locally, and just pushing the whole thing to a Bitbucket website: https://bitbucket.org/twic/twic.bitbucket.org/src/14ac48d4c4a9/repository/?at=default https://bitbucket.org/twic/twic.bitbucket.org/src/14ac48d4c4... You could do something similar, perhaps going from Nexus to S3.

HiJon89 10y ago
This similar to our setup with the slimfast-plugin. We use the maven-jar-plugin to add the Main-Class and Class-Path entries to the manifest. At build time, we use the upload goal of the slimfast-plugin to upload the dependencies. It automatically reads the configuration of the maven-jar-plugin to make sure the paths it generates match the classpath in the manifest. Then it spits out a JSON file that has info on each dependency, including its location in S3, file size, checksum, and relative path where it needs to be copied to match the Class-Path in the manifest (the JSON entries look like this https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e1bc https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e...). On deploy, the simplest way to get up and running is to use the download goal of the slimfast-plugin. It reads this JSON file, downloads each dependency (using an optional cache folder), verifies the file size and checksum, and copies it to the correct relative path. The application will then start up happily with java -jar. At HubSpot, we instead integrated this download step more transparently into our deploy process. At build time we read this JSON file and store the dependency information in our build database. Our deploy infrastructure already accepts S3 URLs and handles downloading, caching, verifying checksums, and copying to arbitrary directories so we just piggybacked on this existing functionality to have it download the application plus all of its dependencies for us.

cmcginty 10y ago

Sounds like an interesting idea. I would be even more excited to see someone leverage Gradle for this. Assuming you publish your custom jar artifacts from your build server, your "deployment artifact" could be a single Gradle file that acts a your JAR dependency definition and execution wrapper. For deployment, sync the Gradle file and execute the wrapper to launch the service with Gradle handling all dependency updates and caching.