3 ms·

Cool! We've been doing something similar with public Maven repositories as the backing store. Jonathan: if you're listening, have you guys looked at Maven as a

by pcl 10y ago

Cool! We've been doing something similar with public Maven repositories as the backing store. Jonathan: if you're listening, have you guys looked at Maven as a source instead of an S3 upload? What percentage of your dependencies are internal vs. external? We get 10-20x space savings from public-Maven-only elision. EDIT: we've been working on a similar technique for executable JARs as well, which shares the same general optimization numbers and approach. It's a bit trickier, since we want to preserve the same 'java -jar' syntax, so requires a bit of classloader tomfoolery, and corresponding classloader awareness in code.

LoneWolf 10y ago
I really like your approach, is there any code you can share on that?
- pcl 10y ago
  Getting it open-sourced has been a bit of a back-burner thing; I'll see if I can stoke the fires a bit.
  - LoneWolf 10y ago
    If it is not open sourced yet, can you at least elaborate a little more on the inner workings?
    - pcl 10y ago
      Going from memory, we do something like this: For WARs: In a mvn build step: - identify candidate dependencies to exclude from the WAR. This includes a filter step to keep internal dependencies in the WEB-INF dir. - generate a JSON manifest of all the excluded dependencies, including their local SHAs, and store the manifest in WEB-INF. - copy a shell script into the new Maven Archive (MAR) that knows how to reverse the process above and output a WAR Then, in our build process, we pass around the MAR files until the moment of deploy, at which time we expand back into a WAR and deploy. We looked at just deploying the MARs themselves, but the way our containers manage file cache space, it was easier to rehydrate them pre-deploy. For executable JARs: again, we start in a Maven build phase, and again identify the dependencies to include vs. the dependencies to exclude and fetch later. We create a totally-new JAR that contains all the classes in the thin Maven-built jar (I believe as a referenced JAR, not as expanded classes). We then bundle all the included JARs directly into the target JAR, and write a CSV manifest identifying which external dependencies are needed at runtime. We then copy a raw .class file with no dependencies [1] into the target JAR and write a manifest that identifies that class as the Main-Class. That class is responsible for invoking 'mvn' via shell execution to load dependencies, building a classloader with the right classpath, etc., and then handing off to the original Main-Class. [1] hence CSV instead of JSON
jpollock 10y ago
I would recommend against depending on public repositories (of any form) as a backing store for repeatable builds. We've repeatedly seen public repositories remove packages, disappear entirely or otherwise make third party projects a release depends on unavailable. If bringing up a production service requires the presence of Maven, it will suffer an outage. Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.
- niftich 10y ago
  You can run Nexus or Archiva or some other Maven artifact mirror within your network to get the best of both worlds -- maven builds and none of your stuff will break if central is down.
  - jpollock 10y ago
    Since those are proxy caches, they don't provide repeatable builds. The referred to release can be changed without your knowledge and Nexus will happily serve that to you as correct.
    - niftich 10y ago
      I don't follow. I shouldn't have mentioned Nexus as I'm a lot more familiar with Archiva. But, - if the release of the same version was changed upstream, and you want this change to be picked up, that's not a repeatable build anyway. - if the release was changed upstream and you don't want to pick it up, configure Archiva to fail on checksum mismatch [1]. - or as an alternative to enforcing the checksum, configure your local repo as internal instead of proxied, and handle it all yourself [2]. [1] https://archiva.apache.org/docs/2.2.1/adminguide/proxy-connectors.html https://archiva.apache.org/docs/2.2.1/adminguide/proxy-conne... [2] https://archiva.apache.org/docs/2.2.1/adminguide/repositories.html https://archiva.apache.org/docs/2.2.1/adminguide/repositorie...
    - qw 10y ago
      If Nexus doesn't have that functionality, you can probably put an Nginx reverse proxy behind it that will always serve local files if available.
makmanalp 10y ago
The iffy thing here (if I understand you correctly) is creating a dependency in your deployments that requires the maven servers to be alive, which isn't an uncommon kind of thing, but just saying - larger companies usually run their own package manager mirrors.
HiJon89 10y ago
We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc. We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.
pron 10y ago
Capsule.io supports Maven as a source, and requires no classloader tomfoolery. It also supports native libraries, either embedded or in a Maven repo, java and native agents, and automatically applies JVM flags embedded in the manifest. It's got Maven, Gradle and Leiningen plugins to assist in capsule creation.

by pcl 10y ago

LoneWolf 10y ago

I really like your approach, is there any code you can share on that?

pcl 10y ago
Getting it open-sourced has been a bit of a back-burner thing; I'll see if I can stoke the fires a bit.
- LoneWolf 10y ago
  If it is not open sourced yet, can you at least elaborate a little more on the inner workings?
  - pcl 10y ago
    Going from memory, we do something like this: For WARs: In a mvn build step: - identify candidate dependencies to exclude from the WAR. This includes a filter step to keep internal dependencies in the WEB-INF dir. - generate a JSON manifest of all the excluded dependencies, including their local SHAs, and store the manifest in WEB-INF. - copy a shell script into the new Maven Archive (MAR) that knows how to reverse the process above and output a WAR Then, in our build process, we pass around the MAR files until the moment of deploy, at which time we expand back into a WAR and deploy. We looked at just deploying the MARs themselves, but the way our containers manage file cache space, it was easier to rehydrate them pre-deploy. For executable JARs: again, we start in a Maven build phase, and again identify the dependencies to include vs. the dependencies to exclude and fetch later. We create a totally-new JAR that contains all the classes in the thin Maven-built jar (I believe as a referenced JAR, not as expanded classes). We then bundle all the included JARs directly into the target JAR, and write a CSV manifest identifying which external dependencies are needed at runtime. We then copy a raw .class file with no dependencies [1] into the target JAR and write a manifest that identifies that class as the Main-Class. That class is responsible for invoking 'mvn' via shell execution to load dependencies, building a classloader with the right classpath, etc., and then handing off to the original Main-Class. [1] hence CSV instead of JSON

jpollock 10y ago

I would recommend against depending on public repositories (of any form) as a backing store for repeatable builds. We've repeatedly seen public repositories remove packages, disappear entirely or otherwise make third party projects a release depends on unavailable. If bringing up a production service requires the presence of Maven, it will suffer an outage. Treat any open source project your product depends on the same way you would anything you paid cash for - store a copy in your code repository and back it up with the rest of your source code.

niftich 10y ago
You can run Nexus or Archiva or some other Maven artifact mirror within your network to get the best of both worlds -- maven builds and none of your stuff will break if central is down.
- jpollock 10y ago
  Since those are proxy caches, they don't provide repeatable builds. The referred to release can be changed without your knowledge and Nexus will happily serve that to you as correct.
  - niftich 10y ago
    I don't follow. I shouldn't have mentioned Nexus as I'm a lot more familiar with Archiva. But, - if the release of the same version was changed upstream, and you want this change to be picked up, that's not a repeatable build anyway. - if the release was changed upstream and you don't want to pick it up, configure Archiva to fail on checksum mismatch [1]. - or as an alternative to enforcing the checksum, configure your local repo as internal instead of proxied, and handle it all yourself [2]. [1] https://archiva.apache.org/docs/2.2.1/adminguide/proxy-connectors.html https://archiva.apache.org/docs/2.2.1/adminguide/proxy-conne... [2] https://archiva.apache.org/docs/2.2.1/adminguide/repositories.html https://archiva.apache.org/docs/2.2.1/adminguide/repositorie...
  - qw 10y ago
    If Nexus doesn't have that functionality, you can probably put an Nginx reverse proxy behind it that will always serve local files if available.

makmanalp 10y ago

The iffy thing here (if I understand you correctly) is creating a dependency in your deployments that requires the maven servers to be alive, which isn't an uncommon kind of thing, but just saying - larger companies usually run their own package manager mirrors.

HiJon89 10y ago

We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc. We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.

pron 10y ago

Capsule.io supports Maven as a source, and requires no classloader tomfoolery. It also supports native libraries, either embedded or in a Maven repo, java and native agents, and automatically applies JVM flags embedded in the manifest. It's got Maven, Gradle and Leiningen plugins to assist in capsule creation.