3 ms·

It's neat. But is it worth it? I thought inbound for S3 was free so the 150 Mb doesn't cost much. You're paying $0.25 a month to store up to a gig of fat jars s

It's neat. But is it worth it? I thought inbound for S3 was free so the 150 Mb doesn't cost much. You're paying $0.25 a month to store up to a gig of fat jars so really nkthing. It's 20-ish seconds faster. Assuming a CI build, that doesn't matter. It is a more complex solution. This makes it more brittle. Which makes it more bad. Finally you're losing the single deployable. When something goes bad, you can't just download the jar. You have to fish through the maven dependencies.

swsieber 10y ago
I will respectfully disagree and say it's worth it in this case. Here's the reasons I see that makes it worth it: * It reduces the build time to less than 50% * They are doing this for 1000+ java applications * The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.
- brianwawok 10y ago
  So the real question is.. do they NEED 1000-2000 java applications? Or could it be much more sanely organized with say 15 or 20 applications? I know nothing about the codebase but it sounds like they may have nanoservices...
  - HiJon89 10y ago
    It's roughly 300 REST APIs and the rest are cron jobs, kafka workers, Hadoop or Spark jobs, etc. etc. Each part of the product gets its own API which is where most of them come from as our product is quite broad
jacques_chester 10y ago
> It's neat. But is it worth it? Saving traffic is good in general, insofar as it speeds deploys and allows higher container density. But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect. I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it. Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.
- HiJon89 10y ago
  Agreed, we might not have done it if we had to resort to having the application download its own dependencies at startup. In our case, our Mesos scheduler, Singularity, handles S3 artifacts at deploy time for us. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. By the time the app starts up, all of its dependencies are guaranteed to be present (if an S3 download failed, the deploy would have failed) so it's totally transparent to the application.
  - jacques_chester 10y ago
    That makes sense. I'm used to having to worry about disconnected environments (I worked on Cloud Foundry buildpacks for a while), for which Spring Boot's JARs work a treat. In a fully connected environment this approach looks promising. If I bump into any of the Spring folks I will mention it.
HiJon89 10y ago
20-30 seconds ends up being 50% for most of our Java builds, so it was a huge speedup. It also saves 20-30 seconds at deploy time when we need to download this file again. And with lots of concurrent deploys we were occasionally seeing these large downloads saturate the NIC on our application servers. Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. When something goes wrong we rarely downloaded the JAR, but if you really wanted to you could download the thin JAR and use the SlimFast metadata bundled with it to download the exact dependencies it would get when deployed.

swsieber 10y ago

I will respectfully disagree and say it's worth it in this case. Here's the reasons I see that makes it worth it: * It reduces the build time to less than 50% * They are doing this for 1000+ java applications * The code vs library ratio is 1:100 (roughly) - that reduces the uploads from 50-100 GB cited in the article down to 50-100 MB.

brianwawok 10y ago
So the real question is.. do they NEED 1000-2000 java applications? Or could it be much more sanely organized with say 15 or 20 applications? I know nothing about the codebase but it sounds like they may have nanoservices...
- HiJon89 10y ago
  It's roughly 300 REST APIs and the rest are cron jobs, kafka workers, Hadoop or Spark jobs, etc. etc. Each part of the product gets its own API which is where most of them come from as our product is quite broad

jacques_chester 10y ago

> It's neat. But is it worth it? Saving traffic is good in general, insofar as it speeds deploys and allows higher container density. But for me the key was trying to get out of dependency hell. The shade plugin isn't perfect. I work at Pivotal, we inherited the Spring team from VMWare. When I first saw Spring Boot I wondered what all the fuss was about. Then I worked with a classic Spring-with-hand-rolled-Maven app and oh boy, let me tell you, I got it. Having someone else level the dependencies for you? Huge. I'd be interest if this tool can help Spring Boot too, though the runtime downloading of dependencies is not without problems.

HiJon89 10y ago
Agreed, we might not have done it if we had to resort to having the application download its own dependencies at startup. In our case, our Mesos scheduler, Singularity, handles S3 artifacts at deploy time for us. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. By the time the app starts up, all of its dependencies are guaranteed to be present (if an S3 download failed, the deploy would have failed) so it's totally transparent to the application.
- jacques_chester 10y ago
  That makes sense. I'm used to having to worry about disconnected environments (I worked on Cloud Foundry buildpacks for a while), for which Spring Boot's JARs work a treat. In a fully connected environment this approach looks promising. If I bump into any of the Spring folks I will mention it.

HiJon89 10y ago

20-30 seconds ends up being 50% for most of our Java builds, so it was a huge speedup. It also saves 20-30 seconds at deploy time when we need to download this file again. And with lots of concurrent deploys we were occasionally seeing these large downloads saturate the NIC on our application servers. Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. When something goes wrong we rarely downloaded the JAR, but if you really wanted to you could download the thin JAR and use the SlimFast metadata bundled with it to download the exact dependencies it would get when deployed.