4 ms·

Seems like a rather unnecessarily complex dependency to do remote JIT compiles. "it can reduce compute resources by up to 50 per cent" How ... Do they not inc

Seems like a rather unnecessarily complex dependency to do remote JIT compiles. "it can reduce compute resources by up to 50 per cent" How ... Do they not include their cloud service in the computations required?

hyperman1 5y ago
Presumably because it becomes a shared cost. 100 JVMs in 100 containers have to pay that cost individually. Now 1 or 2 central instances have the resources for compiling, the 100 worker bees don't need them. They also might becable to optimize faster as they have runtime statistics from other instances, skipping warmup.
- dragontamer 5y ago
  100 JVMs in 100 containers (on one machine) also have to pay the costs of 100x individual garbage collections, instead of centralizing those garbage collections and merging the memories of everyone together. Hmmm... maybe just having a monolith at that point is better? If you can run 100 containers on one box, why not just run one environment on one box?
  - vlovich123 5y ago
    I know you're joking but there is an important distinction that enables this. The optimization choices you're going to make of how to lower bytecode to native are going to be largely the same across all your machines - the minute differences in profiles aren't going to make significant differences unless you have the same codebase doing drastically different things. Thus you can share cost easily that so you accumulate execution profiles across your cluster and then lower to native code efficiently. The garbage collection choices are going to be unique to the workload running on that server and have little information worth sharing that will speed up operations.
    - hinkley 5y ago
      We could in theory get a lot of this benefit by supporting profile guided optimization and building a simple distribution system. You could even instrument a fraction of your servers and distribute to the rest.
    - dragontamer 5y ago
      > The garbage collection choices are going to be unique to the workload running on that server and have little information worth sharing that will speed up operations. Java's GC is generational though. Which means that Java-style GC benefits from excessive memory (ie: the more free memory you have, the less the GC is run, and the more that any such object is "dead" before the GC is run). With more "dead objects" left in the heap, a "bigger heap" benefits all processes. -------- Consider this simple case: 1MB of live data in a 6MB heap. The garbage collector could run 10 times in some given time period (the heap drops down to 1MB each time the GC is run, and 10-runs means that ~51MBs were allocated in this timeframe 6MBs before the first garbage collection, and then 5MB for each of the remaining 9 collections). Now what happens if you had a 11MB heap instead? Well, the garbage collector won't run until 11MBs were allocated (1st run), and with 1MB of live data after each collection, each remaining run would only be done after 10MBs of allocations. That is to say: only 5-collections would happen. So with the same algorithm, the 11MB heap with 1MB of live data is twice-as-efficient (ie: half the collections) as the 6MB heap (with 1MB of live data). ------- Java and its garbage collector is multithreaded these days and the analysis is more difficult. But that doesn't change the fact that generational collectors have this property, where the more RAM they use between collections, the "better they perform". So just "defacto-sharing more RAM" means all of the individual Java-threads will be more efficient at garbage collection. Well-written heaps get more efficient the more you throw at them.
      - vlovich123 5y ago
        Oh. I misread your suggestion. Yes, if you can somehow have all the services living within the same address space, a shared heap could probably be better. In the model described here we're talking about a multi-computer distributed system though, not a bunch of containers on one machine. Also, by merging everything into one address space you're now potentially making the blast radius of a security vulnerability much larger. Also, it's not instantly clear things actually get better because while your heap space is larger, so is your object graph and the rate at which your objects are getting created is larger. You'll get some compression out of it by sharing the class information across services but not much more than that.
chrisseaton 5y ago
> How ... Not sure why you're so confused? Instead of compiling one method a thousand times, you compile it once and share the result. You still count the computation needed for the one compilation, but strictly reduces compute resources as less work is being done overall.
- usrusr 5y ago
  A remake of those old p2p file-sharing networks but instead of advertising file availability nodes would advertise availability of particularly well-optimized versions of the basic code unit every node already has? Sounds ridiculous and even a bit pointless because you'd still need the capacity to run lots of compilations along not-yet-optimized code during startup (e.g. after redeploy), but with cluster size being highly variable and usage cost more and more replacing capacity cost as the primary concern this might actually work out. For very large clusters and certain workloads. And the more you modulate cluster size depending on demand, the bigger the benefit from reusing across nodes. Sure, it would be more straight-forward to just clone a fully baked node, but there's a nice peace of mind in not having to decide between cold start and hot clone, just leave it to the self-organization mechanism.
  - chrisseaton 5y ago
    I think you're a bit confused about the basic idea. > every node already has? No every node doesn't already have it. That's the point. One node computes on behalf of everyone. > you'd still need the capacity to run lots of compilations along not-yet-optimized code during startup Well no - nodes also have interpreters so they don't wait. And it's not any slower for someone else to run the compilation on your behalf - the result comes in the same time. And most code doesn't change between redeploys so doesn't need to be recompiled.
    - usrusr 5y ago
      Every node already has the bytecode and the interpreter, or more likely fast, superficial JIT without expensive optimisation. But that's slow, so when you need to serve a given load you'll need more nodes before they have settled into a hot optimization state. Regarding unchanged code after redeploy, usage patterns might change a lot even for code that hasn't changed at all. With profile guided optimizations (which I'd expect to be more norm than exception for something like azul), this would mean that nominally unchanged code could suffer hard from running in a version optimized for a different calling environment (the previous version of the application). In the grand scheme of things, I'd assume that this would be the main benefit and goal of a JIT outcome sharing mechanism like this: allowing nodes to cooperate by each profiling a different part of their shared code, kind of like becoming experts in a tiny specialization, and then sharing that expertise.

hyperman1 5y ago

Presumably because it becomes a shared cost. 100 JVMs in 100 containers have to pay that cost individually. Now 1 or 2 central instances have the resources for compiling, the 100 worker bees don't need them. They also might becable to optimize faster as they have runtime statistics from other instances, skipping warmup.

dragontamer 5y ago
100 JVMs in 100 containers (on one machine) also have to pay the costs of 100x individual garbage collections, instead of centralizing those garbage collections and merging the memories of everyone together. Hmmm... maybe just having a monolith at that point is better? If you can run 100 containers on one box, why not just run one environment on one box?
- vlovich123 5y ago
  I know you're joking but there is an important distinction that enables this. The optimization choices you're going to make of how to lower bytecode to native are going to be largely the same across all your machines - the minute differences in profiles aren't going to make significant differences unless you have the same codebase doing drastically different things. Thus you can share cost easily that so you accumulate execution profiles across your cluster and then lower to native code efficiently. The garbage collection choices are going to be unique to the workload running on that server and have little information worth sharing that will speed up operations.
  - hinkley 5y ago
    We could in theory get a lot of this benefit by supporting profile guided optimization and building a simple distribution system. You could even instrument a fraction of your servers and distribute to the rest.
  - dragontamer 5y ago
    > The garbage collection choices are going to be unique to the workload running on that server and have little information worth sharing that will speed up operations. Java's GC is generational though. Which means that Java-style GC benefits from excessive memory (ie: the more free memory you have, the less the GC is run, and the more that any such object is "dead" before the GC is run). With more "dead objects" left in the heap, a "bigger heap" benefits all processes. -------- Consider this simple case: 1MB of live data in a 6MB heap. The garbage collector could run 10 times in some given time period (the heap drops down to 1MB each time the GC is run, and 10-runs means that ~51MBs were allocated in this timeframe 6MBs before the first garbage collection, and then 5MB for each of the remaining 9 collections). Now what happens if you had a 11MB heap instead? Well, the garbage collector won't run until 11MBs were allocated (1st run), and with 1MB of live data after each collection, each remaining run would only be done after 10MBs of allocations. That is to say: only 5-collections would happen. So with the same algorithm, the 11MB heap with 1MB of live data is twice-as-efficient (ie: half the collections) as the 6MB heap (with 1MB of live data). ------- Java and its garbage collector is multithreaded these days and the analysis is more difficult. But that doesn't change the fact that generational collectors have this property, where the more RAM they use between collections, the "better they perform". So just "defacto-sharing more RAM" means all of the individual Java-threads will be more efficient at garbage collection. Well-written heaps get more efficient the more you throw at them.
    - vlovich123 5y ago
      Oh. I misread your suggestion. Yes, if you can somehow have all the services living within the same address space, a shared heap could probably be better. In the model described here we're talking about a multi-computer distributed system though, not a bunch of containers on one machine. Also, by merging everything into one address space you're now potentially making the blast radius of a security vulnerability much larger. Also, it's not instantly clear things actually get better because while your heap space is larger, so is your object graph and the rate at which your objects are getting created is larger. You'll get some compression out of it by sharing the class information across services but not much more than that.

chrisseaton 5y ago

> How ... Not sure why you're so confused? Instead of compiling one method a thousand times, you compile it once and share the result. You still count the computation needed for the one compilation, but strictly reduces compute resources as less work is being done overall.

usrusr 5y ago
A remake of those old p2p file-sharing networks but instead of advertising file availability nodes would advertise availability of particularly well-optimized versions of the basic code unit every node already has? Sounds ridiculous and even a bit pointless because you'd still need the capacity to run lots of compilations along not-yet-optimized code during startup (e.g. after redeploy), but with cluster size being highly variable and usage cost more and more replacing capacity cost as the primary concern this might actually work out. For very large clusters and certain workloads. And the more you modulate cluster size depending on demand, the bigger the benefit from reusing across nodes. Sure, it would be more straight-forward to just clone a fully baked node, but there's a nice peace of mind in not having to decide between cold start and hot clone, just leave it to the self-organization mechanism.
- chrisseaton 5y ago
  I think you're a bit confused about the basic idea. > every node already has? No every node doesn't already have it. That's the point. One node computes on behalf of everyone. > you'd still need the capacity to run lots of compilations along not-yet-optimized code during startup Well no - nodes also have interpreters so they don't wait. And it's not any slower for someone else to run the compilation on your behalf - the result comes in the same time. And most code doesn't change between redeploys so doesn't need to be recompiled.
  - usrusr 5y ago
    Every node already has the bytecode and the interpreter, or more likely fast, superficial JIT without expensive optimisation. But that's slow, so when you need to serve a given load you'll need more nodes before they have settled into a hot optimization state. Regarding unchanged code after redeploy, usage patterns might change a lot even for code that hasn't changed at all. With profile guided optimizations (which I'd expect to be more norm than exception for something like azul), this would mean that nominally unchanged code could suffer hard from running in a version optimized for a different calling environment (the previous version of the application). In the grand scheme of things, I'd assume that this would be the main benefit and goal of a JIT outcome sharing mechanism like this: allowing nodes to cooperate by each profiling a different part of their shared code, kind of like becoming experts in a tiny specialization, and then sharing that expertise.