4 ms·

I'd typically agree with your comment but ... Given that they also experienced a 90% reduction in Memory Usage (presumably from Java GC vs Swift AOT memory man

I'd typically agree with your comment but ... Given that they also experienced a 90% reduction in Memory Usage (presumably from Java GC vs Swift AOT memory management) - it seems more likely the gains are in fact from the difference in languages.

plorkyeran 1y ago
The typical rule of thumb is that getting good performance out of tracing GC requires doubling your memory usage, so a 90% reduction suggests that they made significant improvements on top of the language switch.
- username223 1y ago
  I remember the 2x rule from 20 years ago - do you know if things have changed? If locality is more important now, tracing GC might never be as performant as reference counting. Either you use 2x the memory and thrash your cache, or you use less and spend too much CPU time collecting.
  - jeroenhd 1y ago
    Java has had AOT compilation for a while, so traditional GC and its massive overhead are no longer a strict necessity. Even AOT Java will probably stay behind Swift or any other natively compiled language in terms of memory usage, but it shouldn't be that drastic. As for performance and locality, Java's on-the-fly pointer reordering/compression can give it an edge over even some compiled languages in certain algorithms. Hard to say if that's relevant for whatever web framework Apple based their service on, but I wouldn't discount Java's locality optimisations just because it uses a GC.
    - username223 1y ago
      > Java has had AOT compilation for a while, so traditional GC and its massive overhead are no longer a strict necessity. You mean it does escape analysis and stack-allocates what it can? That would definitely help, but not eliminate the GC. Or are you thinking of something else? Thinking about it more, I remember that Java also has some performance-hostile design decisions baked in (e.g. almost everything's an Object, arrays aren't packed, dynamic dispatch everywhere). Swift doesn't have that legacy to deal with.
      - acdha 1y ago
        Java also has a lot of culture making optimization-resistant code so there’s the question of whether you’re talking about the language itself or various widespread libraries, especially if they’re old enough to have patterns designed around aesthetics or now-moot language limitations rather than performance. I’ve replaced Java code with Python a few times and each time even though we did it for maintenance (more Python devs available) we saw memory usage more than halved while performance at least doubled because the code used simpler functions and structures. Java has a far more advanced GC and JIT but at some point the weight of code and indirection wins out.
        winrid 1y ago
        That's interesting. I did a line by line rewrite of a large Django route to Quarkus and it was 10x faster, not using async or anything.
        acdha 1y ago
        That’s why I said “culture” - by all rights the JVM should win that competition. I wrote a bit more about the most recent one in a sibling comment but I’d summarize it as “the JVM can’t stop an enterprise Java developer”. https://news.ycombinator.com/item?id=44179589 https://news.ycombinator.com/item?id=44179589
        pjmlp 1y ago
        Enterprise developers and architects would be the same, regardless of the programming language. I am old enough to have seen enterprise C and C++ developers. Where do you think stuff like DCE, CORBA, DCOM has come from? Also many of the things people blame Java for, where born as Smalltalk, Objective-C and C++ frameworks before being re-written in Java. Since we are in a Apple discussion thread, here is some Objective-C ids from Apple frameworks, https://github.com/Quotation/LongestCocoa https://github.com/Quotation/LongestCocoa I also advise getting hold of the original WebObjects documentation in Objective-C, before its port to Java.
    - pjmlp 1y ago
      For a while means since around 2000, althought toolchains like Excelsior JET and Websphere Real Time, among others, were only available to companies that cared enough to pay for AOT compilers, and JIT caches. Nowadays to add to your comment, all major free beer implementations, OpenJDK, OpenJ9, GraalVM, and the ART cousin do AOT and JIT caches. Even without Valhala, there are quite a few tricks possible with Panama, one can manually create C like struct memory layouts. Yes it is a lot of boilerplate, however one can get around the boilerplate with AI (maybe), or just write the C declarations and point jextract to it.
  - cogman10 1y ago
    It all depends, but one major advantage of the way the JVM GCs is related memory will tend to be colocated. This is particularly true of the serial, parallel, and G1GC collectors. Let's say you have an object that looks like A -> B -> C. Even if the allocation of A/B/C happened at very temporally different times and inbetween different allocations, the next time the GC runs as it traverses the graph it will see and place in memory [A, B, C] assuming A is still live. That means even if the memory originally looks something like [A, D, B, Q, R, S, T, C] the act of collecting and compacting has a tendency to colocate.
    - username223 1y ago
      That's the theory -- a compacting collector will reduce fragmentation and can put linked objects next to each other. On the other hand, a reference count lives in the object, so you're likely using that cache line already when you change it. I don't know which of these is more important on a modern machine, and it probably depends upon the workload.
      - cogman10 1y ago
        The problem is memory colocation not RC management. But I agree, it'll likely be workload dependent. One major positive aspect of RC is the execution costs are very predictable. There's little external state which can negatively impact performance (like the GC currently running). The downside is fragmentation and the CPU time required for memory management. If you have an A -> B -> C chain where A is the only owner of the B and B is the only owner of C, then when A hits 0, it has to do 2 pointer hops to deallocate B and then deallocate C (plus arena management for the deallocs). One of the big benefits of JVM moving style collectors is that when A dies, the collector does not need to visit B or C to deallocate them. The collector only visits and moves live memory.
        dwaite 1y ago
        > The downside is fragmentation and the CPU time required for memory management. If you have an A -> B -> C chain where A is the only owner of the B and B is the only owner of C, then when A hits 0, it has to do 2 pointer hops to deallocate B and then deallocate C (plus arena management for the deallocs). I suspect this puts greater emphasis on functionality like value types and flexibility in compositionally creating objects. You can trend toward larger objects rather than nesting inner objects for functionality. For example, you can use tagged unions to represent optionality rather than pointers. The cost of deep A->B->C relationships in Java comes during collections, which still default to be halting. The difference is a reference counting GC will evaluate these chains while removing objects, while a reference tracking GC will evaluate live objects. So, garbage collection is expensive for ref-counting if you are creating large transient datasets, and is expensive for ref-tracking GC if you are retaining large datasets.
- geodel 1y ago
  That is just GC part. Another big difference is Reference type (Java) vs Value Type (Swift).
- mrighele 1y ago
  The 90% reduction doesn't necessarily have to be related only to GC. In my experience Java is a memory hog even compared to other garbage collected languages (that's my main gripe about the language). I think a good part of the reason is that if you exclude primitive types, almost everything in Java is an heap-allocated object and Java objects are fairly "fat": every single instance has an header of between 96 and 128 bit on 64-bit architectures [1]. That's... a lot. Just by making the headers smaller (the topic of the above link) you can get 20% decrease in heap usage and improvements in cpu and GC time [2]. My hope is that once value classes arrive [3][4], and libraries start to use them, we will see a substantial decrease in heap usage in an average java app. [1] https://openjdk.org/jeps/450 https://openjdk.org/jeps/450 [2] https://openjdk.org/jeps/519 https://openjdk.org/jeps/519 [3] https://openjdk.org/jeps/401 https://openjdk.org/jeps/401 [4] https://www.youtube.com/watch?v=Dhn-JgZaBWo https://www.youtube.com/watch?v=Dhn-JgZaBWo
  - formerly_proven 1y ago
    If Java's 128-bit object headers are already fairly fat, then what adjective applies to CPython's? [] is about a whole cache line. Trivial Python objects are barely smaller.
  - cogman10 1y ago
    The java GC approach is somewhat unique compared to other languages. There are multiple GCs and pretty much all of them are moving collectors. That means the JVM fairly rarely ends up freeing memory that it's claimed. A big spike will mean that it holds onto the spike's amount of memory. Many other GCed languages, such as swift, CPython, Go, do not use a moving collector. Instead, they allocate and pin memory and free it when not in use. The benefit to the JVM approach is heap allocations are wicked fast on pretty much all its collectors. Generally, to allocate it's a check to see if space is available and a pointer bump. For the other languages, you are bound to end up using a skiplist and/or arena allocator provided by your malloc implementation. Roughly O(log(n)) vs O(1) in performance terms. Don't get me wrong, the object header does eat a fair chunk of memory. Roughly double what another language will take. However, a lot of people confuse the memory which the JVM has claimed from the OS (and is thus reported by the OS) with the memory the JVM is actively using. 2 different things. It just so happens that for moving collectors like the JVM typically uses more reserved memory means fewer garbage collections and time spend garbage collecting.
loxs 1y ago
The java runtime is a beast. Even the fact that another runtime is just capable to do a similar thing is impressive, disregard the fact that it might be better. Even being on par makes it interesting for me to maybe try it on my own.
javanonymous 1y ago
The JVM tends to use as much memory as it can for performance reasons. It is not a reliable indicator of how much memory it actually needs. Why spend resources on clearing memory if there's still unused memory left? If memory is an issue, you can set a limit and the JVM will probably still work fine
- cellularmitosis 1y ago
  If it were such an easy problem to fix, don’t you think they would have done so rather than rewriting in Swift?
  - cogman10 1y ago
    Having been in development for a long time, no. Frankly, it just takes some motivated senior devs and the tantalizing ability to put out the OP blog post and you've got something management will sign off on. Bonus points you get to talk about how amazing it was to use Apple tech to get the job done. I don't think they seriously approached this because the article only mentioned tuning G1GC. The fact is, they should have been talking about ZGC, AppCDS, and probably Graal if pause times and startup times were really that big a problem for them. Heck, even CRaC should have been mentioned. It is not hard to get a JVM to startup in sub second time. Here's one framework where that's literally the glossy print on the front page. [1] [1] https://quarkus.io/ https://quarkus.io/
    - jamesfinlayson 1y ago
      Yep, resume-driven development. I remember at a previous company a small group of people pushed a Go rewrite to speed everything up. The serious speed improvements came from re-architecting (elimination of a heavy custom framework, using a message queue instead of handling requests synchronously etc). They would have been better off fixing the original system so that everything could benefit from the improvements, not just the tiny bits that they carved off. Then the next annual report talked about improved scalability because of this amazing technology from Google.

plorkyeran 1y ago

The typical rule of thumb is that getting good performance out of tracing GC requires doubling your memory usage, so a 90% reduction suggests that they made significant improvements on top of the language switch.

username223 1y ago
I remember the 2x rule from 20 years ago - do you know if things have changed? If locality is more important now, tracing GC might never be as performant as reference counting. Either you use 2x the memory and thrash your cache, or you use less and spend too much CPU time collecting.
- jeroenhd 1y ago
  Java has had AOT compilation for a while, so traditional GC and its massive overhead are no longer a strict necessity. Even AOT Java will probably stay behind Swift or any other natively compiled language in terms of memory usage, but it shouldn't be that drastic. As for performance and locality, Java's on-the-fly pointer reordering/compression can give it an edge over even some compiled languages in certain algorithms. Hard to say if that's relevant for whatever web framework Apple based their service on, but I wouldn't discount Java's locality optimisations just because it uses a GC.
  - username223 1y ago
    > Java has had AOT compilation for a while, so traditional GC and its massive overhead are no longer a strict necessity. You mean it does escape analysis and stack-allocates what it can? That would definitely help, but not eliminate the GC. Or are you thinking of something else? Thinking about it more, I remember that Java also has some performance-hostile design decisions baked in (e.g. almost everything's an Object, arrays aren't packed, dynamic dispatch everywhere). Swift doesn't have that legacy to deal with.
    - acdha 1y ago
      Java also has a lot of culture making optimization-resistant code so there’s the question of whether you’re talking about the language itself or various widespread libraries, especially if they’re old enough to have patterns designed around aesthetics or now-moot language limitations rather than performance. I’ve replaced Java code with Python a few times and each time even though we did it for maintenance (more Python devs available) we saw memory usage more than halved while performance at least doubled because the code used simpler functions and structures. Java has a far more advanced GC and JIT but at some point the weight of code and indirection wins out.
      - winrid 1y ago
        That's interesting. I did a line by line rewrite of a large Django route to Quarkus and it was 10x faster, not using async or anything.
        acdha 1y ago
        That’s why I said “culture” - by all rights the JVM should win that competition. I wrote a bit more about the most recent one in a sibling comment but I’d summarize it as “the JVM can’t stop an enterprise Java developer”. https://news.ycombinator.com/item?id=44179589 https://news.ycombinator.com/item?id=44179589
        pjmlp 1y ago
        Enterprise developers and architects would be the same, regardless of the programming language. I am old enough to have seen enterprise C and C++ developers. Where do you think stuff like DCE, CORBA, DCOM has come from? Also many of the things people blame Java for, where born as Smalltalk, Objective-C and C++ frameworks before being re-written in Java. Since we are in a Apple discussion thread, here is some Objective-C ids from Apple frameworks, https://github.com/Quotation/LongestCocoa https://github.com/Quotation/LongestCocoa I also advise getting hold of the original WebObjects documentation in Objective-C, before its port to Java.
  - pjmlp 1y ago
    For a while means since around 2000, althought toolchains like Excelsior JET and Websphere Real Time, among others, were only available to companies that cared enough to pay for AOT compilers, and JIT caches. Nowadays to add to your comment, all major free beer implementations, OpenJDK, OpenJ9, GraalVM, and the ART cousin do AOT and JIT caches. Even without Valhala, there are quite a few tricks possible with Panama, one can manually create C like struct memory layouts. Yes it is a lot of boilerplate, however one can get around the boilerplate with AI (maybe), or just write the C declarations and point jextract to it.
- cogman10 1y ago
  It all depends, but one major advantage of the way the JVM GCs is related memory will tend to be colocated. This is particularly true of the serial, parallel, and G1GC collectors. Let's say you have an object that looks like A -> B -> C. Even if the allocation of A/B/C happened at very temporally different times and inbetween different allocations, the next time the GC runs as it traverses the graph it will see and place in memory [A, B, C] assuming A is still live. That means even if the memory originally looks something like [A, D, B, Q, R, S, T, C] the act of collecting and compacting has a tendency to colocate.
  - username223 1y ago
    That's the theory -- a compacting collector will reduce fragmentation and can put linked objects next to each other. On the other hand, a reference count lives in the object, so you're likely using that cache line already when you change it. I don't know which of these is more important on a modern machine, and it probably depends upon the workload.
    - cogman10 1y ago
      The problem is memory colocation not RC management. But I agree, it'll likely be workload dependent. One major positive aspect of RC is the execution costs are very predictable. There's little external state which can negatively impact performance (like the GC currently running). The downside is fragmentation and the CPU time required for memory management. If you have an A -> B -> C chain where A is the only owner of the B and B is the only owner of C, then when A hits 0, it has to do 2 pointer hops to deallocate B and then deallocate C (plus arena management for the deallocs). One of the big benefits of JVM moving style collectors is that when A dies, the collector does not need to visit B or C to deallocate them. The collector only visits and moves live memory.
      - dwaite 1y ago
        > The downside is fragmentation and the CPU time required for memory management. If you have an A -> B -> C chain where A is the only owner of the B and B is the only owner of C, then when A hits 0, it has to do 2 pointer hops to deallocate B and then deallocate C (plus arena management for the deallocs). I suspect this puts greater emphasis on functionality like value types and flexibility in compositionally creating objects. You can trend toward larger objects rather than nesting inner objects for functionality. For example, you can use tagged unions to represent optionality rather than pointers. The cost of deep A->B->C relationships in Java comes during collections, which still default to be halting. The difference is a reference counting GC will evaluate these chains while removing objects, while a reference tracking GC will evaluate live objects. So, garbage collection is expensive for ref-counting if you are creating large transient datasets, and is expensive for ref-tracking GC if you are retaining large datasets.
geodel 1y ago
That is just GC part. Another big difference is Reference type (Java) vs Value Type (Swift).
mrighele 1y ago
The 90% reduction doesn't necessarily have to be related only to GC. In my experience Java is a memory hog even compared to other garbage collected languages (that's my main gripe about the language). I think a good part of the reason is that if you exclude primitive types, almost everything in Java is an heap-allocated object and Java objects are fairly "fat": every single instance has an header of between 96 and 128 bit on 64-bit architectures [1]. That's... a lot. Just by making the headers smaller (the topic of the above link) you can get 20% decrease in heap usage and improvements in cpu and GC time [2]. My hope is that once value classes arrive [3][4], and libraries start to use them, we will see a substantial decrease in heap usage in an average java app. [1] https://openjdk.org/jeps/450 https://openjdk.org/jeps/450 [2] https://openjdk.org/jeps/519 https://openjdk.org/jeps/519 [3] https://openjdk.org/jeps/401 https://openjdk.org/jeps/401 [4] https://www.youtube.com/watch?v=Dhn-JgZaBWo https://www.youtube.com/watch?v=Dhn-JgZaBWo
- formerly_proven 1y ago
  If Java's 128-bit object headers are already fairly fat, then what adjective applies to CPython's? [] is about a whole cache line. Trivial Python objects are barely smaller.
- cogman10 1y ago
  The java GC approach is somewhat unique compared to other languages. There are multiple GCs and pretty much all of them are moving collectors. That means the JVM fairly rarely ends up freeing memory that it's claimed. A big spike will mean that it holds onto the spike's amount of memory. Many other GCed languages, such as swift, CPython, Go, do not use a moving collector. Instead, they allocate and pin memory and free it when not in use. The benefit to the JVM approach is heap allocations are wicked fast on pretty much all its collectors. Generally, to allocate it's a check to see if space is available and a pointer bump. For the other languages, you are bound to end up using a skiplist and/or arena allocator provided by your malloc implementation. Roughly O(log(n)) vs O(1) in performance terms. Don't get me wrong, the object header does eat a fair chunk of memory. Roughly double what another language will take. However, a lot of people confuse the memory which the JVM has claimed from the OS (and is thus reported by the OS) with the memory the JVM is actively using. 2 different things. It just so happens that for moving collectors like the JVM typically uses more reserved memory means fewer garbage collections and time spend garbage collecting.

loxs 1y ago

The java runtime is a beast. Even the fact that another runtime is just capable to do a similar thing is impressive, disregard the fact that it might be better. Even being on par makes it interesting for me to maybe try it on my own.

javanonymous 1y ago

The JVM tends to use as much memory as it can for performance reasons. It is not a reliable indicator of how much memory it actually needs. Why spend resources on clearing memory if there's still unused memory left? If memory is an issue, you can set a limit and the JVM will probably still work fine

cellularmitosis 1y ago
If it were such an easy problem to fix, don’t you think they would have done so rather than rewriting in Swift?
- cogman10 1y ago
  Having been in development for a long time, no. Frankly, it just takes some motivated senior devs and the tantalizing ability to put out the OP blog post and you've got something management will sign off on. Bonus points you get to talk about how amazing it was to use Apple tech to get the job done. I don't think they seriously approached this because the article only mentioned tuning G1GC. The fact is, they should have been talking about ZGC, AppCDS, and probably Graal if pause times and startup times were really that big a problem for them. Heck, even CRaC should have been mentioned. It is not hard to get a JVM to startup in sub second time. Here's one framework where that's literally the glossy print on the front page. [1] [1] https://quarkus.io/ https://quarkus.io/
  - jamesfinlayson 1y ago
    Yep, resume-driven development. I remember at a previous company a small group of people pushed a Go rewrite to speed everything up. The serious speed improvements came from re-architecting (elimination of a heavy custom framework, using a message queue instead of handling requests synchronously etc). They would have been better off fixing the original system so that everything could benefit from the improvements, not just the tiny bits that they carved off. Then the next annual report talked about improved scalability because of this amazing technology from Google.