5 ms·

I've installed the application, it needs 840 MB of the memory. Firefox with more than 100 tabs opend consumes 1.2 GB, IntelliJ with a huge java project consumes

I've installed the application, it needs 840 MB of the memory. Firefox with more than 100 tabs opend consumes 1.2 GB, IntelliJ with a huge java project consumes 1.6 GB. I understand that this is proof of concept application, but what eats so much RAM? Is that the data that was read or the framework itself?

fnord123 7y ago
Clojure.
- AA-BA-94-2A-56 7y ago
  Case clojed.
  - vlaaad 7y ago
    Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html I explained the situation in a reply to grandparent comment if you are interested to know why it is like that.
    - fnord123 7y ago
      >Please don't post shallow dismissals, especially of other people's work. Nothing's been dismissed. The post is about a throw away program in a nascent React-on-JVM environment. People want to know why it uses so much memory. It's Clojure. Clojure is a memory hog. Case clojed. Please don't be disheartened. It's a nice show and tell and resulting discussion.
vlaaad 7y ago
It does not need 840 MB of memory — it uses it because the purpose of this example is to show that packaging is now easy, and non-essential parts were left out to keep it simple. I just tried limiting it to 50 MB and it runs just fine.
- fnord123 7y ago
  If I put -Xmx50m in the JavaOptions section of /opt/cljfx-hn/lib/app/cljfx-hn.cfg it still uses 650MB. How do you go about limiting it to 50MB?
- klibertp 7y ago
  I'm sorry to say this, but I think you made the wrong call here by treating startup time and memory usage as "non-essential". At least when presenting the case here, on HN. Programmers here tend to be experienced in many different languages and platforms, and when we see a post about packaging an app, we stop and ask what is it that gets packaged first. In other words, what's the point of easy packaging, if the packaged result is slow as molasses and eats RAM like there's no tomorrow. We have literally dozens of tools at our disposal which make packaging as simple, yet the result would runs circles around your demo, even if - and here's the kicker - we considered performance and memory optimization as non-essential. Sane defaults and all that... To sum it up: we're just as interested in how you package it as in what is in the package. If it's really easy to optimize for startup time and memory footprint as you say, then all the more reason to apply these to the demo. No one here would think the demo is too complex (not simple) just because of that. And if it's not easy... well, that's also something we'd like to know.
thu2111 7y ago
It doesn't really need 840mb. If you force a GC after waiting for it to load the stories it needs "only" 350mb. Try "jcmd main GC.run" (if you have any JDK 11+ installed). The problem is Java sees lots of memory lying unused and figures ... why should I work harder to free it up? It'll waste CPU time and energy and I might need the memory later. In theory it actually will do background GCs and free memory back to the OS anyway so I'm not sure why it's not happening here; possibly that feature needs to be explicitly activated or maybe I just didn't wait long enough to see it happen. As for why it needs 350mb of RAM, let's take a quick look: $ jcmd main GC.heap_info Tue Mar 31 10:53:42 2020 61074: garbage-first heap total 102400K, used 28139K [0x0000000700000000, 0x0000000800000000) region size 1024K, 0 young (0K), 0 survivors (0K) Metaspace used 58206K, capacity 65566K, committed 65868K, reserved 1101824K class space used 10342K, capacity 12810K, committed 12928K, reserved 1048576K So as a first approximation, even after a GC and subsequent heap shrink, it's holding onto about 3x what it really needs. Total heap memory is only 28mb. That's a bit odd. Perhaps G1 is still not aggressive enough in releasing memory back to the OS. For desktop apps it'd really be better to aggressively let go of heap as fast as possible (NB: native apps don't always do this either). The "metaspace" is holding a lot of stuff, mostly VM internal data structures. But I thought metaspace was primarily class metadata, but here there's only 10mb of that, so what's filling it up? There's "jcmd main VM.metaspace" which tells us about 8mb of it is wasted due to fragmentation (I think). It doesn't tell us much about what's going on though. These numbers look extremely high for a normal Java app. If I recall correctly Clojure [ab]uses the JVM in odd ways, for instance, by creating a class for every single function in the program instead of mapping them to methods as you would expect. This can easily cause a lot of overhead in class metadata compared to a normal program. However the bulk of the memory usage you see is really just an "optimisation" - it's there, so why not be lazy and leave garbage lying around until you need the space.
fctorial 7y ago
> Firefox with more than 100 tabs opend consumes 1.2 GB, Really? On my machine it eats 2gb for 10 tabs.
- capableweb 7y ago
  Just saying 100 tabs = 1.2 GB or 10 tabs = 2 GB doesn't make much sense as it depends on a lot more things than just the count of the tabs. You'll need to also outline what pages are in those tabs, how much each tab consumes, what extensions you have installed, how your config looks, what your OS is, how much history you have, what is your usage pattern, what experimental flags have you/the system activated and more. It just depends on so many variables. 10 tabs of CNN is gonna use way more memory than 100 tabs of Hacker News comments, as a basic example. But you have a content script that is injected on every HN page with live feeds of something, now 1 HN tab might use more memory than 10 tabs of CNN.
- klibertp 7y ago
  Yes, well, if they're just "opened" and not fully "loaded", then maybe it can get away with 1.2Gb. ~100 fully loaded tabs, which happens quite often with various online docs, take 24Gb RAM last time I checked. So your figure (2gb for 10) looks quite right to me. Well, it was worse with Chrome, though not by much: the main memory savings come from Multi-Account Containers, which is a really neat feature, but otherwise they seem almost tied.
imtringued 7y ago
It doesn't actually need 840MB of RAM. This is an inherent limitation of the way the JVM works. It will reserve a configured amount of memory upfront and grow the heap if memory is full until it reaches the configured maximum heap size. One especially annoying thing is that the JVM will not garbage collect until the heap is actually full. This means if you allocate a little bit here and there (like every single Java program in existence) over time you are going to reach the maximum configured heap. Once the JVM has allocated heap space it will never let go of it. Therefore you might as well assume that the minimum amount of memory a JVM process consumes is the maximum that it has been configured for. In this case the maximum was probably set to 840MB. The application itself probably only uses something like 50MB. Remember those stupid arguments that a competent programmer can write well performing code [0] in any language? Here is the thing... You actually can't. Especially not with the JVM. Let's say you use a super efficient framework that is purely optimized for startup performance and low memory like Micronaut or Quarkus. When you look at your metrics you see that the Java program is consuming 10MB of heap space and pat yourself smugly on the shoulder thinking how well written these frameworks are. Tough luck. Even if you configure the heap size to something absurdly low like 32MB your JVM process will still use 120MB. Why? Because the JVM has a minimum footprint. It has lots of parked GC threads in the background that need 2MB stacks. It has to keep the class files in memory, keep profiling data, the generated code and lots of other things. If I need something that needs to consume as little memory as possible I just stick with C++ or Lua. With those two languages staying under 1 - 2 MB of RAM is trivial. Also... it is possible your coworkers do not even set a maximum heap and just let it stay on the default values. That's how a tiny little CRUD app as a dashboard ended up allocating 10GB of RAM to itself on a server with 128GB RAM. [0] well performing = low memory usage in this case
- mrighele 7y ago
  > Once the JVM has allocated heap space it will never let go of it. While this used to be true in the past, some of the GCs available to the JVM may now uncommit heap memory. This is for sure the case with ZGC [1], G1 [2] and I think also with Shenandoah [3] [1] https://openjdk.java.net/jeps/351 https://openjdk.java.net/jeps/351 [2] https://openjdk.java.net/jeps/346 https://openjdk.java.net/jeps/346 [3] https://shipilev.net/jvm/anatomy-quarks/21-heap-uncommit/ https://shipilev.net/jvm/anatomy-quarks/21-heap-uncommit/

fnord123 7y ago

Clojure.

AA-BA-94-2A-56 7y ago
Case clojed.
- vlaaad 7y ago
  Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html I explained the situation in a reply to grandparent comment if you are interested to know why it is like that.
  - fnord123 7y ago
    >Please don't post shallow dismissals, especially of other people's work. Nothing's been dismissed. The post is about a throw away program in a nascent React-on-JVM environment. People want to know why it uses so much memory. It's Clojure. Clojure is a memory hog. Case clojed. Please don't be disheartened. It's a nice show and tell and resulting discussion.

vlaaad 7y ago

It does not need 840 MB of memory — it uses it because the purpose of this example is to show that packaging is now easy, and non-essential parts were left out to keep it simple. I just tried limiting it to 50 MB and it runs just fine.

fnord123 7y ago
If I put -Xmx50m in the JavaOptions section of /opt/cljfx-hn/lib/app/cljfx-hn.cfg it still uses 650MB. How do you go about limiting it to 50MB?
klibertp 7y ago
I'm sorry to say this, but I think you made the wrong call here by treating startup time and memory usage as "non-essential". At least when presenting the case here, on HN. Programmers here tend to be experienced in many different languages and platforms, and when we see a post about packaging an app, we stop and ask what is it that gets packaged first. In other words, what's the point of easy packaging, if the packaged result is slow as molasses and eats RAM like there's no tomorrow. We have literally dozens of tools at our disposal which make packaging as simple, yet the result would runs circles around your demo, even if - and here's the kicker - we considered performance and memory optimization as non-essential. Sane defaults and all that... To sum it up: we're just as interested in how you package it as in what is in the package. If it's really easy to optimize for startup time and memory footprint as you say, then all the more reason to apply these to the demo. No one here would think the demo is too complex (not simple) just because of that. And if it's not easy... well, that's also something we'd like to know.

thu2111 7y ago

It doesn't really need 840mb. If you force a GC after waiting for it to load the stories it needs "only" 350mb. Try "jcmd main GC.run" (if you have any JDK 11+ installed). The problem is Java sees lots of memory lying unused and figures ... why should I work harder to free it up? It'll waste CPU time and energy and I might need the memory later. In theory it actually will do background GCs and free memory back to the OS anyway so I'm not sure why it's not happening here; possibly that feature needs to be explicitly activated or maybe I just didn't wait long enough to see it happen. As for why it needs 350mb of RAM, let's take a quick look: $ jcmd main GC.heap_info Tue Mar 31 10:53:42 2020 61074: garbage-first heap total 102400K, used 28139K [0x0000000700000000, 0x0000000800000000) region size 1024K, 0 young (0K), 0 survivors (0K) Metaspace used 58206K, capacity 65566K, committed 65868K, reserved 1101824K class space used 10342K, capacity 12810K, committed 12928K, reserved 1048576K So as a first approximation, even after a GC and subsequent heap shrink, it's holding onto about 3x what it really needs. Total heap memory is only 28mb. That's a bit odd. Perhaps G1 is still not aggressive enough in releasing memory back to the OS. For desktop apps it'd really be better to aggressively let go of heap as fast as possible (NB: native apps don't always do this either). The "metaspace" is holding a lot of stuff, mostly VM internal data structures. But I thought metaspace was primarily class metadata, but here there's only 10mb of that, so what's filling it up? There's "jcmd main VM.metaspace" which tells us about 8mb of it is wasted due to fragmentation (I think). It doesn't tell us much about what's going on though. These numbers look extremely high for a normal Java app. If I recall correctly Clojure [ab]uses the JVM in odd ways, for instance, by creating a class for every single function in the program instead of mapping them to methods as you would expect. This can easily cause a lot of overhead in class metadata compared to a normal program. However the bulk of the memory usage you see is really just an "optimisation" - it's there, so why not be lazy and leave garbage lying around until you need the space.

fctorial 7y ago

> Firefox with more than 100 tabs opend consumes 1.2 GB, Really? On my machine it eats 2gb for 10 tabs.

capableweb 7y ago
Just saying 100 tabs = 1.2 GB or 10 tabs = 2 GB doesn't make much sense as it depends on a lot more things than just the count of the tabs. You'll need to also outline what pages are in those tabs, how much each tab consumes, what extensions you have installed, how your config looks, what your OS is, how much history you have, what is your usage pattern, what experimental flags have you/the system activated and more. It just depends on so many variables. 10 tabs of CNN is gonna use way more memory than 100 tabs of Hacker News comments, as a basic example. But you have a content script that is injected on every HN page with live feeds of something, now 1 HN tab might use more memory than 10 tabs of CNN.
klibertp 7y ago
Yes, well, if they're just "opened" and not fully "loaded", then maybe it can get away with 1.2Gb. ~100 fully loaded tabs, which happens quite often with various online docs, take 24Gb RAM last time I checked. So your figure (2gb for 10) looks quite right to me. Well, it was worse with Chrome, though not by much: the main memory savings come from Multi-Account Containers, which is a really neat feature, but otherwise they seem almost tied.

imtringued 7y ago

It doesn't actually need 840MB of RAM. This is an inherent limitation of the way the JVM works. It will reserve a configured amount of memory upfront and grow the heap if memory is full until it reaches the configured maximum heap size. One especially annoying thing is that the JVM will not garbage collect until the heap is actually full. This means if you allocate a little bit here and there (like every single Java program in existence) over time you are going to reach the maximum configured heap. Once the JVM has allocated heap space it will never let go of it. Therefore you might as well assume that the minimum amount of memory a JVM process consumes is the maximum that it has been configured for. In this case the maximum was probably set to 840MB. The application itself probably only uses something like 50MB. Remember those stupid arguments that a competent programmer can write well performing code [0] in any language? Here is the thing... You actually can't. Especially not with the JVM. Let's say you use a super efficient framework that is purely optimized for startup performance and low memory like Micronaut or Quarkus. When you look at your metrics you see that the Java program is consuming 10MB of heap space and pat yourself smugly on the shoulder thinking how well written these frameworks are. Tough luck. Even if you configure the heap size to something absurdly low like 32MB your JVM process will still use 120MB. Why? Because the JVM has a minimum footprint. It has lots of parked GC threads in the background that need 2MB stacks. It has to keep the class files in memory, keep profiling data, the generated code and lots of other things. If I need something that needs to consume as little memory as possible I just stick with C++ or Lua. With those two languages staying under 1 - 2 MB of RAM is trivial. Also... it is possible your coworkers do not even set a maximum heap and just let it stay on the default values. That's how a tiny little CRUD app as a dashboard ended up allocating 10GB of RAM to itself on a server with 128GB RAM. [0] well performing = low memory usage in this case

mrighele 7y ago
> Once the JVM has allocated heap space it will never let go of it. While this used to be true in the past, some of the GCs available to the JVM may now uncommit heap memory. This is for sure the case with ZGC [1], G1 [2] and I think also with Shenandoah [3] [1] https://openjdk.java.net/jeps/351 https://openjdk.java.net/jeps/351 [2] https://openjdk.java.net/jeps/346 https://openjdk.java.net/jeps/346 [3] https://shipilev.net/jvm/anatomy-quarks/21-heap-uncommit/ https://shipilev.net/jvm/anatomy-quarks/21-heap-uncommit/