4 ms·

> I’ll never understand the logic behind “high memory usage” as a metric for desktop environments.. If you had the hardware, would you rather see it used or see

by senko 4y ago

> I’ll never understand the logic behind “high memory usage” as a metric for desktop environments.. If you had the hardware, would you rather see it used or see it dog slow trying to load UI components from disk? Linux already uses all the availabe RAM for disk cache. I would rather not an app do that (except in special cases like databases, which know better than the underlying file system what needs to be cached). If a desktop environment is using a lot of memory, this means the same memory is not available to my other apps. So yeah, even on a 32GB machine, I do care about what each app uses. As I type this, on my 32GB GNOME system, 21GB is currently in use. Of those gnome-shell uses 2,1GB + another 700MB for gjs and zoom (which I haven't used for a couple days, it just sits in my tray) uses 1,4GB. That's 12% for doing virtually nothing (there are window managers that easily take 5% of the resources of gnome shell and one of these days I'll be fed up enough to switch; and zoom does literally nothing right now except hog memory), and that's just looking at the two top memory users.

eptcyka 4y ago
Whilst some components inherently use more memory than other viable alternatives (gnome-shell's JS runtime), gnome-shell generally has a lot more components and thus features than the more barebones window managers - and this should always be pointed out. Of course, if you are certain you will never need the extra functionality, then don't use Gnome. But generally, I don't think it makes sense to compare plain window managers with desktop environments. I think an interesting comparison would also be looking at quartz, the macOS window manager/desktop environment.
- senko 4y ago
  You can use Gnome with other window managers. I was talking specifically about gnome-shell.
- dman 4y ago
  xfce isnt just a window manager.
- inetknght 4y ago
  > gnome-shell generally has a lot more components and thus features than the more barebones window managers My experience with gnome-shell is that it is in fact a huge massive bloat for what it provides. What features does it provide that more barebones window managers don't provide?
avar 4y ago
> If a desktop environment is using a lot > of memory, this means the same memory is > not available to my other apps. No, it doesn't mean that. Your mental model of a how an OS allocates and manages memory is stuck in the 60s. Nowadays getting a handle on "real" memory use is increasingly illusive. Probably the best metric is to ignore it altogether, and instead look at memory churn. I.e. how much the OS ends up having to evict some pages from memory to make space for others. Even that is only meaningful if it's happening on an ongoing basis. I.e. if your DE allocates 1GB that it uses once (or never) having to evict those pages once generally isn't a problem. It's only a problem once there isn't enough memory to go around for those pages that are in active use.
- kohlerm 4y ago
  For a developer doing software builds frequently, the amount of available memory for caching disk access (buffer) can very much make a difference. The more the better.
- senko 4y ago
  > No, it doesn't mean that. [...] It's only a problem once there isn't enough memory to go around for those pages that are in active use. So it does mean that. The fact that I said it plainly instead of mentioning private dirty pages gobbling up most of the available ram so any executable readonly pages need to be evicted / read all the time, thrashing the system much earlier than the OOM killer can kick in, doesn't mean that "if apps take up lots of memory, other apps have less" statement is untrue.
  - avar 4y ago
    It's unclear exactly what you mean by the upthread comment, i.e. how "using a lot of memory" maps to e.g. Linux smaps statistics. But you do mention e.g. gnome-shell taking ~2GB. My own gnome-shell has e.g. a 100MB allocation where >95MB is in "Private_Dirty", then a ~200MB allocation where all the columns except "Size" in the "/proc/$(pidof gnome-shell)/smaps" are zero. People typically exclude the latter category when talking about "real" memory use. But the former is the sort of thing that would make it into "real" memory use by most definitions, and show up in "RSS". What I'm pointing out is that knowing that doesn't tell you anything about whether the memory use contributes to memory contention, which is the interesting question. It's entirely possible (and quite common) that most/all of that was used as a one-off, and is either sitting there unused, or would be swapped out as part of expunging stale pages to disk. So, you could have a process that at one point used 2GB of memory, and still hasn't free()'d it, but for the purposes of needing to fit a 7GB browser process into a total of 8GB of RAM won't contribute to contention. Even though it "uses" 2GB still (as in "RSS" etc.) it's only ever accessing 100MB of those 2GB. The kernel will happily page out 1.9GB to swap, and performance will be mostly unaffected. To make any claim about whether an application uses a lot of memory on a modern OS you need to know a lot about its lifetime management of data.
amelius 4y ago
> If a desktop environment is using a lot of memory, this means the same memory is not available to my other apps. How does that work if you run 2 virtual machines, each running Linux?
- dijit 4y ago
  Whatever you have allocated to those VMs is not available for the host at all. QEMU-KVM will allocate "2GiB" of memory if the VM has 2GiB available.. Even if the linux guest is only using 2% of that actively.
  - bayindirh 4y ago
    That's not correct. With a VirtIO ballooning device [0], you can create faux memory allocations on the VMs and reclaim that allocated memory space back to host. This will increase the memory pressure on the guest, but will provide extra memory to the host. [0]: https://pmhahn.github.io/virtio-balloon/ https://pmhahn.github.io/virtio-balloon/
    - dijit 4y ago
      That's not exactly how it works. a virtio balloon is intended to be used in "near OOM conditions" (due to performance implications, mostly) a normal VM is not doing this. it's also true that even if it was opportunistic: qemu's memory allocation on the host would take precedence over host filesystem caches which would be consequently evicted. The reason people really like containers at the moment is partially due to this memory situation, you can "fit" more containers on a host due to better fitting of the memory.
      - bayindirh 4y ago
        > a virtio balloon is intended to be used in "near OOM conditions" Of course. This is why you need to plan your VM deployments on your system carefully, even if it's just a desktop system. > it's also true that even if it was opportunistic: qemu's memory allocation on the host would take precedence over host filesystem caches which would be consequently evicted. A filesystem cache on the RAM is one of the most opportunistic mechanisms in Linux, and AFAICS, the buffers are the least important allocations on RAM, since they can be re-accessed from disk when necessary. So, the ballooning device enters at very high memory pressure scenarios with almost full swap area. The truth is, you shouldn't be nearing to this point at any of your systems, running VMs or not. Lastly, a bare bones Linux system w/o a graphical interface needs comparable memory to a biggish daemon, hence when used smartly, they're invisible sans the context switching load they induce on the system.
StillBored 4y ago
<i> 700MB for gjs </i> Well, i'm obviously in the minority (despite this kind of system programming being my day job for the past 30 years), but JIT'ed Garbage collected languages should never be part of the core system toolkit for actual technical reasons. gjs (and the kde equivilant) should have been kept as end user system scripting languages, rather than having shipped parts of the DE written in them. Its a large reason why lxqt is significantly lighter than KDE. It all comes down to the fact that no one has really solved the triumvirate of low latency, reasonable performance, and high collection rates, with a GC that aggressively returns unused ram back to the OS. Its a hard problem when the end application isn't known. For something like a generic control panel the (excluding event/loggging) the upper memory footprint can be bounded by the programmer, and the GC will be more aggressive as it approaches that limit, and the final result is a reasonable compromise. That said the hardcoded limit is likely 2X+ what would have been required by a non GC'ed environment. But for a general anything goes environment, the GC's are tuned towards performance and largely unbounded GC regions. Which means in order to maintain reasonable performance they will just keep reserving address space from the OS, touching parts of it temporary and then avoiding compacting it because those operations are very expensive. Plus with mozjs/spidermonkey (another whole set of problems for gjs due to the lack of a stable API) if the GC+JIT are aggressive about memory minimization then there can be frequent noticeable application lags as the memory is freed then reacquired from the OS/etc. Then because your desktop isn't under memory pressure, its not going to page out the fragmented bits of address space by default. Until it is under pressure, then the whole system goes laggy because while the LRU like algorithms the kernel is using to take the memory back work well, the GC is completely unaware of which bits of the address space it thinks it owns are suddenly going to respond much more slowly than it expects, and since its shotgunned (or worse is actually heap compressing everything) stuff all over the address space your taking a lot more pagein requests than one might expect. So the problem gets compounded. The end result (like with java on the server) is that what benchmarks well in a trivial sandbox by itself on a machine, starts to have longer term negative consequences when its sharing a machine with another application (or dozen) also written under the assumption that address space is free and the OS will just take care of the problem. So, javascript DE bindings are a good idea, but should have been reserved for windows macro recorder, or vba types of use cases rather than having parts of the default installed runtime env rewritten to use it.

by senko 4y ago

eptcyka 4y ago

Whilst some components inherently use more memory than other viable alternatives (gnome-shell's JS runtime), gnome-shell generally has a lot more components and thus features than the more barebones window managers - and this should always be pointed out. Of course, if you are certain you will never need the extra functionality, then don't use Gnome. But generally, I don't think it makes sense to compare plain window managers with desktop environments. I think an interesting comparison would also be looking at quartz, the macOS window manager/desktop environment.

senko 4y ago
You can use Gnome with other window managers. I was talking specifically about gnome-shell.
dman 4y ago
xfce isnt just a window manager.
inetknght 4y ago
> gnome-shell generally has a lot more components and thus features than the more barebones window managers My experience with gnome-shell is that it is in fact a huge massive bloat for what it provides. What features does it provide that more barebones window managers don't provide?

avar 4y ago

> If a desktop environment is using a lot > of memory, this means the same memory is > not available to my other apps. No, it doesn't mean that. Your mental model of a how an OS allocates and manages memory is stuck in the 60s. Nowadays getting a handle on "real" memory use is increasingly illusive. Probably the best metric is to ignore it altogether, and instead look at memory churn. I.e. how much the OS ends up having to evict some pages from memory to make space for others. Even that is only meaningful if it's happening on an ongoing basis. I.e. if your DE allocates 1GB that it uses once (or never) having to evict those pages once generally isn't a problem. It's only a problem once there isn't enough memory to go around for those pages that are in active use.

kohlerm 4y ago
For a developer doing software builds frequently, the amount of available memory for caching disk access (buffer) can very much make a difference. The more the better.
senko 4y ago
> No, it doesn't mean that. [...] It's only a problem once there isn't enough memory to go around for those pages that are in active use. So it does mean that. The fact that I said it plainly instead of mentioning private dirty pages gobbling up most of the available ram so any executable readonly pages need to be evicted / read all the time, thrashing the system much earlier than the OOM killer can kick in, doesn't mean that "if apps take up lots of memory, other apps have less" statement is untrue.
- avar 4y ago
  It's unclear exactly what you mean by the upthread comment, i.e. how "using a lot of memory" maps to e.g. Linux smaps statistics. But you do mention e.g. gnome-shell taking ~2GB. My own gnome-shell has e.g. a 100MB allocation where >95MB is in "Private_Dirty", then a ~200MB allocation where all the columns except "Size" in the "/proc/$(pidof gnome-shell)/smaps" are zero. People typically exclude the latter category when talking about "real" memory use. But the former is the sort of thing that would make it into "real" memory use by most definitions, and show up in "RSS". What I'm pointing out is that knowing that doesn't tell you anything about whether the memory use contributes to memory contention, which is the interesting question. It's entirely possible (and quite common) that most/all of that was used as a one-off, and is either sitting there unused, or would be swapped out as part of expunging stale pages to disk. So, you could have a process that at one point used 2GB of memory, and still hasn't free()'d it, but for the purposes of needing to fit a 7GB browser process into a total of 8GB of RAM won't contribute to contention. Even though it "uses" 2GB still (as in "RSS" etc.) it's only ever accessing 100MB of those 2GB. The kernel will happily page out 1.9GB to swap, and performance will be mostly unaffected. To make any claim about whether an application uses a lot of memory on a modern OS you need to know a lot about its lifetime management of data.

amelius 4y ago

> If a desktop environment is using a lot of memory, this means the same memory is not available to my other apps. How does that work if you run 2 virtual machines, each running Linux?

dijit 4y ago
Whatever you have allocated to those VMs is not available for the host at all. QEMU-KVM will allocate "2GiB" of memory if the VM has 2GiB available.. Even if the linux guest is only using 2% of that actively.
- bayindirh 4y ago
  That's not correct. With a VirtIO ballooning device [0], you can create faux memory allocations on the VMs and reclaim that allocated memory space back to host. This will increase the memory pressure on the guest, but will provide extra memory to the host. [0]: https://pmhahn.github.io/virtio-balloon/ https://pmhahn.github.io/virtio-balloon/
  - dijit 4y ago
    That's not exactly how it works. a virtio balloon is intended to be used in "near OOM conditions" (due to performance implications, mostly) a normal VM is not doing this. it's also true that even if it was opportunistic: qemu's memory allocation on the host would take precedence over host filesystem caches which would be consequently evicted. The reason people really like containers at the moment is partially due to this memory situation, you can "fit" more containers on a host due to better fitting of the memory.
    - bayindirh 4y ago
      > a virtio balloon is intended to be used in "near OOM conditions" Of course. This is why you need to plan your VM deployments on your system carefully, even if it's just a desktop system. > it's also true that even if it was opportunistic: qemu's memory allocation on the host would take precedence over host filesystem caches which would be consequently evicted. A filesystem cache on the RAM is one of the most opportunistic mechanisms in Linux, and AFAICS, the buffers are the least important allocations on RAM, since they can be re-accessed from disk when necessary. So, the ballooning device enters at very high memory pressure scenarios with almost full swap area. The truth is, you shouldn't be nearing to this point at any of your systems, running VMs or not. Lastly, a bare bones Linux system w/o a graphical interface needs comparable memory to a biggish daemon, hence when used smartly, they're invisible sans the context switching load they induce on the system.

StillBored 4y ago

<i> 700MB for gjs </i> Well, i'm obviously in the minority (despite this kind of system programming being my day job for the past 30 years), but JIT'ed Garbage collected languages should never be part of the core system toolkit for actual technical reasons. gjs (and the kde equivilant) should have been kept as end user system scripting languages, rather than having shipped parts of the DE written in them. Its a large reason why lxqt is significantly lighter than KDE. It all comes down to the fact that no one has really solved the triumvirate of low latency, reasonable performance, and high collection rates, with a GC that aggressively returns unused ram back to the OS. Its a hard problem when the end application isn't known. For something like a generic control panel the (excluding event/loggging) the upper memory footprint can be bounded by the programmer, and the GC will be more aggressive as it approaches that limit, and the final result is a reasonable compromise. That said the hardcoded limit is likely 2X+ what would have been required by a non GC'ed environment. But for a general anything goes environment, the GC's are tuned towards performance and largely unbounded GC regions. Which means in order to maintain reasonable performance they will just keep reserving address space from the OS, touching parts of it temporary and then avoiding compacting it because those operations are very expensive. Plus with mozjs/spidermonkey (another whole set of problems for gjs due to the lack of a stable API) if the GC+JIT are aggressive about memory minimization then there can be frequent noticeable application lags as the memory is freed then reacquired from the OS/etc. Then because your desktop isn't under memory pressure, its not going to page out the fragmented bits of address space by default. Until it is under pressure, then the whole system goes laggy because while the LRU like algorithms the kernel is using to take the memory back work well, the GC is completely unaware of which bits of the address space it thinks it owns are suddenly going to respond much more slowly than it expects, and since its shotgunned (or worse is actually heap compressing everything) stuff all over the address space your taking a lot more pagein requests than one might expect. So the problem gets compounded. The end result (like with java on the server) is that what benchmarks well in a trivial sandbox by itself on a machine, starts to have longer term negative consequences when its sharing a machine with another application (or dozen) also written under the assumption that address space is free and the OS will just take care of the problem. So, javascript DE bindings are a good idea, but should have been reserved for windows macro recorder, or vba types of use cases rather than having parts of the default installed runtime env rewritten to use it.