6 ms·

I got fed up with Docker having a "Virtual machine service" using like 4GB of memory (while running no containers) on my 8GB Mac so I investigated what I though

I got fed up with Docker having a "Virtual machine service" using like 4GB of memory (while running no containers) on my 8GB Mac so I investigated what I thought would be "native" (arm64) Lima. I tried this k8s https://github.com/lima-vm/lima/blob/f7e7addab557da560da7146a829c32c59e939a33/examples/k8s.yaml https://github.com/lima-vm/lima/blob/f7e7addab557da560da7146... example thinking it'd be a thin wrapper around QEMU. 6gb+ usage RAM with nothing deployed lol The disconnect to me is weird. I'm pretty sure with qemu-system-aarch64, Debian/Alpine don't use more than 100-200mb of RAM sitting idle. Where does 6GB of RAM usage come from? Not that I think it's a great use of time to optimize for this whatsoever. I just thought it'd be a fun exercise.

voxic11 4y ago
Its page cache[0]. A similar thing happens with wsl2 on Windows[1]. The issue is the linux kernel running in the lima-vm thinks all the ram you give the VM is free to cache files on, it doesn't know there is a host OS that could also make use of that ram. So after loading up a ton of containers where each has its own OS dependencies that ends up being a lot of files the kernel keeps cached so long as nothing else needs the memory. There isn't really a fix besides right sizing the amount of ram you give the VM. Or I guess you could disable file caching but I wouldn't recommend it[2]. [0] https://www.linuxatemyram.com/ https://www.linuxatemyram.com/ [1] https://news.ycombinator.com/item?id=26897095 https://news.ycombinator.com/item?id=26897095 [2] https://github.com/Feh/nocache https://github.com/Feh/nocache
- 3np 4y ago
  One coukd also put their VM root filesystem in ZFS, switching the Linux page cache for more tunable ARC.
- deleted 4y ago
  [deleted]
- MuffinFlavored 4y ago
  When you're right, you're right. https://github.com/lima-vm/lima/blob/master/examples/default.yaml#L41 https://github.com/lima-vm/lima/blob/master/examples/default... 4gig default ~$ free -m total used free shared buff/cache available Mem: 3907 304 1446 1 2156 3431 Swap: 0 Thank you.
  - ecnahc515 4y ago
    To add to this, the hypervisor can do more to integrate with the guest so that this issue is mitigated somewhat. For example, with KVM and a Linux guest there's a lot of optimizations to share memory pages between the two. For reference: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/chap-ksm https://access.redhat.com/documentation/en-us/red_hat_enterp....
- FooBarWidget 4y ago
  Does HyperV have a solution where the hypervisor integrates with the guest kernel, telling it to evict page caches when the host needs more memory?
  - pzmarzly 4y ago
    The mechanism is called balooning and technically doesn't need cooperation on guest kernel side (but it makes everything much smoother). Not idea about HyperV, but it is implemented e.g. in virtio https://pmhahn.github.io/virtio-balloon/ https://pmhahn.github.io/virtio-balloon/ . You can google some failed experiments to add it to qemu
- bloopernova 4y ago
  Interesting! It looks like Fedora CoreOS inside podman maybe doesn't do this? [root@localhost ~]# free -h total used free shared buff/cache available Mem: 7.8Gi 212Mi 7.2Gi 8.0Mi 333Mi 7.3Gi Swap: 0B 0B 0B And Activity Monitor says that qemu-system-x86_64 is using 1.27GB of real memory.
- saynay 4y ago
  I remember VMWare solved this with a balloon driver, that basically asks the guest for a lot of memory at a very low priority, so it has precedence over the page cache, but under any program that actually needs it.
deleted 4y ago
[deleted]
californical 4y ago
Yeah I have a recurring issue where docker desktop on Mac will use >100% cpu with no containers running (verified with docker ps and docker stats commands). I’ve tried all of the troubleshooting they have posted online. Currently got it down to ~40% cpu usage when idle, but no guarantee that it stays there. Tried switching to podman with docker-compose and podman-compose, but can’t get the auth to work for private repos on dockerhub. So basically just accepting my fate of crazy idle memory and cpu usage.
- sa46 4y ago
  My most popular StackOverflow question is that phenomenon exactly: https://stackoverflow.com/questions/58277794 https://stackoverflow.com/questions/58277794 No root cause after 3 years. The best guess is the overhead of IO somewhere in the stack.
  - charrondev 4y ago
    Are you using the “new” file synchronization? The combination of that, the new virtualization framework that stabilized last year, and upgrading to a MacBook with an M1 Pro, the relatively large codebase I run daily zips along very quickly. It’s made up of: - a PHP-fpm container running a 5m LoC codebase - a nodeJS container running a webpack dev server instance on a 1m LoC frontend codebase - 2 elasticsearch containers (one for logs and 1 for for app search) - 2 Kibana container - 1 logstash container - A rabbitmq container - a memcached container - a redis container - a Percona container - 3-4 micro services each in there own container. - an imgproxy container All of this runs quite snappily in ~6 GB of RAM and I’ve never noticed any slowness while also running Edge, slack, VSCode, PHPStorm, zoom, and a bunch of terminal windows, and I could probably get that down a bit by consolidating the elasticsearch and kibana instances.
- jwmoz 4y ago
  I had a similar prob and it was fixed by changing one of the experimental virtualisation options. Still a resource hog though.
kdrag0n 4y ago
I spent some time digging into this for the fast, light, and easy-to-use replacement for Docker Desktop and (co)lima that I've been working on. The root of the problem is that VM memory is allocated on demand, but never freed after it's used for the first time. In other words, once used, it can never be released. Since my app has a lighter userspace, it starts out using less memory than other VMs, but eventually reaches the memory limit given enough usage and time. (My optimized memory management setup means the VM works well with a lower memory limit than others, but it doesn't solve the fundamental problem.) Linux has a feature called "page reporting" to report chunks of memory that are no longer used to the hypervisor, which can then drop the to reduce usage on the host side. WSL 2 actually uses this feature, but I suspect it becomes less effective with longer VM uptime because memory becomes more fragmented over time. Since Hyper-V has been limited to dropping contiguous 2 MiB chunks of memory until recently [1], fragmentation is likely the reason many users report high memory usage. Page cache is definitely a contributor as well, but a much easier one to fix. It looks like Microsoft is working on the problem with page reporting. Although Apple's Virtualization.framework doesn't support page reporting, I was able to implement it with some workarounds and confirmed that this works with QEMU on Linux. Unfortunately, while free memory is correctly reported to macOS, nothing actually seems to get freed. I'm planning to report this to Apple because it seems like memory ballooning (essentially a more limited and primitive version of page reporting) doesn't work as documented, whether it's Virtualization.framework or another VMM like QEMU. If/when Apple fixes this, it'll be possible to reduce memory usage significantly. Details from my investigation into what's going on with memory management on the XNU side: https://twitter.com/kdrag0n/status/1612309883411640321 https://twitter.com/kdrag0n/status/1612309883411640321 The good news: From my testing, the issue isn't as bad as it appears. The "free" memory tends to compress quite well, so XNU's memory compression does a good job at taking care of it when you're actually running low on memory. (Shameless plug on this topic: The app I'm working on already has quite a few improvements over others: fast networking (30 Gbps), VirtioFS and bidirectional filesystem sharing, Rosetta for fast x86, full Linux (not only Docker), lower CPU usage, native Swift UI, and other tweaks. Email in bio for waitlist. Details to avoid spamming this thread: https://news.ycombinator.com/item?id=34374176 https://news.ycombinator.com/item?id=34374176) [1] https://lkml.org/lkml/2022/9/30/81 https://lkml.org/lkml/2022/9/30/81
- 4y ago
apatheticonion 4y ago
Honestly, Apple either need to support native containerization in the MacOS kernel (allowing people to create MacOS native images) or help bring Linux to the MBP. Docker based workflows are rubbish on my MBP.
- petre 4y ago
  That's one reason why we ditched the Mac desktops for development in favour of SFF PCs running Linux.
- happymellon 4y ago
  Honestly I hate using my Mac for container based development. I know it is "hypervisors so faster than VMs", but the workflow is so slow that when I do personal Dev work on my older £400 Lenovo with Linux it runs rings around my MacBook when dealing with containers.
  - apatheticonion 4y ago
    Agreed, I too have been moving off MacOS for my workflow. Wouldn't it be great if we could take full advantage of the incredible battery life and single threaded performance of our MBPs? I think Linux running flawlessly, with full hardware acceleration, all the drivers and equal or close battery life on a MBP would make it the best laptop money could buy. Until that day, it's a ferrari with square wheels
  - kramerger 4y ago
    > hypervisors so faster than VMs Not sure what you mean by that, hypervisor is what allows the VMs to exist. Anyway, containers are a Linux thing so I use a Linux laptop when working with containers. No VM is needed and life is great.
    - happymellon 4y ago
      Because it's hard to talk about some of these things without getting someone trying to "correct" me on terminology. Hypervisors are one method of running virtual machines, you also have classical virtualisation methods and when I bring up the virtual machines being slow, I always get someone who pushes up their glasses with an "well actually its a Hypervisor running it, so it's essentially native speed". I find Linux is a much nicer development environment all around though. [Edit] Yup, already got one.
brightball 4y ago
This is actually one of the driving factors that caused me to migrate away from Mac to Linux 5-6 years ago. It's been great. It wasn't the only factor though. The lack of options for laptops and inability to self repair drove me away too. Used to be one of the strong points.

voxic11 4y ago

Its page cache[0]. A similar thing happens with wsl2 on Windows[1]. The issue is the linux kernel running in the lima-vm thinks all the ram you give the VM is free to cache files on, it doesn't know there is a host OS that could also make use of that ram. So after loading up a ton of containers where each has its own OS dependencies that ends up being a lot of files the kernel keeps cached so long as nothing else needs the memory. There isn't really a fix besides right sizing the amount of ram you give the VM. Or I guess you could disable file caching but I wouldn't recommend it[2]. [0] https://www.linuxatemyram.com/ https://www.linuxatemyram.com/ [1] https://news.ycombinator.com/item?id=26897095 https://news.ycombinator.com/item?id=26897095 [2] https://github.com/Feh/nocache https://github.com/Feh/nocache

3np 4y ago
One coukd also put their VM root filesystem in ZFS, switching the Linux page cache for more tunable ARC.
deleted 4y ago
[deleted]
MuffinFlavored 4y ago
When you're right, you're right. https://github.com/lima-vm/lima/blob/master/examples/default.yaml#L41 https://github.com/lima-vm/lima/blob/master/examples/default... 4gig default ~$ free -m total used free shared buff/cache available Mem: 3907 304 1446 1 2156 3431 Swap: 0 Thank you.
- ecnahc515 4y ago
  To add to this, the hypervisor can do more to integrate with the guest so that this issue is mitigated somewhat. For example, with KVM and a Linux guest there's a lot of optimizations to share memory pages between the two. For reference: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/chap-ksm https://access.redhat.com/documentation/en-us/red_hat_enterp....
FooBarWidget 4y ago
Does HyperV have a solution where the hypervisor integrates with the guest kernel, telling it to evict page caches when the host needs more memory?
- pzmarzly 4y ago
  The mechanism is called balooning and technically doesn't need cooperation on guest kernel side (but it makes everything much smoother). Not idea about HyperV, but it is implemented e.g. in virtio https://pmhahn.github.io/virtio-balloon/ https://pmhahn.github.io/virtio-balloon/ . You can google some failed experiments to add it to qemu
bloopernova 4y ago
Interesting! It looks like Fedora CoreOS inside podman maybe doesn't do this? [root@localhost ~]# free -h total used free shared buff/cache available Mem: 7.8Gi 212Mi 7.2Gi 8.0Mi 333Mi 7.3Gi Swap: 0B 0B 0B And Activity Monitor says that qemu-system-x86_64 is using 1.27GB of real memory.
saynay 4y ago
I remember VMWare solved this with a balloon driver, that basically asks the guest for a lot of memory at a very low priority, so it has precedence over the page cache, but under any program that actually needs it.

deleted 4y ago

[deleted]

californical 4y ago

Yeah I have a recurring issue where docker desktop on Mac will use >100% cpu with no containers running (verified with docker ps and docker stats commands). I’ve tried all of the troubleshooting they have posted online. Currently got it down to ~40% cpu usage when idle, but no guarantee that it stays there. Tried switching to podman with docker-compose and podman-compose, but can’t get the auth to work for private repos on dockerhub. So basically just accepting my fate of crazy idle memory and cpu usage.

sa46 4y ago
My most popular StackOverflow question is that phenomenon exactly: https://stackoverflow.com/questions/58277794 https://stackoverflow.com/questions/58277794 No root cause after 3 years. The best guess is the overhead of IO somewhere in the stack.
- charrondev 4y ago
  Are you using the “new” file synchronization? The combination of that, the new virtualization framework that stabilized last year, and upgrading to a MacBook with an M1 Pro, the relatively large codebase I run daily zips along very quickly. It’s made up of: - a PHP-fpm container running a 5m LoC codebase - a nodeJS container running a webpack dev server instance on a 1m LoC frontend codebase - 2 elasticsearch containers (one for logs and 1 for for app search) - 2 Kibana container - 1 logstash container - A rabbitmq container - a memcached container - a redis container - a Percona container - 3-4 micro services each in there own container. - an imgproxy container All of this runs quite snappily in ~6 GB of RAM and I’ve never noticed any slowness while also running Edge, slack, VSCode, PHPStorm, zoom, and a bunch of terminal windows, and I could probably get that down a bit by consolidating the elasticsearch and kibana instances.
jwmoz 4y ago
I had a similar prob and it was fixed by changing one of the experimental virtualisation options. Still a resource hog though.

kdrag0n 4y ago

I spent some time digging into this for the fast, light, and easy-to-use replacement for Docker Desktop and (co)lima that I've been working on. The root of the problem is that VM memory is allocated on demand, but never freed after it's used for the first time. In other words, once used, it can never be released. Since my app has a lighter userspace, it starts out using less memory than other VMs, but eventually reaches the memory limit given enough usage and time. (My optimized memory management setup means the VM works well with a lower memory limit than others, but it doesn't solve the fundamental problem.) Linux has a feature called "page reporting" to report chunks of memory that are no longer used to the hypervisor, which can then drop the to reduce usage on the host side. WSL 2 actually uses this feature, but I suspect it becomes less effective with longer VM uptime because memory becomes more fragmented over time. Since Hyper-V has been limited to dropping contiguous 2 MiB chunks of memory until recently [1], fragmentation is likely the reason many users report high memory usage. Page cache is definitely a contributor as well, but a much easier one to fix. It looks like Microsoft is working on the problem with page reporting. Although Apple's Virtualization.framework doesn't support page reporting, I was able to implement it with some workarounds and confirmed that this works with QEMU on Linux. Unfortunately, while free memory is correctly reported to macOS, nothing actually seems to get freed. I'm planning to report this to Apple because it seems like memory ballooning (essentially a more limited and primitive version of page reporting) doesn't work as documented, whether it's Virtualization.framework or another VMM like QEMU. If/when Apple fixes this, it'll be possible to reduce memory usage significantly. Details from my investigation into what's going on with memory management on the XNU side: https://twitter.com/kdrag0n/status/1612309883411640321 https://twitter.com/kdrag0n/status/1612309883411640321 The good news: From my testing, the issue isn't as bad as it appears. The "free" memory tends to compress quite well, so XNU's memory compression does a good job at taking care of it when you're actually running low on memory. (Shameless plug on this topic: The app I'm working on already has quite a few improvements over others: fast networking (30 Gbps), VirtioFS and bidirectional filesystem sharing, Rosetta for fast x86, full Linux (not only Docker), lower CPU usage, native Swift UI, and other tweaks. Email in bio for waitlist. Details to avoid spamming this thread: https://news.ycombinator.com/item?id=34374176 https://news.ycombinator.com/item?id=34374176) [1] https://lkml.org/lkml/2022/9/30/81 https://lkml.org/lkml/2022/9/30/81

4y ago

apatheticonion 4y ago

Honestly, Apple either need to support native containerization in the MacOS kernel (allowing people to create MacOS native images) or help bring Linux to the MBP. Docker based workflows are rubbish on my MBP.

petre 4y ago
That's one reason why we ditched the Mac desktops for development in favour of SFF PCs running Linux.
happymellon 4y ago
Honestly I hate using my Mac for container based development. I know it is "hypervisors so faster than VMs", but the workflow is so slow that when I do personal Dev work on my older £400 Lenovo with Linux it runs rings around my MacBook when dealing with containers.
- apatheticonion 4y ago
  Agreed, I too have been moving off MacOS for my workflow. Wouldn't it be great if we could take full advantage of the incredible battery life and single threaded performance of our MBPs? I think Linux running flawlessly, with full hardware acceleration, all the drivers and equal or close battery life on a MBP would make it the best laptop money could buy. Until that day, it's a ferrari with square wheels
- kramerger 4y ago
  > hypervisors so faster than VMs Not sure what you mean by that, hypervisor is what allows the VMs to exist. Anyway, containers are a Linux thing so I use a Linux laptop when working with containers. No VM is needed and life is great.
  - happymellon 4y ago
    Because it's hard to talk about some of these things without getting someone trying to "correct" me on terminology. Hypervisors are one method of running virtual machines, you also have classical virtualisation methods and when I bring up the virtual machines being slow, I always get someone who pushes up their glasses with an "well actually its a Hypervisor running it, so it's essentially native speed". I find Linux is a much nicer development environment all around though. [Edit] Yup, already got one.

brightball 4y ago

This is actually one of the driving factors that caused me to migrate away from Mac to Linux 5-6 years ago. It's been great. It wasn't the only factor though. The lack of options for laptops and inability to self repair drove me away too. Used to be one of the strong points.