9 ms·

I helped fix sleep-wake hangs on Linux with AMD GPUs

772 points by fanf2 2y ago

dralley 2y ago
Fantastic news. AMD's linux graphics drivers have mostly worked great for me but this has been the one exception that I've hit multiple times.
- juujian 2y ago
  Same great experience, but I experience similar issues when I disconnect thunderbolt with monitors when my machine is asleep. Laptop though, so very different driver set, no GPU via pci
  - jorvi 2y ago
    You can probably write a udev rule for the Thunderbolt / USB-C port with either ACTION=="offline", "remove" or "online". "offline" is for when your system turns off or suspends, "online" vice versa, and "remove" is self-explanatory. If you go with "offline", I'd look into hard disconnecting the monitors. This might cause monitor rearrangement (= you'll need to manually assign) or blinking on laptop bootup. Might also stop any charging. But that could be mitigated by checking the "subsystems" attribute. If you go with "online", you probably need run some sort of clean-up / refresh script or rule If you go with "remove", you'll need the same clean-up / refresh script. It'll take some trial-and-error, but it'll be satisfying once it works. Also highly recommended to check the NixOS repositories and official wiki and Arch wiki to see if your laptop or monitors have workarounds for their quirks.
- MegaDeKay 2y ago
  My luck has been a little less good. Latest problem I'm having is the driver spamming my logs after waking from sleep with "[drm] scheduler comp_1.0.n is not ready, skipping" after "WARNING: CPU: 12 PID: 11871 at drivers/gpu/drm/amd/amdgpu/../display/dc/dc_helper.c:100 generic_reg_update_ex+0x1d2/0x290 [amdgpu]" https://gitlab.freedesktop.org/drm/amd/-/issues/3911 https://gitlab.freedesktop.org/drm/amd/-/issues/3911
  - binkHN 2y ago
    I have something similar with dmesg spam and a possibly related issue, sadly: https://gitlab.freedesktop.org/drm/amd/-/issues/3790 https://gitlab.freedesktop.org/drm/amd/-/issues/3790
    - MegaDeKay 2y ago
      Using an ICC profile per the last comment in that report didn't fix it for you?
jorvi 2y ago
> Through some digging, I found that when a desktop enters S3 sleep, the system cuts power to PCIe GPUs I am not sure how correct this assumption is. S3 is supposed to cut power to everything but RAM, but for example Gigabyte Aorus motherboards are notorious for an NVMe SSD sleep bug that randomly prevents the system from properly sleeping or waking. This is fixed by adding the following udev rule: # Generic PCIe fix for sleep bugs by preventing wakeup from any PCIe port ACTION=="offline", SUBSYSTEM=="pci", DRIVER=="pcieport", ATTR{power/wakeup}="disabled" or more targeted: # Gigabyte sleep fix by preventing wakeup from problematic PCIe port, depends on motherboard model ACTION=="offline", SUBSYSTEM=="pci", ATTR{vendor}=="0x8086", ATTR{device}=="0x43bc", ATTR{power/wakeup}="disabled" You can find any glitched PCIe wakeup device with: 1. cat /proc/acpi/wakeup (you'll have to trial and error your way through the wakeup devices if it isn't immediately clear) 2. cat /sys/class/pci_bus/*/*/yourWakeupDevicePci/uevent | grep PCI_ID 3. prepend "0x" You also have the option of: udevadm info --attribute-walk /dev/whatever but for that you need to know some basic identifier of your glitchy device. Or if you want to shellscript it (less reliable than letting udev do it for you and needs to be done via systemd service file or another automation): # Gigabyte sleep fix, port depends on mobo model /bin/bash -c 'if grep 'RP05' /proc/acpi/wakeup | grep -q 'enabled'; then echo 'RP05' > /proc/acpi/wakeup; fi'"; Yes I really hate this (and other) Linux sleep issues.
- VMG 2y ago
  As somebody with an Aorus motherboard who has probably burned a few kWh on this issue, I was really excited to try these solutions - no luck. Thank you anyway!
  - jorvi 2y ago
    Did you try the general fix? And reload udev rules? You also have to make sure it applies after the default rules. You can check if the rule applies once you have everything set up by doing an `udevadm` attribute walk of your SSD device (not partition), and then following it up all the way up the device tree until you see your specific device port (target fix) or PCIe driver subsystem (general fix). Then check if "power/wakeup" is set to "disabled". If it is set to disabled, something else is keeping your device awake on sleep. For that you can check /proc/acpi/wakeup, and there's also a specific systemd invocation (that I forgot) you can do that shows if your device slept, how long it slept, how much battery was drained, and if your device woke-up, slept or failed to resume, it'll give you a reason.. to the best of its ability.
Gormo 2y ago
My sincere personal thanks for this. My main laptop is a Ryzen-based ThinkPad running Linux that I suspend and hibernate regularly, and I sporadically encounter this issue. Looking forward to 6.14!
- imp0cat 2y ago
  This. Thanks a lot!
raffraffraff 2y ago
> I dug a PS/2 keyboard out of a dusty closet and plugged it into my system (only safe when the PC is off!) Lol, I remember.
yellow_lead 2y ago
I have an Nvidia GPU and a sporadic crash (black screen) with no logs on Linux. I suspect it's a driver issue too. Going to try some of these tips to enable the debug shell, but I'm not sure if they'll be effective. Anyone have other tips for this type of thing? I did try upgrading drivers/kernels already
- an_ko 2y ago
  Run a memtest. Graphics cards usually crash badly when given invalid data, which can happen sporadically if you have bad RAM. If memtest shows a specific memory region as failing, swap out sticks to check which it is, and buy a new one. (Or if you're on a tight budget, you can disable that region with kernel boot options.) If memtest gives errors in lots of places, might be a bad overclock. Loosen timings or give it more voltage.
  - bmicraft 2y ago
    > If memtest shows a specific memory region as failing, swap out sticks to check which it is, and buy a new one. (Or if you're on a tight budget, you can disable that region with kernel boot options.) Honestly, I'd always do that as the first option. In most cases you can still get years of life out of that stick of ram (but do check regularly, like after a week and then double the interval every time if it didn't get worse).
  - porridgeraisin 2y ago
    This. I would also recommend running it on one memory slot only (do this one by one). Recently it happened that even a multi-day memtest run did not catch the defect in one of my ram slots. But when I emptied it, all the gpu driver problems immediately stopped. The symptom was a crash followed by garbage on screen and a "zzzz" sound. Sometimes followed by a shutdown.
  - yellow_lead 2y ago
    Thanks, I'll try it!
- nyanpasu64 2y ago
empiricus 2y ago
I notice I am confused how the code needed for the GPU to sleep was implemented. It was failing when simply saving/copying gigabytes of flat memory, but on the other hand it was able to recover successfully the previous complex hw and sw state and data structures?! I guess it probably makes sense if after waking up that data is actually dropped and the gpu and ui is reinitialized and redrawn.
- nyanpasu64 2y ago
  As I understand if the GPU fails to save VRAM it drops the RAM copy(?) before restoring the GPU data structures or abandoning sleep entirely. But if it saves VRAM then crashes after the GPU is suspended, it will often fail to wake up the GPU, resulting in no monitor signal. Though I'm fuzzy on the details (and the order of operations depends on which kernel you're running).
kkarpkkarp 2y ago
omg, thank you
Asmod4n 2y ago
> On my laptop, I opened a terminal and ran sudo minicom --device /dev/ttyUSB0 --baudrate 115200 to monitor the computer over serial. In addition to saving logs You can just use screen for that and have a working terminal with color support et al.
- vbernat 2y ago
  Another lightweight option is tio.
deleted 2y ago
[deleted]
0xTJ 2y ago
Very excited to see 6.14 hit Arch! Hanging around sleep (with symptoms that sound like what's described in the write-up) has been the one persistent occasional issue, so I'm hoping that this fixes what I'm seeing.
progforlyfe 2y ago
Extremely high level genius stuff -- nice work and thank you for your efforts!
sabujp 2y ago
This is amazing work! If folks have ever wondered why suspend is so difficult to get working on linux and why debugging it is equally difficult, this is a single datapoint with lots of information about all the things that can go wrong. Even now I have a thinkpad P1G4 where the fans won't turn off automatically unless I turn them off before going into suspend. Recently I also started having crackling issues with my bluetooth headphones after resuming from suspend and had to disable node suspension there also (https://wiki.archlinux.org/title/PipeWire#Noticeable_audio_delay_or_audible_pop/crack_when_starting_playback https://wiki.archlinux.org/title/PipeWire#Noticeable_audio_d...).
- Apofis 2y ago
  Remarkable that it's 2025 and laptop sleep/suspend still doesn't work right on linux. I think the first time I encountered this was probably 15 years ago now?
  - bongodongobob 2y ago
    It doesn't work right on Windows either to be fair. With a mixed laptop fleet at work, we've just disabled sleep/hibernate company wide because it causes way too many problems.
    - larrik 2y ago
      That seems like a good way to cook your laptop when you throw it into a bag with the CPU pegged.
      - bongodongobob 2y ago
        Not if you turn it off first.
        Joker_vD 2y ago
        For some reason, people really insist on being able to put their machines to sleep, I honestly don't know why. Long boot times, maybe?
Voultapher 2y ago
Since a couple Linux versions something around 6.10 IIRC I've had it where my Nvidia system wakes into a black screen, but with a cursor and alt shells work, specifically KDE Plasma seems bugged here, but they say it's a kernel issue, or at least there are dozens of separate issues open about this kind of bug and it's rather annoying that I can't put my machine to sleep. If anyone has ideas what could fix this I'd really appreciate it. The machine is dual booted with Windows, and there sleep works without issue, so it's clearly possible, as it was for years before that on Linux as well.
- thangalin 2y ago
  I have a similar issue. When turning off the monitor using its power button, the system semi-crashes. After powering the monitor back on, I have to go into an alt shell and kill lightdm, which also kills all running GUI applications. Not the greatest workflow. $ uname -a Linux hostname 6.13.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 02 Feb 2025 01:02:29 +0000 x86_64 GNU/Linux NVIDIA-SMI 570.86.16 Driver Version: 570.86.16 CUDA Version: 12.8 Instead of powering off the monitor, I've created an alias: alias off='xset dpms force off && await.sh && xset -dpms' And shell script (press any key to unblank the screen): $ cat ~/bin/await.sh #!/usr/bin/env bash xinput test-xi2 --root 2>&1 | \ grep --line-buffered -m 1 'EVENT type 2 (KeyPress)' &> /dev/null This allows me to run "off", a comromise to put the monitor in low-power mode. This solves the problem of having to kill lightdm, which improves the workflow. Still, sometimes I have to open an alt shell and then press Alt+F7 to return to X. Rarely, I'll have to go to the alt shell/Alt+F7 a second time to suppress sporadic screen glitching. Adding nomodeset to the GRUB configuration was another suggestion: # grep vmlinuz /boot/grub/grub.cfg | head -1 linux /vmlinuz-linux root=UUID=... rw nomodeset quiet resume=UUID=... Hope it helps.
  - saltcured 2y ago
    Now I'm not sure if my Fedora 41 experience is better or worse than your problem. An older Thinkpad with secondary NVIDIA dGPU used to work fine, but now every time the monitor powers off (or is unplugged), Xorg instantly dies with no apparent logs. So I don't have to do anything special to recover except login over again and start a whole new MATE session.
jph 2y ago
Your work will help me on a Framework AMD laptop with the GPU extension and dual boot Linux/Windows. May I donate to you or to your favorite charity? My contact info is in my profile.
- lelandfe 2y ago
  Love this!
zrm 2y ago
> To make room for VRAM, memreserver allocates system RAM based on used VRAM plus 1 gigabyte, then fills the RAM with 0xFF bytes and mlocks the memory (so none of it is swapped out). That seems like a bit of trouble if you have 16GB of system RAM and a 24GB GPU.
deleted 2y ago
[deleted]
Daunk 2y ago
For all the years I've been using Linux, I've always had some kind of sleep issues. I've used Intel, AMD, ATI, and NVIDIA hardware across countless distros and setups, yet nothing seems to make a difference, there's always something that doesn’t work properly with sleep or hibernation. Honestly, it's one of the main issues I wish the Linux community would take a closer look at and finally fix!
- Narishma 2y ago
  I think it it's because there are too many subsystems involved in sleep/resume all being worked on as independent projects (kernel, drivers which sometimes have both kernel and user space components, init system, display server, desktop environment, probably others I'm not aware of). That said, I've had my share of sleep issues on Windows as well over the years, I suspect for the same reasons.
- 1970-01-01 2y ago
  This. Linux users must resort to bronze-age tooling in 2025; Crafting and launching handmade scripts by candlelight to diagnose their plethora of sleep issues. But the community likes it this way. Meanwhile, Mac users continue to have sleep that 'just works' and Windows users have an entire sleep troubleshooting toolkit: https://learn.microsoft.com/en-us/windows-hardware/design/device-experiences/tools-available https://learn.microsoft.com/en-us/windows-hardware/design/de...
  - nyanpasu64 2y ago
    I once ran into a corporate laptop that had been downgraded from Windows 11 to 10, that would burn up its CPU during Modern Standby and eventually enter hibernate after burning a good fraction of its battery. The sleep study identified various PCIe devices and I tried installing drivers but they did not help. I wonder if it would've worked better on Windows 11 with stock drivers.
- kiwijamo 2y ago
  Interestingly sleep/wake is something I've found to work almost always just fine out of the box in Linux, including on machines Windows has sleeping issues! It used to be quite bad but things has improved heaps over the last 10 or so years -- however I've also stuck with Lenovo laptops which does generally seem to have better support in Linux.
sidkshatriya 2y ago
TL;DR: During suspend, for graphics cards, GPU VRAM needs to be transferred to system RAM. However, during high memory usage scenarios the VRAM + RAM usage could exceed system memory -- this would ordinarily involve system swap coming into play and handling the temporarily out of memory issue. However system swap was already deactivated when it came time to suspending the AMD card causing all sorts of problems. The fix was asking the GPU to evict its VRAM to system RAM via the hook ("suspend prepare") before swap was deactivated in linux kernel.
- nyanpasu64 2y ago
  <s>Technically it was the suspend notifier; even suspend prepare executes after swap is disabled.</s> See replies.
  - sidkshatriya 2y ago
    Isn’t the hook called PM_SUPEND_PREPARE as per https://nyanpasu64.gitlab.io/blog/amdgpu-sleep-wake-hang/#solution-power-management-notifiers https://nyanpasu64.gitlab.io/blog/amdgpu-sleep-wake-hang/#so...
    - nyanpasu64 2y ago
      Sorry I had forgotten my own article :( It is confusing that PM_SUSPEND_PREPARE and dpm_prepare share the same word.
ej1 2y ago
[dead]
whatever1 2y ago
Apple became a trillion dollar company by mastering sleeping / waking up of electronic devices. why nobody else sees this?
- edoceo 2y ago
  That and app store fees
- deleted 2y ago
  [deleted]
- rafaelmn 2y ago
  I still have i9 2018 that will drain overnight in sleep mode, and that sleep discharge put the most battery cycles on the device. I think they only fixed it when they ditched the x86 ecosystem.
- megous 2y ago
  Pretty much all smartphones can do this.
  - whatever1 2y ago
    *Recent. It took many years for smartphones to catch up with iPhones in terms of energy drain. Even today iPhones have the smallest batteries across their competitors.
    - megous 2y ago
      Battery life has nothing to do with suspend/resume reliability. Every smartphone does suspend/resume thousands of times each day, it has to be nearly 100% reliable feature and it is, otherwise users would be majorly pissed off.
- meowkit 2y ago
  Because they own the now hardware, and MacOS iOS run exclusively on their hardware Source: I work on windows power management and I know system engineers at apple.
voytec 2y ago
I've had zero problems with S3 wake/sleep on AMD ThinkPad with FreeBSD for years. And FreeBSD uses AMD drivers pulled from Linux. How is this still a problem on Linux? hw.acpi.lid_switch_state=S3
- larrik 2y ago
  I battled sleep issues on my laptop for months, and had very different results than most people with my model. I think the behavior is a total crapshoot from machine to machine. In fact, it turned out that a BIOS update that happened in the middle of my issues broke sleep functionality for the whole machine for a few months, so that wasn't even my fault.
- kiwijamo 2y ago
  Ditto. My Thinkpad X1 and X390 both have reasonably reliable sleep/wake under Debian.
nyanpasu64 2y ago
Update: I upgraded to an Intel Arc B570 GPU... and ran into the exact same problem on an independent driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/4288 https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/4288
- KronisLV 2y ago
  Oh hey, another Intel Arc user! I got the LE B580 and so far it's been a pretty good experience for me, though I don't really use suspend. Also I got curious about your blog (haven't heard about Zola, recently migrated my own blog over to a newer version of Grav, which is a flat file CMS with no DB) and saw your post about looking for a new job. Hope things work out for you, stay safe!
  - nyanpasu64 2y ago
    All the non-scalped B580s were out of stock by the time I decided to buy a GPU :( Thanks for the moral support!
tgsovlerkhgsel 2y ago
AMD GPU linux drivers are (were?) a nightmare in general, and this includes iGPUs in their processors. Sadly, I don't have the impression that AMD is actively working on fixing this. Just to make sure I'm not griping over something long fixed, I took a quick look and instantly found someone with a very similar issue to the one I ran into happening on a semi-recent kernel: https://community.amd.com/t5/pc-drivers-software/linux-amdgpu-ring-gfx-timeouts-in-various-applications/td-p/723574 https://community.amd.com/t5/pc-drivers-software/linux-amdgp... It looks to me that if you want to have a working computer under Linux, it's worth the extra cost to avoid AMD.
- tostiheld 2y ago
  > It looks to me that if you want to have a working computer under Linux, it's worth the extra cost to avoid AMD. I think this is a rather hasty conclusion. The popular opinion is the opposite. If you want a working computer under Linux, it's worth it to avoid NVIDIA, especially for laptops. Sure, AMD are not perfect contributors to the kernel, but they are contributing more than NVIDIA[0]. NVIDIA has made some moves recently[1], but the AMD GPUs are still better integrated. Notably since the Steam Deck had been released, the situation has been excellent. Anecdotally, my laptop with an NVIDIA GPU has many issues that have persisted over the years with things like high idle power draw or frequent straight up crashes, or incomplete Wayland support. My 3 devices that have an AMD GPU (1 desktop, 2 laptops) however, have been working flawlessly from day 1. [0] https://www.phoronix.com/news/NVIDIA-Contributions-2010s-Kern https://www.phoronix.com/news/NVIDIA-Contributions-2010s-Ker... [1] https://github.com/NVIDIA/open-gpu-kernel-modules https://github.com/NVIDIA/open-gpu-kernel-modules
  - MegaDeKay 2y ago
    It depends on the use case. AMD is notorious for their "AMD reset bug" when passing a GPU through to a VM using VFIO. Restart the guest and most cards will lock up because AMD doesn't handle PCI resets properly. You then need to reboot the host to fix it (!). This has been a problem since Polaris if not before and AMD hasn't fixed it, despite knowing full well that the problem exists. At least in this regard, NVIDIA (and Intel as far as I know) work fine. The community has been able to come up with a workaround for some older cards but the problem persists even in their current cards. https://github.com/gnif/vendor-reset https://github.com/gnif/vendor-reset
_cenw 2y ago
Some AMD integrated GPUs are surprisingly fragile with this. I have a GPD Win Max 2 8840U (a "concept car" handheld laptop hybrid) and when I got it last year, it would fail to wake from suspend and hibernate about half the time in Windows, with Linux actually being more reliable (but also not perfect), and only this year did an AMD GPU driver fix this.
jchw 2y ago
Memory management and specifically OOM conditions remain an unbelievably painful nightmare on Linux. It's not like I run into these issues constantly, but I've definitely tried to debug issues like these (unsuccessfully). Ultimately if I OOM a machine I usually wind up installing more RAM, which is wasteful/expensive, but it's pretty clear that handling OOM conditions gracefully is going to be a hard problem for Linux to solve into the future. This is really great work and will serve as a reference point for debugging similar issues in the future. Pretty happy about systemd's debug-shell feature, I had no idea that existed. I don't think my X670E Steel Legend board has a serial header anywhere on it, though. How do modern built-in serial ports work, anyway? Are they attached off of the chipset PCIe lanes? Something that's also very useful when trying to dive into the Linux kernel is that there's a bunch of great talks discussing Linux kernel subsystems from conferences like FOSDEM and Linux Plumber's Conference which you can usually find recordings of online. For example, there's this one for TTM, the memory subsystem that most of the desktop GPU DRM drivers use: https://www.youtube.com/watch?v=MG7_tUNKSt0 https://www.youtube.com/watch?v=MG7_tUNKSt0
- Skunkleton 2y ago
  I’ve had good luck containing ooms with cgroups. I’m not sure if there is a state of the art for handling oom conditions beyond what Linux does. If anyone knows and can recommend some reading I would appreciate it.
  - jchw 2y ago
    There's really two problems as I understand it: - Overcommit. Linux will "overcommit" memory: allocations will succeed when there's no memory, and then hang when the page is actually mapped if no physical pages are available (to my understanding.) Windows NT doesn't do this. Not sure exactly how macOS/XNU handles it. - The OOM killer. Because allocations don't fail, to actually recover from an OOM situation the kernel will enumerate processes and try to kill ones that are using a lot of memory, by scoring them using heuristics. The big problem? If there isn't a single process hogging the memory, this approach is likely to work very poorly. As an example, consider a highly parallel task like make -j32. An individual C++ compiler invocation is unlikely to use more than a gigabyte or two of memory, so it's more likely that things like Electron apps will get caught first. The thrashing of memory combined with the high CPU consumption of compilers that are not getting killed will grind the machine to a near-complete halt. If you are lucky, then it will finally pick a compiler to kill, and set off a chain reaction that ends your make invocation. There are solutions... Indeed, you can use quotas with cgroups. There's tools like systemd-oomd that try to provide better userspace OOM killing using cgroups. You can disable overcommit, but some software will not function very well like this as they like to allocate a ton of pages ahead of time and potentially use them later. Overcommit fundamentally improves the ability to efficiently utilize all available memory. Ultimately I think overcommit is probably a bad idea... but it is hard to come up with a zero-compromises solution that keeps optimal memory/CPU utilization but avoids pathological OOM conditions by design.
mistyvales 2y ago
Highly relevant! Thanks for this.
dekhn 2y ago
I used to think that naming things, cache invalidation, and off-by-one errors were the 2 biggest problems in CS, but then I learned about the "sleep/wake" problem and realized it's NP-complete.
- nikanj 2y ago
  Only on Linux though, on Windows it’s O(n2) and on Macos it’s O(log n)
  - itsn0tm3 2y ago
    Well only as long as you don‘t hackintosh. That stuff used to be a horror sometimes!
    - 0x38B 2y ago
      Remembering the hours I spent going through KEXTs and bootloader config in ill-fated attempts to set up a Hackintosh fills me now with a kind of horror. Worst of all were the ACPI tables - SSDT and the like. In contrast to that, running MacOS in a VM is heaven. Figuring out how to pass my iPhone through to Xcode took as long as the initial setup.
  - ncann 2y ago
    With how much trouble I had with trying (and failing) to make my brand new Dell laptop sleep properly and not the "Modern Standby" crap, plus my desktop randomly breaking GPU hardware acceleration in browser after waking up, I would say it's around O(n4) now. Or maybe even O(n!).
- verall 2y ago
  I think sleep/wake is a subset of cache invalidation - if all peripherals were stateless, probably it wouldn't be an issue.
deepsun 2y ago
I have to unplug Logitech wireless receivers for mouse/keyboard, otherwise desktop wakes up immediately.
podiki 2y ago
I didn't have issues with sleep/wake until somewhat recently (not sure when) and found this post. Grabbing the patch from the commit referenced and using it on top of 6.12 and 6.13 kernels seems to have fixed it for me too (for the past couple of weeks and counting). Great work!
Bobaso 2y ago
My thinkpad E14 gen5 intel + ubuntu 24.04 is my first ever laptop where sleep work exactly as intended. with ~1% battery waste per hour
tombot 2y ago
> This took over a year of debugging and multiple attempts by many people to fix. 2025 finally Linux on the desktop
- Nezghul 2y ago
  I'm also programmer and such bugs immediately reminds me of all my managers demanding exact time it would take me to fix such bug and me telling them it could take from 1 day to 1 year was never taken seriously :(
devilsdata 2y ago
Props to you for this. I am not clever/experienced enough to solve my own issues with sleep-wake hangs on Linux at work. I’ve instead opted to work around it. I use Firefox, Obsidian, and Tmux with Neovim for all my work. Tmux has resurrect and a plugin that saves my entire terminal state automatically every few minutes. I also have a command that automatically sets up my i3wm/regolith windows exactly how I like. Basically if I run `wkup`, I’m exactly where I was, down to the line of code open on NeoVim, Firefox tab, and dev server or cargo running.
- bbkane 2y ago
  Could you link to your i3wm/regolith window setup code? I'd like to set up something similar!
  - devilsdata 2y ago
    Here we go: https://github.com/lkdm/dotfiles/blob/main/bin/executable_wkup https://github.com/lkdm/dotfiles/blob/main/bin/executable_wk... Hope it helps you out
lorenzbrun 2y ago
Author of memreserver (one of the mentioned userspace workarounds) here. I've debugged this a few years back, only public comment I can quickly find is [1]. I also remember some mailing list discussions, but it basically came down to the isuse that Linux didn't have staggered suspend hooks that reliably ran before disks and parts of the memory subsystem were frozen. Apparently this is now possible. Sadly the Freedesktop Gitlab doesn't seem indexable so this knowledge seems to have gotten lost. [1] https://gitlab.freedesktop.org/drm/amd/-/issues/2125#note_1708592 https://gitlab.freedesktop.org/drm/amd/-/issues/2125#note_17...
badsectoracula 2y ago
I wonder if this will help a similar problem i have with my AMD GPU: very often when i wake/resume the PC, the output is almost frozen. "Almost" because it actually isn't frozen, if there is any output/animation/etc going on it plays fine, but once i try to move the mouse it freezes and everything updates at a single frame per couple of seconds - sometimes freezing completely. I can usually Ctrl+Alt+Fn to another virtual desktop in text mode and, if that is possible (i.e. the computer hasn't completely frozen, though sometimes it takes about a minute to switch), i can Ctrl+Alt+Fn back and everything works fine. Dmesg has a ton of spam messages from amdgpu after that. AFAICT (from the behavior) something isn't properly saved/restored and communicating with the GPU (the mouse cursor is a hardware cursor thus needs to send commands to the GPU to update its position) causes some sort of issue. Switching to another virtual terminal that is running in text mode probably forces the driver to reset its graphics state. Of course that is just my assumption based on what i see going on. Weirdly enough this only happens after i replaced my RX 5700 XT with a RX 7900 XTX so it might be something GPU (or GPU arch) specific. I've been considering plugging my laptop and see if there is something i can figure out (GPU aside the PC is usable, but i guess if this a kernel side thing i'd need a second computer connected to it to debug it), but as this isn't something i've tried before (though i know someone who has and said it isn't anything special) my annoyance still hasn't gone over the "i need to get to the bottom of this" threshold :-P. It'd be nice if 6.14 fixes the issue, though i am not sure as i rarely have more than 1/3rd of the system RAM (32GB) in use and VRAM (24GB) barely goes above 1-2GB of use outside games. But this post might be helpful in diagnosing the issue next time it happens :-).
- nyanpasu64 2y ago
  I've heard reports of reduced frame rate issues on laptops from DMUB/DMCUB/panel self-refresh, though your issue sounds different and I probably can't usefully debug it without a full dmesg/journalctl --system log.
schainks 2y ago
Ugh the first time I was debugging problems like this, it was in production and for some IoT hardware we had deployed in the field. Fortunately, although that's not the focus of this article, system hibernate is WAY more reliable than system sleep in Linux due to the way it works. Use system hibernate if your SSD is fast enough. It works better than system sleep and isn't a ton slower.
igtztorrero 2y ago
Thanks, this happen to me, I avoided sleep functions
dado3212 2y ago
> So I did the natural thing: I saved and extracted the amdgpu.ko kernel module, decompiled it in Ghidra, and mapped the location of the crash in dm_resume to the corresponding lines in the kernel source. This is always my favorite part of debugging.
thrdbndndn 2y ago
what is 'agd5f/linux'?
- nyanpasu64 2y ago
  As I understand https://gitlab.freedesktop.org/agd5f/linux https://gitlab.freedesktop.org/agd5f/linux is Alex Deucher (amdgpu maintainer)'s personal Git tree/repo (https://docs.kernel.org/process/maintainers.html#radeon-and-amdgpu-drm-drivers https://docs.kernel.org/process/maintainers.html#radeon-and-...), where amdgpu changes get merged into before they reach the main Linux tree (and/or the drm/tip repo, I'm not sure their relationship).
fowl2 2y ago
I guess "hibernating" (writing VRAM to swap) works better than expecting userspace to gracefully handle device resets. One linear read vs. a thundering herd of processes re-initialising, decompressing, etc. should be more predictable/reliable at least. I do wonder however how much VRAM is "volatile" - ie. framebuffers - and could just be thrown away. And web browsers seem to handle GPU resets just fine, so maybe they could opt-in?
- nyanpasu64 2y ago
  I didn't mention in the article but the Nvidia drivers at one point would drop VRAM rather than preserving it, leading to corrupted RGB noise textures in window managers and browsers (and potential crashes though I don't think I encountered them). I suggested doing this on the AMD bug tracker (https://gitlab.freedesktop.org/drm/amd/-/issues/2362#note_2094417 https://gitlab.freedesktop.org/drm/amd/-/issues/2362#note_20...), but the amdgpu developers weren't interested.
  - fowl2 2y ago
    Oh wow didn't really expect anything other than whole device loss. Just returning garbage does sound bad. Do you know if they tried to communicate with clients and were just ignored/not implemented or if the APIs just don't support it? A quick search indicates that "residency"[1] exists, but no idea the extent it's useful/implemented. [1] https://learn.microsoft.com/en-us/windows/win32/direct3d12/residency https://learn.microsoft.com/en-us/windows/win32/direct3d12/r...
    - nyanpasu64 2y ago
      As I understand, KWin does call glGetGraphicsResetStatusARB(), and on Nvidia GPUs, sleep-wake (or opening apps if it crashed the GPU) would cause KDE to detect a graphics reset and print "Desktop effects were restarted due to a graphics reset" (https://github.com/KDE/kwin/blob/10c04995c1f9f82ddbd6610e5e007efe9d8af13b/src/compositor_x11.cpp#L457 https://github.com/KDE/kwin/blob/10c04995c1f9f82ddbd6610e5e0...). I haven't used Nvidia GPUs in years and don't know if this is still an issue. I think many apps don't check for graphics resets? I'm not sure if residency is relevant here; the Microsoft link indicates that eviction makes memory inaccessible from the GPU to make room for other memory, which explains why Linux uses the same name for "backing up" VRAM before sleep (through the same underlying mechanism).
stycznik 2y ago
It seems like the GPU should grow the capability to keep its VRAM intact through suspend, it's already complex[0] enough it's basically another computer attached to your computer anyway.. [0] https://github.com/jhuber6/doomgeneric https://github.com/jhuber6/doomgeneric
rakejake 2y ago
I have a problem similar to what OP faced but with an NVIDIA GPU (RTX 4080). When I wake the computer from suspend, the computer will usually wake up, show me the timestamp changing on the lock screen. But sometimes randomly, the timestamp will not change the the screen will freeze on the old timestamp. After this, it is either REISUB or hard reset. @nyanpasu64, do you think enabling nvidia-suspend/nvidia-resume services will do the trick? I didn't go through the codefix in the above post in detail, but it looks like the fix is to raise a relevant notification to all listeners in the suspend_prepare() method for amdgpus, not nvidia. Running Ubuntu 24.10.
- fata1ity_ 2y ago
  I have encountered this running Ubuntu with an nvidia gpu. Have you tried adding button.lid_init_state=open as a kernel boot parameter? That is what solved it for me.
  - rakejake 2y ago
    Thanks. Will try.
- nyanpasu64 2y ago
  I would try both nvidia-suspend enabled and disabled; https://forums.developer.nvidia.com/t/fixed-suspend-resume-issues-with-the-driver-version-470/187150/3 https://forums.developer.nvidia.com/t/fixed-suspend-resume-i... suggests turning it off, but I'd try with it both on and off.
Namidairo 2y ago
For a second I thought this was referring to the other reset bug on Polaris, Vega and Navi. (These apparently have broken Function Level Reset sequences, requiring quite specific reset code as a separate module or a system reboot to bring back to a working state.)
mkesper 2y ago
Why was dm->cached_state storing -12 instead of a pointer? Most likely this happened because earlier during suspend, dm_suspend() assigned dm.cached_state = drm_atomic_helper_suspend(adev_to_drm(adev)). The callee drm_atomic_helper_suspend() could return either a valid pointer, or ERR_PTR(err) which encoded errors as negative pointers. But the caller function assigned the return value directly to a pointer which gets dereferenced upon resume, instead of testing the return value for an error. One more point for rust in the kernel. Just can't happen if you're required to handle a Result type.
- vlovich123 2y ago
  You can also get algebraic sum types in C with the C preprocessor: https://github.com/Hirrolot/datatype99 https://github.com/Hirrolot/datatype99 But of course defaults matter and the kernel’s rich history of not modernizing coding practices is going to work against improvements in C land. Ironically, it’s that same resistance that frustrates the Rust devs so much because their resistant to even cleaning up their own subsystems or putting down markers documenting how the subsystems are supposed to work. Maybe https://github.com/llvm/llvm-project/issues/74205 https://github.com/llvm/llvm-project/issues/74205 would help once it trickles down into the kernel, but I suspect that people are still going to choose to do this manual overloading of the pointer instead of using types for safety.
sim7c00 2y ago
very nice and detailed writeup, so many interesting stuff in here really. reading about systemd (bugreport) always hurts my brain (bugreport) but all the low level interactions between OS and drivers /firmware regarding these states and what kind of issues can happen between them, how to find out whats happening better, verynice :). many thanks! for the year long hunt and the excellent writeup.
isodude 2y ago
There should exist something like memtest86, but for S3 and S0, that you can run on the laptop to identify hardware that do not suspend properly.
gmokki 2y ago
Has anyone tried the AMD suspend/resume patches that Alibaba? https://lists.freedesktop.org/archives/amd-gfx/2025-January/118673.html https://lists.freedesktop.org/archives/amd-gfx/2025-January/... "We have tried to solve these issues case by case, but found that may not be the right way. Especially about the unbalanced irq reference count, there will be new issues appear once we fixed the current known issues. After analyzing related source code, we found that there may be some fundamental implementation flaws behind these resource tracking issues. So we try to fix those issues by two enhancements/refinements to current device management state machines."

772 points by fanf2 2y ago

Fantastic news. AMD's linux graphics drivers have mostly worked great for me but this has been the one exception that I've hit multiple times.

juujian 2y ago
Same great experience, but I experience similar issues when I disconnect thunderbolt with monitors when my machine is asleep. Laptop though, so very different driver set, no GPU via pci
- jorvi 2y ago
  You can probably write a udev rule for the Thunderbolt / USB-C port with either ACTION=="offline", "remove" or "online". "offline" is for when your system turns off or suspends, "online" vice versa, and "remove" is self-explanatory. If you go with "offline", I'd look into hard disconnecting the monitors. This might cause monitor rearrangement (= you'll need to manually assign) or blinking on laptop bootup. Might also stop any charging. But that could be mitigated by checking the "subsystems" attribute. If you go with "online", you probably need run some sort of clean-up / refresh script or rule If you go with "remove", you'll need the same clean-up / refresh script. It'll take some trial-and-error, but it'll be satisfying once it works. Also highly recommended to check the NixOS repositories and official wiki and Arch wiki to see if your laptop or monitors have workarounds for their quirks.
MegaDeKay 2y ago
My luck has been a little less good. Latest problem I'm having is the driver spamming my logs after waking from sleep with "[drm] scheduler comp_1.0.n is not ready, skipping" after "WARNING: CPU: 12 PID: 11871 at drivers/gpu/drm/amd/amdgpu/../display/dc/dc_helper.c:100 generic_reg_update_ex+0x1d2/0x290 [amdgpu]" https://gitlab.freedesktop.org/drm/amd/-/issues/3911 https://gitlab.freedesktop.org/drm/amd/-/issues/3911
- binkHN 2y ago
  I have something similar with dmesg spam and a possibly related issue, sadly: https://gitlab.freedesktop.org/drm/amd/-/issues/3790 https://gitlab.freedesktop.org/drm/amd/-/issues/3790
  - MegaDeKay 2y ago
    Using an ICC profile per the last comment in that report didn't fix it for you?

jorvi 2y ago

> Through some digging, I found that when a desktop enters S3 sleep, the system cuts power to PCIe GPUs I am not sure how correct this assumption is. S3 is supposed to cut power to everything but RAM, but for example Gigabyte Aorus motherboards are notorious for an NVMe SSD sleep bug that randomly prevents the system from properly sleeping or waking. This is fixed by adding the following udev rule: # Generic PCIe fix for sleep bugs by preventing wakeup from any PCIe port ACTION=="offline", SUBSYSTEM=="pci", DRIVER=="pcieport", ATTR{power/wakeup}="disabled" or more targeted: # Gigabyte sleep fix by preventing wakeup from problematic PCIe port, depends on motherboard model ACTION=="offline", SUBSYSTEM=="pci", ATTR{vendor}=="0x8086", ATTR{device}=="0x43bc", ATTR{power/wakeup}="disabled" You can find any glitched PCIe wakeup device with: 1. cat /proc/acpi/wakeup (you'll have to trial and error your way through the wakeup devices if it isn't immediately clear) 2. cat /sys/class/pci_bus/*/*/yourWakeupDevicePci/uevent | grep PCI_ID 3. prepend "0x" You also have the option of: udevadm info --attribute-walk /dev/whatever but for that you need to know some basic identifier of your glitchy device. Or if you want to shellscript it (less reliable than letting udev do it for you and needs to be done via systemd service file or another automation): # Gigabyte sleep fix, port depends on mobo model /bin/bash -c 'if grep 'RP05' /proc/acpi/wakeup | grep -q 'enabled'; then echo 'RP05' > /proc/acpi/wakeup; fi'"; Yes I really hate this (and other) Linux sleep issues.

VMG 2y ago
As somebody with an Aorus motherboard who has probably burned a few kWh on this issue, I was really excited to try these solutions - no luck. Thank you anyway!
- jorvi 2y ago
  Did you try the general fix? And reload udev rules? You also have to make sure it applies after the default rules. You can check if the rule applies once you have everything set up by doing an `udevadm` attribute walk of your SSD device (not partition), and then following it up all the way up the device tree until you see your specific device port (target fix) or PCIe driver subsystem (general fix). Then check if "power/wakeup" is set to "disabled". If it is set to disabled, something else is keeping your device awake on sleep. For that you can check /proc/acpi/wakeup, and there's also a specific systemd invocation (that I forgot) you can do that shows if your device slept, how long it slept, how much battery was drained, and if your device woke-up, slept or failed to resume, it'll give you a reason.. to the best of its ability.

Gormo 2y ago

My sincere personal thanks for this. My main laptop is a Ryzen-based ThinkPad running Linux that I suspend and hibernate regularly, and I sporadically encounter this issue. Looking forward to 6.14!

imp0cat 2y ago
This. Thanks a lot!

raffraffraff 2y ago

> I dug a PS/2 keyboard out of a dusty closet and plugged it into my system (only safe when the PC is off!) Lol, I remember.

yellow_lead 2y ago

I have an Nvidia GPU and a sporadic crash (black screen) with no logs on Linux. I suspect it's a driver issue too. Going to try some of these tips to enable the debug shell, but I'm not sure if they'll be effective. Anyone have other tips for this type of thing? I did try upgrading drivers/kernels already

an_ko 2y ago
Run a memtest. Graphics cards usually crash badly when given invalid data, which can happen sporadically if you have bad RAM. If memtest shows a specific memory region as failing, swap out sticks to check which it is, and buy a new one. (Or if you're on a tight budget, you can disable that region with kernel boot options.) If memtest gives errors in lots of places, might be a bad overclock. Loosen timings or give it more voltage.
- bmicraft 2y ago
  > If memtest shows a specific memory region as failing, swap out sticks to check which it is, and buy a new one. (Or if you're on a tight budget, you can disable that region with kernel boot options.) Honestly, I'd always do that as the first option. In most cases you can still get years of life out of that stick of ram (but do check regularly, like after a week and then double the interval every time if it didn't get worse).
- porridgeraisin 2y ago
  This. I would also recommend running it on one memory slot only (do this one by one). Recently it happened that even a multi-day memtest run did not catch the defect in one of my ram slots. But when I emptied it, all the gpu driver problems immediately stopped. The symptom was a crash followed by garbage on screen and a "zzzz" sound. Sometimes followed by a shutdown.
- yellow_lead 2y ago
  Thanks, I'll try it!
nyanpasu64 2y ago

empiricus 2y ago

I notice I am confused how the code needed for the GPU to sleep was implemented. It was failing when simply saving/copying gigabytes of flat memory, but on the other hand it was able to recover successfully the previous complex hw and sw state and data structures?! I guess it probably makes sense if after waking up that data is actually dropped and the gpu and ui is reinitialized and redrawn.

nyanpasu64 2y ago
As I understand if the GPU fails to save VRAM it drops the RAM copy(?) before restoring the GPU data structures or abandoning sleep entirely. But if it saves VRAM then crashes after the GPU is suspended, it will often fail to wake up the GPU, resulting in no monitor signal. Though I'm fuzzy on the details (and the order of operations depends on which kernel you're running).

kkarpkkarp 2y ago

omg, thank you

Asmod4n 2y ago

> On my laptop, I opened a terminal and ran sudo minicom --device /dev/ttyUSB0 --baudrate 115200 to monitor the computer over serial. In addition to saving logs You can just use screen for that and have a working terminal with color support et al.

vbernat 2y ago
Another lightweight option is tio.

deleted 2y ago

[deleted]

0xTJ 2y ago

Very excited to see 6.14 hit Arch! Hanging around sleep (with symptoms that sound like what's described in the write-up) has been the one persistent occasional issue, so I'm hoping that this fixes what I'm seeing.

progforlyfe 2y ago

Extremely high level genius stuff -- nice work and thank you for your efforts!

sabujp 2y ago

This is amazing work! If folks have ever wondered why suspend is so difficult to get working on linux and why debugging it is equally difficult, this is a single datapoint with lots of information about all the things that can go wrong. Even now I have a thinkpad P1G4 where the fans won't turn off automatically unless I turn them off before going into suspend. Recently I also started having crackling issues with my bluetooth headphones after resuming from suspend and had to disable node suspension there also (https://wiki.archlinux.org/title/PipeWire#Noticeable_audio_delay_or_audible_pop/crack_when_starting_playback https://wiki.archlinux.org/title/PipeWire#Noticeable_audio_d...).

Apofis 2y ago
Remarkable that it's 2025 and laptop sleep/suspend still doesn't work right on linux. I think the first time I encountered this was probably 15 years ago now?
- bongodongobob 2y ago
  It doesn't work right on Windows either to be fair. With a mixed laptop fleet at work, we've just disabled sleep/hibernate company wide because it causes way too many problems.
  - larrik 2y ago
    That seems like a good way to cook your laptop when you throw it into a bag with the CPU pegged.
    - bongodongobob 2y ago
      Not if you turn it off first.
      - Joker_vD 2y ago
        For some reason, people really insist on being able to put their machines to sleep, I honestly don't know why. Long boot times, maybe?

Voultapher 2y ago

Since a couple Linux versions something around 6.10 IIRC I've had it where my Nvidia system wakes into a black screen, but with a cursor and alt shells work, specifically KDE Plasma seems bugged here, but they say it's a kernel issue, or at least there are dozens of separate issues open about this kind of bug and it's rather annoying that I can't put my machine to sleep. If anyone has ideas what could fix this I'd really appreciate it. The machine is dual booted with Windows, and there sleep works without issue, so it's clearly possible, as it was for years before that on Linux as well.

thangalin 2y ago
I have a similar issue. When turning off the monitor using its power button, the system semi-crashes. After powering the monitor back on, I have to go into an alt shell and kill lightdm, which also kills all running GUI applications. Not the greatest workflow. $ uname -a Linux hostname 6.13.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 02 Feb 2025 01:02:29 +0000 x86_64 GNU/Linux NVIDIA-SMI 570.86.16 Driver Version: 570.86.16 CUDA Version: 12.8 Instead of powering off the monitor, I've created an alias: alias off='xset dpms force off && await.sh && xset -dpms' And shell script (press any key to unblank the screen): $ cat ~/bin/await.sh #!/usr/bin/env bash xinput test-xi2 --root 2>&1 | \ grep --line-buffered -m 1 'EVENT type 2 (KeyPress)' &> /dev/null This allows me to run "off", a comromise to put the monitor in low-power mode. This solves the problem of having to kill lightdm, which improves the workflow. Still, sometimes I have to open an alt shell and then press Alt+F7 to return to X. Rarely, I'll have to go to the alt shell/Alt+F7 a second time to suppress sporadic screen glitching. Adding nomodeset to the GRUB configuration was another suggestion: # grep vmlinuz /boot/grub/grub.cfg | head -1 linux /vmlinuz-linux root=UUID=... rw nomodeset quiet resume=UUID=... Hope it helps.
- saltcured 2y ago
  Now I'm not sure if my Fedora 41 experience is better or worse than your problem. An older Thinkpad with secondary NVIDIA dGPU used to work fine, but now every time the monitor powers off (or is unplugged), Xorg instantly dies with no apparent logs. So I don't have to do anything special to recover except login over again and start a whole new MATE session.

jph 2y ago

Your work will help me on a Framework AMD laptop with the GPU extension and dual boot Linux/Windows. May I donate to you or to your favorite charity? My contact info is in my profile.

lelandfe 2y ago
Love this!

zrm 2y ago

> To make room for VRAM, memreserver allocates system RAM based on used VRAM plus 1 gigabyte, then fills the RAM with 0xFF bytes and mlocks the memory (so none of it is swapped out). That seems like a bit of trouble if you have 16GB of system RAM and a 24GB GPU.

deleted 2y ago

[deleted]

Daunk 2y ago

For all the years I've been using Linux, I've always had some kind of sleep issues. I've used Intel, AMD, ATI, and NVIDIA hardware across countless distros and setups, yet nothing seems to make a difference, there's always something that doesn’t work properly with sleep or hibernation. Honestly, it's one of the main issues I wish the Linux community would take a closer look at and finally fix!

Narishma 2y ago
I think it it's because there are too many subsystems involved in sleep/resume all being worked on as independent projects (kernel, drivers which sometimes have both kernel and user space components, init system, display server, desktop environment, probably others I'm not aware of). That said, I've had my share of sleep issues on Windows as well over the years, I suspect for the same reasons.
1970-01-01 2y ago
This. Linux users must resort to bronze-age tooling in 2025; Crafting and launching handmade scripts by candlelight to diagnose their plethora of sleep issues. But the community likes it this way. Meanwhile, Mac users continue to have sleep that 'just works' and Windows users have an entire sleep troubleshooting toolkit: https://learn.microsoft.com/en-us/windows-hardware/design/device-experiences/tools-available https://learn.microsoft.com/en-us/windows-hardware/design/de...
- nyanpasu64 2y ago
  I once ran into a corporate laptop that had been downgraded from Windows 11 to 10, that would burn up its CPU during Modern Standby and eventually enter hibernate after burning a good fraction of its battery. The sleep study identified various PCIe devices and I tried installing drivers but they did not help. I wonder if it would've worked better on Windows 11 with stock drivers.
kiwijamo 2y ago
Interestingly sleep/wake is something I've found to work almost always just fine out of the box in Linux, including on machines Windows has sleeping issues! It used to be quite bad but things has improved heaps over the last 10 or so years -- however I've also stuck with Lenovo laptops which does generally seem to have better support in Linux.

sidkshatriya 2y ago

TL;DR: During suspend, for graphics cards, GPU VRAM needs to be transferred to system RAM. However, during high memory usage scenarios the VRAM + RAM usage could exceed system memory -- this would ordinarily involve system swap coming into play and handling the temporarily out of memory issue. However system swap was already deactivated when it came time to suspending the AMD card causing all sorts of problems. The fix was asking the GPU to evict its VRAM to system RAM via the hook ("suspend prepare") before swap was deactivated in linux kernel.

nyanpasu64 2y ago
<s>Technically it was the suspend notifier; even suspend prepare executes after swap is disabled.</s> See replies.
- sidkshatriya 2y ago
  Isn’t the hook called PM_SUPEND_PREPARE as per https://nyanpasu64.gitlab.io/blog/amdgpu-sleep-wake-hang/#solution-power-management-notifiers https://nyanpasu64.gitlab.io/blog/amdgpu-sleep-wake-hang/#so...
  - nyanpasu64 2y ago
    Sorry I had forgotten my own article :( It is confusing that PM_SUSPEND_PREPARE and dpm_prepare share the same word.

ej1 2y ago

[dead]

whatever1 2y ago

Apple became a trillion dollar company by mastering sleeping / waking up of electronic devices. why nobody else sees this?

edoceo 2y ago
That and app store fees
deleted 2y ago
[deleted]
rafaelmn 2y ago
I still have i9 2018 that will drain overnight in sleep mode, and that sleep discharge put the most battery cycles on the device. I think they only fixed it when they ditched the x86 ecosystem.
megous 2y ago
Pretty much all smartphones can do this.
- whatever1 2y ago
  *Recent. It took many years for smartphones to catch up with iPhones in terms of energy drain. Even today iPhones have the smallest batteries across their competitors.
  - megous 2y ago
    Battery life has nothing to do with suspend/resume reliability. Every smartphone does suspend/resume thousands of times each day, it has to be nearly 100% reliable feature and it is, otherwise users would be majorly pissed off.
meowkit 2y ago
Because they own the now hardware, and MacOS iOS run exclusively on their hardware Source: I work on windows power management and I know system engineers at apple.

voytec 2y ago

I've had zero problems with S3 wake/sleep on AMD ThinkPad with FreeBSD for years. And FreeBSD uses AMD drivers pulled from Linux. How is this still a problem on Linux? hw.acpi.lid_switch_state=S3

larrik 2y ago
I battled sleep issues on my laptop for months, and had very different results than most people with my model. I think the behavior is a total crapshoot from machine to machine. In fact, it turned out that a BIOS update that happened in the middle of my issues broke sleep functionality for the whole machine for a few months, so that wasn't even my fault.
kiwijamo 2y ago
Ditto. My Thinkpad X1 and X390 both have reasonably reliable sleep/wake under Debian.

nyanpasu64 2y ago

Update: I upgraded to an Intel Arc B570 GPU... and ran into the exact same problem on an independent driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/4288 https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/4288

KronisLV 2y ago
Oh hey, another Intel Arc user! I got the LE B580 and so far it's been a pretty good experience for me, though I don't really use suspend. Also I got curious about your blog (haven't heard about Zola, recently migrated my own blog over to a newer version of Grav, which is a flat file CMS with no DB) and saw your post about looking for a new job. Hope things work out for you, stay safe!
- nyanpasu64 2y ago
  All the non-scalped B580s were out of stock by the time I decided to buy a GPU :( Thanks for the moral support!

tgsovlerkhgsel 2y ago

AMD GPU linux drivers are (were?) a nightmare in general, and this includes iGPUs in their processors. Sadly, I don't have the impression that AMD is actively working on fixing this. Just to make sure I'm not griping over something long fixed, I took a quick look and instantly found someone with a very similar issue to the one I ran into happening on a semi-recent kernel: https://community.amd.com/t5/pc-drivers-software/linux-amdgpu-ring-gfx-timeouts-in-various-applications/td-p/723574 https://community.amd.com/t5/pc-drivers-software/linux-amdgp... It looks to me that if you want to have a working computer under Linux, it's worth the extra cost to avoid AMD.

tostiheld 2y ago
> It looks to me that if you want to have a working computer under Linux, it's worth the extra cost to avoid AMD. I think this is a rather hasty conclusion. The popular opinion is the opposite. If you want a working computer under Linux, it's worth it to avoid NVIDIA, especially for laptops. Sure, AMD are not perfect contributors to the kernel, but they are contributing more than NVIDIA[0]. NVIDIA has made some moves recently[1], but the AMD GPUs are still better integrated. Notably since the Steam Deck had been released, the situation has been excellent. Anecdotally, my laptop with an NVIDIA GPU has many issues that have persisted over the years with things like high idle power draw or frequent straight up crashes, or incomplete Wayland support. My 3 devices that have an AMD GPU (1 desktop, 2 laptops) however, have been working flawlessly from day 1. [0] https://www.phoronix.com/news/NVIDIA-Contributions-2010s-Kern https://www.phoronix.com/news/NVIDIA-Contributions-2010s-Ker... [1] https://github.com/NVIDIA/open-gpu-kernel-modules https://github.com/NVIDIA/open-gpu-kernel-modules
- MegaDeKay 2y ago
  It depends on the use case. AMD is notorious for their "AMD reset bug" when passing a GPU through to a VM using VFIO. Restart the guest and most cards will lock up because AMD doesn't handle PCI resets properly. You then need to reboot the host to fix it (!). This has been a problem since Polaris if not before and AMD hasn't fixed it, despite knowing full well that the problem exists. At least in this regard, NVIDIA (and Intel as far as I know) work fine. The community has been able to come up with a workaround for some older cards but the problem persists even in their current cards. https://github.com/gnif/vendor-reset https://github.com/gnif/vendor-reset

_cenw 2y ago

Some AMD integrated GPUs are surprisingly fragile with this. I have a GPD Win Max 2 8840U (a "concept car" handheld laptop hybrid) and when I got it last year, it would fail to wake from suspend and hibernate about half the time in Windows, with Linux actually being more reliable (but also not perfect), and only this year did an AMD GPU driver fix this.

jchw 2y ago

Memory management and specifically OOM conditions remain an unbelievably painful nightmare on Linux. It's not like I run into these issues constantly, but I've definitely tried to debug issues like these (unsuccessfully). Ultimately if I OOM a machine I usually wind up installing more RAM, which is wasteful/expensive, but it's pretty clear that handling OOM conditions gracefully is going to be a hard problem for Linux to solve into the future. This is really great work and will serve as a reference point for debugging similar issues in the future. Pretty happy about systemd's debug-shell feature, I had no idea that existed. I don't think my X670E Steel Legend board has a serial header anywhere on it, though. How do modern built-in serial ports work, anyway? Are they attached off of the chipset PCIe lanes? Something that's also very useful when trying to dive into the Linux kernel is that there's a bunch of great talks discussing Linux kernel subsystems from conferences like FOSDEM and Linux Plumber's Conference which you can usually find recordings of online. For example, there's this one for TTM, the memory subsystem that most of the desktop GPU DRM drivers use: https://www.youtube.com/watch?v=MG7_tUNKSt0 https://www.youtube.com/watch?v=MG7_tUNKSt0

Skunkleton 2y ago
I’ve had good luck containing ooms with cgroups. I’m not sure if there is a state of the art for handling oom conditions beyond what Linux does. If anyone knows and can recommend some reading I would appreciate it.
- jchw 2y ago
  There's really two problems as I understand it: - Overcommit. Linux will "overcommit" memory: allocations will succeed when there's no memory, and then hang when the page is actually mapped if no physical pages are available (to my understanding.) Windows NT doesn't do this. Not sure exactly how macOS/XNU handles it. - The OOM killer. Because allocations don't fail, to actually recover from an OOM situation the kernel will enumerate processes and try to kill ones that are using a lot of memory, by scoring them using heuristics. The big problem? If there isn't a single process hogging the memory, this approach is likely to work very poorly. As an example, consider a highly parallel task like make -j32. An individual C++ compiler invocation is unlikely to use more than a gigabyte or two of memory, so it's more likely that things like Electron apps will get caught first. The thrashing of memory combined with the high CPU consumption of compilers that are not getting killed will grind the machine to a near-complete halt. If you are lucky, then it will finally pick a compiler to kill, and set off a chain reaction that ends your make invocation. There are solutions... Indeed, you can use quotas with cgroups. There's tools like systemd-oomd that try to provide better userspace OOM killing using cgroups. You can disable overcommit, but some software will not function very well like this as they like to allocate a ton of pages ahead of time and potentially use them later. Overcommit fundamentally improves the ability to efficiently utilize all available memory. Ultimately I think overcommit is probably a bad idea... but it is hard to come up with a zero-compromises solution that keeps optimal memory/CPU utilization but avoids pathological OOM conditions by design.

mistyvales 2y ago

Highly relevant! Thanks for this.

dekhn 2y ago

I used to think that naming things, cache invalidation, and off-by-one errors were the 2 biggest problems in CS, but then I learned about the "sleep/wake" problem and realized it's NP-complete.

nikanj 2y ago
Only on Linux though, on Windows it’s O(n2) and on Macos it’s O(log n)
- itsn0tm3 2y ago
  Well only as long as you don‘t hackintosh. That stuff used to be a horror sometimes!
  - 0x38B 2y ago
    Remembering the hours I spent going through KEXTs and bootloader config in ill-fated attempts to set up a Hackintosh fills me now with a kind of horror. Worst of all were the ACPI tables - SSDT and the like. In contrast to that, running MacOS in a VM is heaven. Figuring out how to pass my iPhone through to Xcode took as long as the initial setup.
- ncann 2y ago
  With how much trouble I had with trying (and failing) to make my brand new Dell laptop sleep properly and not the "Modern Standby" crap, plus my desktop randomly breaking GPU hardware acceleration in browser after waking up, I would say it's around O(n4) now. Or maybe even O(n!).
verall 2y ago
I think sleep/wake is a subset of cache invalidation - if all peripherals were stateless, probably it wouldn't be an issue.

deepsun 2y ago

I have to unplug Logitech wireless receivers for mouse/keyboard, otherwise desktop wakes up immediately.

podiki 2y ago

I didn't have issues with sleep/wake until somewhat recently (not sure when) and found this post. Grabbing the patch from the commit referenced and using it on top of 6.12 and 6.13 kernels seems to have fixed it for me too (for the past couple of weeks and counting). Great work!

Bobaso 2y ago

My thinkpad E14 gen5 intel + ubuntu 24.04 is my first ever laptop where sleep work exactly as intended. with ~1% battery waste per hour

tombot 2y ago

> This took over a year of debugging and multiple attempts by many people to fix. 2025 finally Linux on the desktop

Nezghul 2y ago
I'm also programmer and such bugs immediately reminds me of all my managers demanding exact time it would take me to fix such bug and me telling them it could take from 1 day to 1 year was never taken seriously :(

devilsdata 2y ago

Props to you for this. I am not clever/experienced enough to solve my own issues with sleep-wake hangs on Linux at work. I’ve instead opted to work around it. I use Firefox, Obsidian, and Tmux with Neovim for all my work. Tmux has resurrect and a plugin that saves my entire terminal state automatically every few minutes. I also have a command that automatically sets up my i3wm/regolith windows exactly how I like. Basically if I run `wkup`, I’m exactly where I was, down to the line of code open on NeoVim, Firefox tab, and dev server or cargo running.

bbkane 2y ago
Could you link to your i3wm/regolith window setup code? I'd like to set up something similar!
- devilsdata 2y ago
  Here we go: https://github.com/lkdm/dotfiles/blob/main/bin/executable_wkup https://github.com/lkdm/dotfiles/blob/main/bin/executable_wk... Hope it helps you out

lorenzbrun 2y ago

Author of memreserver (one of the mentioned userspace workarounds) here. I've debugged this a few years back, only public comment I can quickly find is [1]. I also remember some mailing list discussions, but it basically came down to the isuse that Linux didn't have staggered suspend hooks that reliably ran before disks and parts of the memory subsystem were frozen. Apparently this is now possible. Sadly the Freedesktop Gitlab doesn't seem indexable so this knowledge seems to have gotten lost. [1] https://gitlab.freedesktop.org/drm/amd/-/issues/2125#note_1708592 https://gitlab.freedesktop.org/drm/amd/-/issues/2125#note_17...

badsectoracula 2y ago

I wonder if this will help a similar problem i have with my AMD GPU: very often when i wake/resume the PC, the output is almost frozen. "Almost" because it actually isn't frozen, if there is any output/animation/etc going on it plays fine, but once i try to move the mouse it freezes and everything updates at a single frame per couple of seconds - sometimes freezing completely. I can usually Ctrl+Alt+Fn to another virtual desktop in text mode and, if that is possible (i.e. the computer hasn't completely frozen, though sometimes it takes about a minute to switch), i can Ctrl+Alt+Fn back and everything works fine. Dmesg has a ton of spam messages from amdgpu after that. AFAICT (from the behavior) something isn't properly saved/restored and communicating with the GPU (the mouse cursor is a hardware cursor thus needs to send commands to the GPU to update its position) causes some sort of issue. Switching to another virtual terminal that is running in text mode probably forces the driver to reset its graphics state. Of course that is just my assumption based on what i see going on. Weirdly enough this only happens after i replaced my RX 5700 XT with a RX 7900 XTX so it might be something GPU (or GPU arch) specific. I've been considering plugging my laptop and see if there is something i can figure out (GPU aside the PC is usable, but i guess if this a kernel side thing i'd need a second computer connected to it to debug it), but as this isn't something i've tried before (though i know someone who has and said it isn't anything special) my annoyance still hasn't gone over the "i need to get to the bottom of this" threshold :-P. It'd be nice if 6.14 fixes the issue, though i am not sure as i rarely have more than 1/3rd of the system RAM (32GB) in use and VRAM (24GB) barely goes above 1-2GB of use outside games. But this post might be helpful in diagnosing the issue next time it happens :-).

nyanpasu64 2y ago
I've heard reports of reduced frame rate issues on laptops from DMUB/DMCUB/panel self-refresh, though your issue sounds different and I probably can't usefully debug it without a full dmesg/journalctl --system log.

schainks 2y ago

Ugh the first time I was debugging problems like this, it was in production and for some IoT hardware we had deployed in the field. Fortunately, although that's not the focus of this article, system hibernate is WAY more reliable than system sleep in Linux due to the way it works. Use system hibernate if your SSD is fast enough. It works better than system sleep and isn't a ton slower.

igtztorrero 2y ago

Thanks, this happen to me, I avoided sleep functions

dado3212 2y ago

> So I did the natural thing: I saved and extracted the amdgpu.ko kernel module, decompiled it in Ghidra, and mapped the location of the crash in dm_resume to the corresponding lines in the kernel source. This is always my favorite part of debugging.

thrdbndndn 2y ago

what is 'agd5f/linux'?

nyanpasu64 2y ago
As I understand https://gitlab.freedesktop.org/agd5f/linux https://gitlab.freedesktop.org/agd5f/linux is Alex Deucher (amdgpu maintainer)'s personal Git tree/repo (https://docs.kernel.org/process/maintainers.html#radeon-and-amdgpu-drm-drivers https://docs.kernel.org/process/maintainers.html#radeon-and-...), where amdgpu changes get merged into before they reach the main Linux tree (and/or the drm/tip repo, I'm not sure their relationship).

fowl2 2y ago

I guess "hibernating" (writing VRAM to swap) works better than expecting userspace to gracefully handle device resets. One linear read vs. a thundering herd of processes re-initialising, decompressing, etc. should be more predictable/reliable at least. I do wonder however how much VRAM is "volatile" - ie. framebuffers - and could just be thrown away. And web browsers seem to handle GPU resets just fine, so maybe they could opt-in?

nyanpasu64 2y ago
I didn't mention in the article but the Nvidia drivers at one point would drop VRAM rather than preserving it, leading to corrupted RGB noise textures in window managers and browsers (and potential crashes though I don't think I encountered them). I suggested doing this on the AMD bug tracker (https://gitlab.freedesktop.org/drm/amd/-/issues/2362#note_2094417 https://gitlab.freedesktop.org/drm/amd/-/issues/2362#note_20...), but the amdgpu developers weren't interested.
- fowl2 2y ago
  Oh wow didn't really expect anything other than whole device loss. Just returning garbage does sound bad. Do you know if they tried to communicate with clients and were just ignored/not implemented or if the APIs just don't support it? A quick search indicates that "residency"[1] exists, but no idea the extent it's useful/implemented. [1] https://learn.microsoft.com/en-us/windows/win32/direct3d12/residency https://learn.microsoft.com/en-us/windows/win32/direct3d12/r...
  - nyanpasu64 2y ago
    As I understand, KWin does call glGetGraphicsResetStatusARB(), and on Nvidia GPUs, sleep-wake (or opening apps if it crashed the GPU) would cause KDE to detect a graphics reset and print "Desktop effects were restarted due to a graphics reset" (https://github.com/KDE/kwin/blob/10c04995c1f9f82ddbd6610e5e007efe9d8af13b/src/compositor_x11.cpp#L457 https://github.com/KDE/kwin/blob/10c04995c1f9f82ddbd6610e5e0...). I haven't used Nvidia GPUs in years and don't know if this is still an issue. I think many apps don't check for graphics resets? I'm not sure if residency is relevant here; the Microsoft link indicates that eviction makes memory inaccessible from the GPU to make room for other memory, which explains why Linux uses the same name for "backing up" VRAM before sleep (through the same underlying mechanism).

stycznik 2y ago

It seems like the GPU should grow the capability to keep its VRAM intact through suspend, it's already complex[0] enough it's basically another computer attached to your computer anyway.. [0] https://github.com/jhuber6/doomgeneric https://github.com/jhuber6/doomgeneric

rakejake 2y ago

I have a problem similar to what OP faced but with an NVIDIA GPU (RTX 4080). When I wake the computer from suspend, the computer will usually wake up, show me the timestamp changing on the lock screen. But sometimes randomly, the timestamp will not change the the screen will freeze on the old timestamp. After this, it is either REISUB or hard reset. @nyanpasu64, do you think enabling nvidia-suspend/nvidia-resume services will do the trick? I didn't go through the codefix in the above post in detail, but it looks like the fix is to raise a relevant notification to all listeners in the suspend_prepare() method for amdgpus, not nvidia. Running Ubuntu 24.10.

fata1ity_ 2y ago
I have encountered this running Ubuntu with an nvidia gpu. Have you tried adding button.lid_init_state=open as a kernel boot parameter? That is what solved it for me.
- rakejake 2y ago
  Thanks. Will try.
nyanpasu64 2y ago
I would try both nvidia-suspend enabled and disabled; https://forums.developer.nvidia.com/t/fixed-suspend-resume-issues-with-the-driver-version-470/187150/3 https://forums.developer.nvidia.com/t/fixed-suspend-resume-i... suggests turning it off, but I'd try with it both on and off.

Namidairo 2y ago

For a second I thought this was referring to the other reset bug on Polaris, Vega and Navi. (These apparently have broken Function Level Reset sequences, requiring quite specific reset code as a separate module or a system reboot to bring back to a working state.)

mkesper 2y ago

Why was dm->cached_state storing -12 instead of a pointer? Most likely this happened because earlier during suspend, dm_suspend() assigned dm.cached_state = drm_atomic_helper_suspend(adev_to_drm(adev)). The callee drm_atomic_helper_suspend() could return either a valid pointer, or ERR_PTR(err) which encoded errors as negative pointers. But the caller function assigned the return value directly to a pointer which gets dereferenced upon resume, instead of testing the return value for an error. One more point for rust in the kernel. Just can't happen if you're required to handle a Result type.

vlovich123 2y ago
You can also get algebraic sum types in C with the C preprocessor: https://github.com/Hirrolot/datatype99 https://github.com/Hirrolot/datatype99 But of course defaults matter and the kernel’s rich history of not modernizing coding practices is going to work against improvements in C land. Ironically, it’s that same resistance that frustrates the Rust devs so much because their resistant to even cleaning up their own subsystems or putting down markers documenting how the subsystems are supposed to work. Maybe https://github.com/llvm/llvm-project/issues/74205 https://github.com/llvm/llvm-project/issues/74205 would help once it trickles down into the kernel, but I suspect that people are still going to choose to do this manual overloading of the pointer instead of using types for safety.

sim7c00 2y ago

very nice and detailed writeup, so many interesting stuff in here really. reading about systemd (bugreport) always hurts my brain (bugreport) but all the low level interactions between OS and drivers /firmware regarding these states and what kind of issues can happen between them, how to find out whats happening better, verynice :). many thanks! for the year long hunt and the excellent writeup.

isodude 2y ago

There should exist something like memtest86, but for S3 and S0, that you can run on the laptop to identify hardware that do not suspend properly.

gmokki 2y ago

Has anyone tried the AMD suspend/resume patches that Alibaba? https://lists.freedesktop.org/archives/amd-gfx/2025-January/118673.html https://lists.freedesktop.org/archives/amd-gfx/2025-January/... "We have tried to solve these issues case by case, but found that may not be the right way. Especially about the unbalanced irq reference count, there will be new issues appear once we fixed the current known issues. After analyzing related source code, we found that there may be some fundamental implementation flaws behind these resource tracking issues. So we try to fix those issues by two enhancements/refinements to current device management state machines."