5 ms·

Sorry, Wrong Number: Debugging a Crash Under Wine

120 points by jchw 5y ago

anaisbetts 5y ago
This is great debugging but I wonder why the author immediately drops to debugging Wine directly, rather than running WinDbg/cdb under Wine and trying to debug the app via this way first. This feels a bit like debugging your webpage JS via attaching lldb to Chromium!
- jchw 5y ago
  Actually, I do have an answer for that: debuggers don’t work very well under Wine. It would be nice to get them working better. The standard tool to use for Wine is winedbg, but as I wrote, this doesn’t work here because there’s no way to break before the application entrypoint (which is too late.)
  - anaisbetts 5y ago
    That's a good reason!
develatio 5y ago
But what happened with the app your friend was trying to run?? Were you able to patch it and make it work? (and if "yes", how?)
- jchw 5y ago
  Ahh, sorry I never completed the story. The program was simply fixed. I believe the way they wound up fixing it was simply static linking libpng and zlib. I found another workaround too, which was a very obscure linker flag, but I can’t recall what it was as it has been a few months now. If you run into this problem with a random binary and can’t get it fixed properly, you can use Microsoft’s REBASE.EXE or EDITBIN.EXE to rebase the preferred base address to be within the range of a RIP relative CALL. If you are savvy with PE, writing such a tool by hand is very straightforward; you need to write a PE parser that can map virtual addresses, relocate as if you are a real PE loader, update the preferred base address, and write back to disk. The only case where something like this won’t work is when the executable has some kind of on-disk CRC checking, or something along those lines. (So probably, VMProtect and Themida encumbered binaries can’t be modified this way. But that’s OK, as most tools of this sort only works with Microsoft’s toolchain, which does not have pseudo-relocs.)
  - marklar423 5y ago
    Out of curiosity, if the tool was compiled for MinGW doesn't that mean it would run natively on Linux? Why was your friend trying to run the MinGW version?
    - jchw 5y ago
      MinGW does allow you to write Windows-specific code. The code in question could be ported, and in fact I think it might be now, but it still puzzled us why it didn’t work on Wine. I only wanted to help on this particular avenue for the sake of making Wine better. I had figured that it was most likely an issue with Wine that could be rectified. Unfortunately, emulating Windows ASLR behavior in userspace accurately may prove to be fairly difficult, so it’s probably not worth it :) I’m not convinced pseudo relocs actually work reliably on Windows in this way, although I could be wrong.
k1t 5y ago
This (2013) article[1], when talking about "High Entropy Bottom-up Randomization" says that: "For compatibility reasons, this feature is disabled by default and must be enabled on a per-application basis. This is because some 64-bit applications have latent pointer truncation issues that can surface when dealing with pointers above 4 GB (significant bits set beyond bit 31)." Although OP had an issue with an offset rather than an absolute position, I imagine it would also handle that case. [1] https://msrc-blog.microsoft.com/2013/12/11/software-defense-mitigating-common-exploitation-techniques/ https://msrc-blog.microsoft.com/2013/12/11/software-defense-...
khuey 5y ago
If you already had this in rr, you should have been able to set a hardware watchpoint on 0x36810e3eb and just reverse-continue to the point where the relocation was applied.
- jchw 5y ago
  This is a good point, but I feel like there was a reason that I couldn’t do this. With Wine under rr, the replays were a bit tricky to get working in the first place. I think it is possible i could’ve done this still, but I first needed to find the number of events to skip for that part of the replay. See this issue: https://github.com/rr-debugger/rr/issues/1687 https://github.com/rr-debugger/rr/issues/1687 By the time I got to the point where a reverse continue would’ve been most useful, I’d already left rr due to this issue.
- pm215 5y ago
  Similarly, instead of doing "a few hundred stepis" to get to the call insn that jumps to a garbage address, you can let execution run to the crash point and the reverse stepi once. The "reverse" functionality parts of rr are one of its strongest features and well worth the time investment to get acquainted with them. (I bet my own debugging workflows and habits are stuffed full of this kind of "works but is definitely not optimal" move, though, so I don't mean this as a criticism of the article.)
vgel 5y ago
I didn't know about GDB follow-fork-mode child / follow-exec-mode new! I always just fiddle around trying to launch the child process directly to debug it.
Scramblejams 5y ago
Semi-related: Is anyone aware of a Proton neophyte's guide to fixing a Windows game? I've got a few old games that get stuck various places and I imagine that there may be a simple Windows call that needs implementing, but I don't know the first thing about how to diagnose that.
- iforgotpassword 5y ago
  I once had a game that always crashed under wine. Instead of trying to understand the bug, I got as far as figuring out that this was a virtual function call into nowhere and then just 909090'd the crap out of it. The game then seemed to work perfectly and I couldn't even tell what was missing/different after removing a totally random function call.
- coldpie 5y ago
  I wrote an intro to Wine dev: https://www.codeweavers.com/blog/aeikum/2019/1/3/working-on-wine-part-1-the-wine-ecosystem https://www.codeweavers.com/blog/aeikum/2019/1/3/working-on-... It doesn't go into a lot of debugging detail, as that's its own huge topic, but may get you started. If you're debugging a thing in Proton, you'll want to use that project environment instead of Wine's directly, but the concepts are the same.
- simion314 5y ago
  I would start by running the game from the terminal and read all warnings and errors, compare those with games that work because some of the warnings are always there but don't affect the games.
- mike_hearn 5y ago
  I used to work on Wine, at first as a hobby and then later as a job (for CodeWeavers). As part of that I contributed to the Wine Developers Guide. The Wine docs to help new devs are (or were) remarkably good given the enormous complexity of the project. Unfortunately "how to fix a bug in Wine" can still devolve very rapidly into: Step 1. Learn Windows programming. Step 2. Then learn Linux programming, the Wine architecture, X86 assembly, low level debugging etc. Step 3. Draw the rest of the owl. Nonetheless it's a good place to start: https://wiki.winehq.org/Developer_Hints https://wiki.winehq.org/Developer_Hints https://wiki.winehq.org/Wine_Developer%27s_Guide https://wiki.winehq.org/Wine_Developer%27s_Guide The days when fixing games involved just straight implementation of simple library calls were mostly over, even back when I was doing this a long time ago. Broken apps or games are much more likely to be hitting obscure "bugs" in Wine (which can mean failure to replicate Windows bugs), or other even more subtle compatibility issues, like the one this blog post documents.
rwmj 5y ago
> I never quite figured out how to get debug symbols to map correctly with this Wine-under-GDB setup going on, so I had to manually explore the address space to figure out what I was looking at. Yeah I had the same problem and never solved it. In fact I don't think I even got a handle on what the problem is at all. In what format are symbols created by mingw-gcc (I think PDB??) and in what format either Wine's debugger or GCC can consume symbols (DWARF???)
- rwmj 5y ago
  Edit: For "GCC" I meant of course "GDB"
- jchw 5y ago
  That’s the weird part: MinGW does use DWARF, embedded into PE similar to how it is embedded into ELF, with just one little non-standard trick for long section names. On Windows, GDB supports loading these symbols just fine. But if I forcibly get them to load on Wine under GDB, it doesn’t seem to work even if the load seems to succeed. Confusing.
- KenoFischer 5y ago
  I have a patch that teaches GDB how to do this here: https://github.com/JuliaComputing/gdb-solib-wine https://github.com/JuliaComputing/gdb-solib-wine. Unfortunately, I don't really have the time to upstream this, but perhaps it's useful or maybe somebody sees this and wants to take on the upstreaming.
  - jchw 5y ago
    Awesome! It’s definitely a step in the right direction, and I’d love to see this upstreamed, though personally I have never contributed to GDB before. Do you have any idea how this is impacted by builds of Wine that are using the new MinGW-based PE modules for almost everything?

120 points by jchw 5y ago

This is great debugging but I wonder why the author immediately drops to debugging Wine directly, rather than running WinDbg/cdb under Wine and trying to debug the app via this way first. This feels a bit like debugging your webpage JS via attaching lldb to Chromium!

jchw 5y ago
Actually, I do have an answer for that: debuggers don’t work very well under Wine. It would be nice to get them working better. The standard tool to use for Wine is winedbg, but as I wrote, this doesn’t work here because there’s no way to break before the application entrypoint (which is too late.)
- anaisbetts 5y ago
  That's a good reason!

develatio 5y ago

But what happened with the app your friend was trying to run?? Were you able to patch it and make it work? (and if "yes", how?)

jchw 5y ago
Ahh, sorry I never completed the story. The program was simply fixed. I believe the way they wound up fixing it was simply static linking libpng and zlib. I found another workaround too, which was a very obscure linker flag, but I can’t recall what it was as it has been a few months now. If you run into this problem with a random binary and can’t get it fixed properly, you can use Microsoft’s REBASE.EXE or EDITBIN.EXE to rebase the preferred base address to be within the range of a RIP relative CALL. If you are savvy with PE, writing such a tool by hand is very straightforward; you need to write a PE parser that can map virtual addresses, relocate as if you are a real PE loader, update the preferred base address, and write back to disk. The only case where something like this won’t work is when the executable has some kind of on-disk CRC checking, or something along those lines. (So probably, VMProtect and Themida encumbered binaries can’t be modified this way. But that’s OK, as most tools of this sort only works with Microsoft’s toolchain, which does not have pseudo-relocs.)
- marklar423 5y ago
  Out of curiosity, if the tool was compiled for MinGW doesn't that mean it would run natively on Linux? Why was your friend trying to run the MinGW version?
  - jchw 5y ago
    MinGW does allow you to write Windows-specific code. The code in question could be ported, and in fact I think it might be now, but it still puzzled us why it didn’t work on Wine. I only wanted to help on this particular avenue for the sake of making Wine better. I had figured that it was most likely an issue with Wine that could be rectified. Unfortunately, emulating Windows ASLR behavior in userspace accurately may prove to be fairly difficult, so it’s probably not worth it :) I’m not convinced pseudo relocs actually work reliably on Windows in this way, although I could be wrong.

k1t 5y ago

This (2013) article[1], when talking about "High Entropy Bottom-up Randomization" says that: "For compatibility reasons, this feature is disabled by default and must be enabled on a per-application basis. This is because some 64-bit applications have latent pointer truncation issues that can surface when dealing with pointers above 4 GB (significant bits set beyond bit 31)." Although OP had an issue with an offset rather than an absolute position, I imagine it would also handle that case. [1] https://msrc-blog.microsoft.com/2013/12/11/software-defense-mitigating-common-exploitation-techniques/ https://msrc-blog.microsoft.com/2013/12/11/software-defense-...

khuey 5y ago

If you already had this in rr, you should have been able to set a hardware watchpoint on 0x36810e3eb and just reverse-continue to the point where the relocation was applied.

jchw 5y ago
This is a good point, but I feel like there was a reason that I couldn’t do this. With Wine under rr, the replays were a bit tricky to get working in the first place. I think it is possible i could’ve done this still, but I first needed to find the number of events to skip for that part of the replay. See this issue: https://github.com/rr-debugger/rr/issues/1687 https://github.com/rr-debugger/rr/issues/1687 By the time I got to the point where a reverse continue would’ve been most useful, I’d already left rr due to this issue.
pm215 5y ago
Similarly, instead of doing "a few hundred stepis" to get to the call insn that jumps to a garbage address, you can let execution run to the crash point and the reverse stepi once. The "reverse" functionality parts of rr are one of its strongest features and well worth the time investment to get acquainted with them. (I bet my own debugging workflows and habits are stuffed full of this kind of "works but is definitely not optimal" move, though, so I don't mean this as a criticism of the article.)

vgel 5y ago

I didn't know about GDB follow-fork-mode child / follow-exec-mode new! I always just fiddle around trying to launch the child process directly to debug it.

Scramblejams 5y ago

Semi-related: Is anyone aware of a Proton neophyte's guide to fixing a Windows game? I've got a few old games that get stuck various places and I imagine that there may be a simple Windows call that needs implementing, but I don't know the first thing about how to diagnose that.

iforgotpassword 5y ago
I once had a game that always crashed under wine. Instead of trying to understand the bug, I got as far as figuring out that this was a virtual function call into nowhere and then just 909090'd the crap out of it. The game then seemed to work perfectly and I couldn't even tell what was missing/different after removing a totally random function call.
coldpie 5y ago
I wrote an intro to Wine dev: https://www.codeweavers.com/blog/aeikum/2019/1/3/working-on-wine-part-1-the-wine-ecosystem https://www.codeweavers.com/blog/aeikum/2019/1/3/working-on-... It doesn't go into a lot of debugging detail, as that's its own huge topic, but may get you started. If you're debugging a thing in Proton, you'll want to use that project environment instead of Wine's directly, but the concepts are the same.
simion314 5y ago
I would start by running the game from the terminal and read all warnings and errors, compare those with games that work because some of the warnings are always there but don't affect the games.
mike_hearn 5y ago
I used to work on Wine, at first as a hobby and then later as a job (for CodeWeavers). As part of that I contributed to the Wine Developers Guide. The Wine docs to help new devs are (or were) remarkably good given the enormous complexity of the project. Unfortunately "how to fix a bug in Wine" can still devolve very rapidly into: Step 1. Learn Windows programming. Step 2. Then learn Linux programming, the Wine architecture, X86 assembly, low level debugging etc. Step 3. Draw the rest of the owl. Nonetheless it's a good place to start: https://wiki.winehq.org/Developer_Hints https://wiki.winehq.org/Developer_Hints https://wiki.winehq.org/Wine_Developer%27s_Guide https://wiki.winehq.org/Wine_Developer%27s_Guide The days when fixing games involved just straight implementation of simple library calls were mostly over, even back when I was doing this a long time ago. Broken apps or games are much more likely to be hitting obscure "bugs" in Wine (which can mean failure to replicate Windows bugs), or other even more subtle compatibility issues, like the one this blog post documents.

rwmj 5y ago

> I never quite figured out how to get debug symbols to map correctly with this Wine-under-GDB setup going on, so I had to manually explore the address space to figure out what I was looking at. Yeah I had the same problem and never solved it. In fact I don't think I even got a handle on what the problem is at all. In what format are symbols created by mingw-gcc (I think PDB??) and in what format either Wine's debugger or GCC can consume symbols (DWARF???)

rwmj 5y ago
Edit: For "GCC" I meant of course "GDB"
jchw 5y ago
That’s the weird part: MinGW does use DWARF, embedded into PE similar to how it is embedded into ELF, with just one little non-standard trick for long section names. On Windows, GDB supports loading these symbols just fine. But if I forcibly get them to load on Wine under GDB, it doesn’t seem to work even if the load seems to succeed. Confusing.
KenoFischer 5y ago
I have a patch that teaches GDB how to do this here: https://github.com/JuliaComputing/gdb-solib-wine https://github.com/JuliaComputing/gdb-solib-wine. Unfortunately, I don't really have the time to upstream this, but perhaps it's useful or maybe somebody sees this and wants to take on the upstreaming.
- jchw 5y ago
  Awesome! It’s definitely a step in the right direction, and I’d love to see this upstreamed, though personally I have never contributed to GDB before. Do you have any idea how this is impacted by builds of Wine that are using the new MinGW-based PE modules for almost everything?