6 ms·

Coincidentally, I also stumbled upon a way to make the kernel of Apple Silicon Macs panic and restart while developing the https://lowtechguys.com https://lowte

by alin23 4y ago

Coincidentally, I also stumbled upon a way to make the kernel of Apple Silicon Macs panic and restart while developing the https://lowtechguys.com https://lowtechguys.com website. I distilled the problem in a repo so it can be reproduced with a single command: https://github.com/alin23/m1-panic https://github.com/alin23/m1-panic I found it while on Monterey and reported it 2 times through Feedback Assistant, but it still happens on Ventura. NOTE: Don't try it without saving all your work, it has a very high chance of restarting your computer forcefully.

metadat 4y ago
Do you know what the underlying problematic instruction sequence is? Or the precise location where it halts?
- alin23 4y ago
  I have no idea how I could find that, given that the system freezes completely. Maybe tracing the CPU instructions using LLDB might be possible, but the bug is most likely in the kernel code so this would not help much.
  - catiopatio 4y ago
    You can debug the kernel remotely over Ethernet: https://developer.apple.com/documentation/apple-silicon/debugging-a-custom-kernel-extension https://developer.apple.com/documentation/apple-silicon/debu... If that still fails, virtualization tools provide debugging interfaces you can use to step the execution of the virtualized CPU; e.g. VMware’s “debugStub” feature.
    - Firmwarrior 4y ago
      haha, let's just let the Apple engineers on this thread figure that shit out
    - sharikous 4y ago
      You can't with Apple Silicon. It's shameful in my opinion. You still can load a core dump or view the state after a NMI but you can't run the kernel under a debugger.
      - saagarjha 4y ago
        You can, the experience is just not very good.
  - bri3d 4y ago
    The best public kernel debugger for Apple Silicon is m1n1 from the Asahi project.
- saagarjha 4y ago
  ldr x9, [x8, #0x18]!
macshome 4y ago
Can you put the panic log text into the repo as well?
- alin23 4y ago
  Sure, added here: https://github.com/alin23/m1-panic#panic-crash-report-after-system-restart https://github.com/alin23/m1-panic#panic-crash-report-after-...
catiopatio 4y ago
What’s the feedback ID #?
trollied 4y ago
Might have been better to report it to the security people. This sort of thing can be exploitable.
- nemetroid 4y ago
  They did report it to Apple, multiple times: > I found it while on Monterey and reported it 2 times through Feedback Assistant, but it still happens on Ventura.
  - deleted 4y ago
    [deleted]
  - lilyball 4y ago
    "to the security people" means emailing the relevant email address, not Feedback Assistant.
    - nemetroid 4y ago
      If it doesn't end up with the relevant people, that's Apple's problem.
      - saagarjha 4y ago
        It does eventually. If you want a prompt response you should contact product security.
        hackmiester 4y ago
        Shouldn't Apple be the ones who really want to respond promptly? Why should we work around bugs in Apple's issue reporting system?
        saagarjha 4y ago
        I don’t really see the problem with getting faster responses by contacting Apple’s security team directly for potential vulnerabilities when compared to the general-purpose bug tracker.
MuffinFlavored 4y ago
does it work in zsh or bash? only fish?
saagarjha 4y ago
The bug is in one of these two lines: https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/kern/tty_ptmx.c#L547 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0... https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/kern/tty_ptmx.c#L550 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0... This uses the macro SLIST_REMOVE (https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/kern/kern_event.c#L6047 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0...) to remove an item from the linked list. If you look at the code to do this it's a pretty simple linked list traversal: https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/sys/queue.h#L310 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0.... However, it doesn't have a check for the end of the list, so the item must be in the list, or it's going to walk right off the end and dereference a null pointer. In this case that's exactly what it ended up doing.
- alin23 4y ago
  That's.. wow, ok. How exactly did you end up in that source code at that specific line? I know you're well versed in reverse engineering Darwin, and I'm reading your posts and trying to improve my skills in this daily, but this seems way over my skillset. Did you debug this using KDK or m1n1? Do you have a setup always ready for debugging a Darwin kernel?
  - saagarjha 4y ago
    In theory I have all of those, but currently I have none, so it's manual work. Your best friend in diagnosing a kernel crash is a KDK. If you have one that matches your build, it will have symbols in it. With a little bit of math you can take the backtrace in the crash log and slide it appropriately to match the binary. Personally I use LLDB for this. Here's an example of what this looks like on an x86-64 kernel (Apple silicon has its own math but it's largely the same): https://github.com/saagarjha/unxip/issues/14#issuecomment-1099099784 https://github.com/saagarjha/unxip/issues/14#issuecomment-10.... The kernel is typically compiled with optimization, so there's a lot of inlining and code folding, but with function names, source files, and instruction offsets it's pretty trivial to match it to the code Apple publishes. In this case I do not have a KDK for that build. In fact Apple has been unable to produce one for a couple of months, a inadequacy which I have repeatedly emphasized to them because of how critical they are for stuff like this. Supposedly they are working on it. Whatever; in lieu of that I got to figure out how good the tooling for analyzing kernels is these days, which was my real goal anyways. For this crash log I downloaded the IPSW file for your build, 22A400. All of them get linked on The iPhone Wiki, e.g. https://www.theiphonewiki.com/wiki/Firmware/Mac/13.x https://www.theiphonewiki.com/wiki/Firmware/Mac/13.x. Once you unpack the IPSW (it's a zip file) there are compressed kernelcache files inside. Apple changed the format of these this year so most of the tooling breaks on it, but https://github.com/blacktop/ipsw https://github.com/blacktop/ipsw was able to decompress them. Then I loaded it in to Binary Ninja, which apparently doesn't support them either but compiling this person's plugin (+166 submodules, and a LLVM & Boost build) gets it to work: https://github.com/skr0x1c0/binja_kc https://github.com/skr0x1c0/binja_kc. From there you can load up the faulting address from the crash log and see what the function looks like. In this case, a bunch of junk has been inlined into it but there's a really obvious and fairly unique string reference for "invalid knote %p detach, filter: %d". From there, you can compare it against the actual source code to see which one matches the "shape" of the function you're looking at. I happened to also pull up an older kernel which did have a KDK available and then compared its assembly to the new one to match it up to ptsd_kqops_detach. The disassembly of the crashing code is obviously doing a linked list walk so you can figure out exactly which line it is from that. If I wasn't lazy I might also fire up a debugger to see why the function had walked off the end of the list but without KDKs things get pretty bad, not that they're very good to begin with. I don't have a m1n1 setup (I should probably do this at some point) and the things I do have, like remote debugging or the VM GDB stub, are not really worth suffering through for a Hacker News comment.

by alin23 4y ago

metadat 4y ago

Do you know what the underlying problematic instruction sequence is? Or the precise location where it halts?

alin23 4y ago
I have no idea how I could find that, given that the system freezes completely. Maybe tracing the CPU instructions using LLDB might be possible, but the bug is most likely in the kernel code so this would not help much.
- catiopatio 4y ago
  You can debug the kernel remotely over Ethernet: https://developer.apple.com/documentation/apple-silicon/debugging-a-custom-kernel-extension https://developer.apple.com/documentation/apple-silicon/debu... If that still fails, virtualization tools provide debugging interfaces you can use to step the execution of the virtualized CPU; e.g. VMware’s “debugStub” feature.
  - Firmwarrior 4y ago
    haha, let's just let the Apple engineers on this thread figure that shit out
  - sharikous 4y ago
    You can't with Apple Silicon. It's shameful in my opinion. You still can load a core dump or view the state after a NMI but you can't run the kernel under a debugger.
    - saagarjha 4y ago
      You can, the experience is just not very good.
- bri3d 4y ago
  The best public kernel debugger for Apple Silicon is m1n1 from the Asahi project.
saagarjha 4y ago
ldr x9, [x8, #0x18]!

macshome 4y ago

Can you put the panic log text into the repo as well?

alin23 4y ago
Sure, added here: https://github.com/alin23/m1-panic#panic-crash-report-after-system-restart https://github.com/alin23/m1-panic#panic-crash-report-after-...

catiopatio 4y ago

What’s the feedback ID #?

trollied 4y ago

Might have been better to report it to the security people. This sort of thing can be exploitable.

nemetroid 4y ago
They did report it to Apple, multiple times: > I found it while on Monterey and reported it 2 times through Feedback Assistant, but it still happens on Ventura.
- deleted 4y ago
  [deleted]
- lilyball 4y ago
  "to the security people" means emailing the relevant email address, not Feedback Assistant.
  - nemetroid 4y ago
    If it doesn't end up with the relevant people, that's Apple's problem.
    - saagarjha 4y ago
      It does eventually. If you want a prompt response you should contact product security.
      - hackmiester 4y ago
        Shouldn't Apple be the ones who really want to respond promptly? Why should we work around bugs in Apple's issue reporting system?
        saagarjha 4y ago
        I don’t really see the problem with getting faster responses by contacting Apple’s security team directly for potential vulnerabilities when compared to the general-purpose bug tracker.

MuffinFlavored 4y ago

does it work in zsh or bash? only fish?

saagarjha 4y ago

The bug is in one of these two lines: https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/kern/tty_ptmx.c#L547 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0... https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/kern/tty_ptmx.c#L550 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0... This uses the macro SLIST_REMOVE (https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/kern/kern_event.c#L6047 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0...) to remove an item from the linked list. If you look at the code to do this it's a pretty simple linked list traversal: https://github.com/apple-oss-distributions/xnu/blob/5c2921b07a2480ab43ec66f5b9e41cb872bc554f/bsd/sys/queue.h#L310 https://github.com/apple-oss-distributions/xnu/blob/5c2921b0.... However, it doesn't have a check for the end of the list, so the item must be in the list, or it's going to walk right off the end and dereference a null pointer. In this case that's exactly what it ended up doing.

alin23 4y ago
That's.. wow, ok. How exactly did you end up in that source code at that specific line? I know you're well versed in reverse engineering Darwin, and I'm reading your posts and trying to improve my skills in this daily, but this seems way over my skillset. Did you debug this using KDK or m1n1? Do you have a setup always ready for debugging a Darwin kernel?
- saagarjha 4y ago
  In theory I have all of those, but currently I have none, so it's manual work. Your best friend in diagnosing a kernel crash is a KDK. If you have one that matches your build, it will have symbols in it. With a little bit of math you can take the backtrace in the crash log and slide it appropriately to match the binary. Personally I use LLDB for this. Here's an example of what this looks like on an x86-64 kernel (Apple silicon has its own math but it's largely the same): https://github.com/saagarjha/unxip/issues/14#issuecomment-1099099784 https://github.com/saagarjha/unxip/issues/14#issuecomment-10.... The kernel is typically compiled with optimization, so there's a lot of inlining and code folding, but with function names, source files, and instruction offsets it's pretty trivial to match it to the code Apple publishes. In this case I do not have a KDK for that build. In fact Apple has been unable to produce one for a couple of months, a inadequacy which I have repeatedly emphasized to them because of how critical they are for stuff like this. Supposedly they are working on it. Whatever; in lieu of that I got to figure out how good the tooling for analyzing kernels is these days, which was my real goal anyways. For this crash log I downloaded the IPSW file for your build, 22A400. All of them get linked on The iPhone Wiki, e.g. https://www.theiphonewiki.com/wiki/Firmware/Mac/13.x https://www.theiphonewiki.com/wiki/Firmware/Mac/13.x. Once you unpack the IPSW (it's a zip file) there are compressed kernelcache files inside. Apple changed the format of these this year so most of the tooling breaks on it, but https://github.com/blacktop/ipsw https://github.com/blacktop/ipsw was able to decompress them. Then I loaded it in to Binary Ninja, which apparently doesn't support them either but compiling this person's plugin (+166 submodules, and a LLVM & Boost build) gets it to work: https://github.com/skr0x1c0/binja_kc https://github.com/skr0x1c0/binja_kc. From there you can load up the faulting address from the crash log and see what the function looks like. In this case, a bunch of junk has been inlined into it but there's a really obvious and fairly unique string reference for "invalid knote %p detach, filter: %d". From there, you can compare it against the actual source code to see which one matches the "shape" of the function you're looking at. I happened to also pull up an older kernel which did have a KDK available and then compared its assembly to the new one to match it up to ptsd_kqops_detach. The disassembly of the crashing code is obviously doing a linked list walk so you can figure out exactly which line it is from that. If I wasn't lazy I might also fire up a debugger to see why the function had walked off the end of the list but without KDKs things get pretty bad, not that they're very good to begin with. I don't have a m1n1 setup (I should probably do this at some point) and the things I do have, like remote debugging or the VM GDB stub, are not really worth suffering through for a Hacker News comment.