4 ms·

Debugging a Futex Crash

95 points by rustylife 3y ago

ByteMe95 3y ago
Maybe it’s just me but if I saw a stack trace like that I would think that I had some memory corruption somewhere ( I would think even in this case it’s more plausible than the mutex being allocated misaligned )
- planede 3y ago
  Maybe a mutex accidentally put in packed struct?
- r-r-r 3y ago
  Yeah, same here. I wonder if this'd have been caught by ubsan, though.
RustyRussell 3y ago
Ok, I did a double take. It's a different Rusty playing with futexes!!
- rustylife 3y ago
  Thank you for writing kernel futex code. And your kernel articles are the best, thank you very much for writing those! I am sending love and positive vibes your way!
no_time 3y ago
Do these crash reporting tools have mechanisms in place to not leak sensitive data in the coredumps? Would be awkward if a firefox coredump upload would leak my session cookies.
- ishitatsuyuki 3y ago
  Unfortunately, removal of PIIs also means less data/cues for hard-to-diagnose bugs (it can even range to CPU bugs like [1]). For Firefox, potentially PII-containing memory dumps are still sent to the server, but they're only viewable by authorized personnel [2]. There are also rule-based PII removal methods as used by Sentry minidump processing and described at [3]. [1]: https://news.ycombinator.com/item?id=37063459 https://news.ycombinator.com/item?id=37063459 [2]: https://crash-stats.mozilla.org/documentation/protected_data_access/ https://crash-stats.mozilla.org/documentation/protected_data... [3]: https://docs.sentry.io/product/data-management-settings/scrubbing/attachment-scrubbing/ https://docs.sentry.io/product/data-management-settings/scru...
- hyperman1 3y ago
  That's almost impossible, and one of the reasons automatic crash reporting is problematic. It is an issue with e.g medical data: Doctors using mswindows or any browser based tool can at any moment export random medical data to god knows which country. In the EU, it is probably illegal to even look at it if you receive it, and you don't know which dreports even have this issue. Afaik, the only reason things aren't crashing down is total non-enforcement by all authorities. Which raises interesting question about their will and capability to enforce e.g. the medical secret at all.
  - TazeTSchnitzel 3y ago
    Automatic crash reporting doesn't necessarily contain sensitive info, it depends on how much detail is provided. When I get a crash on macOS, the reporter lets me the report before it's sent, and as far as I can tell it's just plaintext stacktraces of active threads, plus a handful of other details.
    - rocqua 3y ago
      As I understand a 'core-dump' is not just a stack trace, but also contains some of the arguments passed in the functions in the stack-trace. Those can easily contain sensitive information. Especially if the heap is also included.
      - TazeTSchnitzel 3y ago
        Yes, a core dump is sensitive. It concerns me if those are ever sent over the wire automatically.
    - js2 3y ago
      macOS (and iOS) do the stack unwinding on the device. The "backend" then only has to perform symbolication (mapping addresses to symbol names). This is how both the built-in macOS/iOS crash reporter work, as well as the third-party PLCrashReporter and KSCrash crash handlers. On Linux and Android, you can similarly do unwinding using something like libunwind on the device. However, the traditional solution is that a "core" file is dumped and then is unwound later. Years ago Microsoft invented a format called the "minidump" which is basically the stack for each thread, CPU registers, and other data. This format was adopted by Google Breakpad, and which ended up being used by both Firefox and Chrome for their crash reporting. The minidump format is often used when for whatever reason, unwinding can't be done on the device. It's a fairly flexible binary format. Breakpad can perform sanitation before sending the minidump off the device, but this can be hit or miss. Sentry (and probably other services) support minidumps among other formats, but minidumps are a PITA and I'll bet they hate supporting them. It's a large file despite its name, and it's expensive, complicated, and risky to unwind on the backend. The only open-source minidump unwinder used to be fairly esoteric C++ code that was part of Breakpad, but there's now a rust-based unwinder too. Breakpad came up with something called a microdump, which is an even smaller version of a minidump that's text based instead of binary. Both Apple and Google make mobile crash reporting way harder than it need be. Android and iOS have their own crash handlers that can perform on-device out-of-process unwinding, but the reports are not available to the app itself. You only get whatever reports Apple and Google feel like showing you after they get uploaded to their systems. So apps that want more detail have to embed their own crash handlers. Because those handlers are operating in the same memory space as the app which just crashed, their operation is somewhat fraught with peril. At least on iOS, unwinding is straightforward due to mandatory frame pointes, and symbolication is easy because Apple makes iOS symbols available. Android is a cluster fuck when it comes to native code crashes. You may or may not have frame pointers. So unwinding on the device is hit or miss. You also have to deal with both native code and ART (Android Run Time) code. Tracing back through the runtime is non-trivial to impossible. And you will almost definitely not have debugging symbols. So good luck with symbolication. The lack of debugging symbols also make off-device unwinding basically useless, so a minidump often isn't that helpful. I designed and built the mobile crash reporting solution for my company about a decade ago, though we're currently migrating to a SAAS solution. On Android I evolved the solution from minidumps to microdumps to microdumps + on-device unwinding. It also turns out that there's a lot of helpful information sent to logcat and including that with native crash reports can often be more helpful than whatever you can capture from the stack during the crash.
- planede 3y ago
  Found this for coredumps and minidumps: https://support.backtrace.io/hc/en-us/articles/360040517131-Privacy-Control https://support.backtrace.io/hc/en-us/articles/360040517131-... So there are at least some efforts to do it. IMO it can only ever be best effort, but it doesn't mean that it's not worth doing.
WirelessGigabit 3y ago
>>> frame 5 #5 0x00007f7430627733 in pthread_mutex_lock () from /home/rusty/futex/sysroot/lib/x86_64-linux-gnu/libpthread.so.0 >>> disassemble pthread_mutex_lock ... 0x00007f7430627723 <+211>: mov %rdi,0x8(%rsp) 0x00007f7430627728 <+216>: and $0x80,%esi 0x00007f743062772e <+222>: call 0x7f743062efc0 <__lll_lock_wait> => 0x00007f7430627733 <+227>: mov 0x8(%rsp),%rdi 0x00007f7430627738 <+232>: jmp 0x7f7430627689 <pthread_mutex_lock+57> ... >>> x/w ($rsp + 8) 0x7f7371ff1ce8: 0x08001ea1 I love reading these articles, but sometimes they skip over things that the reader might not understand. In my case I don't understand where x and w come from, and how the result of 0x08001ea1 is invalid (I can see it's not aligned to a 4 byte boundary).
- enoent 3y ago
  x/w is a gdb command to dereference and print (x) 1 word-sized data (w) at the address given by the expression. So 0x08001ea1 was the unaligned address stored at $rsp + 8, passed as argument to __lll_lock_wait.

95 points by rustylife 3y ago

Maybe it’s just me but if I saw a stack trace like that I would think that I had some memory corruption somewhere ( I would think even in this case it’s more plausible than the mutex being allocated misaligned )

planede 3y ago
Maybe a mutex accidentally put in packed struct?
r-r-r 3y ago
Yeah, same here. I wonder if this'd have been caught by ubsan, though.

RustyRussell 3y ago

Ok, I did a double take. It's a different Rusty playing with futexes!!

rustylife 3y ago
Thank you for writing kernel futex code. And your kernel articles are the best, thank you very much for writing those! I am sending love and positive vibes your way!

no_time 3y ago

Do these crash reporting tools have mechanisms in place to not leak sensitive data in the coredumps? Would be awkward if a firefox coredump upload would leak my session cookies.

ishitatsuyuki 3y ago
Unfortunately, removal of PIIs also means less data/cues for hard-to-diagnose bugs (it can even range to CPU bugs like [1]). For Firefox, potentially PII-containing memory dumps are still sent to the server, but they're only viewable by authorized personnel [2]. There are also rule-based PII removal methods as used by Sentry minidump processing and described at [3]. [1]: https://news.ycombinator.com/item?id=37063459 https://news.ycombinator.com/item?id=37063459 [2]: https://crash-stats.mozilla.org/documentation/protected_data_access/ https://crash-stats.mozilla.org/documentation/protected_data... [3]: https://docs.sentry.io/product/data-management-settings/scrubbing/attachment-scrubbing/ https://docs.sentry.io/product/data-management-settings/scru...
hyperman1 3y ago
That's almost impossible, and one of the reasons automatic crash reporting is problematic. It is an issue with e.g medical data: Doctors using mswindows or any browser based tool can at any moment export random medical data to god knows which country. In the EU, it is probably illegal to even look at it if you receive it, and you don't know which dreports even have this issue. Afaik, the only reason things aren't crashing down is total non-enforcement by all authorities. Which raises interesting question about their will and capability to enforce e.g. the medical secret at all.
- TazeTSchnitzel 3y ago
  Automatic crash reporting doesn't necessarily contain sensitive info, it depends on how much detail is provided. When I get a crash on macOS, the reporter lets me the report before it's sent, and as far as I can tell it's just plaintext stacktraces of active threads, plus a handful of other details.
  - rocqua 3y ago
    As I understand a 'core-dump' is not just a stack trace, but also contains some of the arguments passed in the functions in the stack-trace. Those can easily contain sensitive information. Especially if the heap is also included.
    - TazeTSchnitzel 3y ago
      Yes, a core dump is sensitive. It concerns me if those are ever sent over the wire automatically.
  - js2 3y ago
    macOS (and iOS) do the stack unwinding on the device. The "backend" then only has to perform symbolication (mapping addresses to symbol names). This is how both the built-in macOS/iOS crash reporter work, as well as the third-party PLCrashReporter and KSCrash crash handlers. On Linux and Android, you can similarly do unwinding using something like libunwind on the device. However, the traditional solution is that a "core" file is dumped and then is unwound later. Years ago Microsoft invented a format called the "minidump" which is basically the stack for each thread, CPU registers, and other data. This format was adopted by Google Breakpad, and which ended up being used by both Firefox and Chrome for their crash reporting. The minidump format is often used when for whatever reason, unwinding can't be done on the device. It's a fairly flexible binary format. Breakpad can perform sanitation before sending the minidump off the device, but this can be hit or miss. Sentry (and probably other services) support minidumps among other formats, but minidumps are a PITA and I'll bet they hate supporting them. It's a large file despite its name, and it's expensive, complicated, and risky to unwind on the backend. The only open-source minidump unwinder used to be fairly esoteric C++ code that was part of Breakpad, but there's now a rust-based unwinder too. Breakpad came up with something called a microdump, which is an even smaller version of a minidump that's text based instead of binary. Both Apple and Google make mobile crash reporting way harder than it need be. Android and iOS have their own crash handlers that can perform on-device out-of-process unwinding, but the reports are not available to the app itself. You only get whatever reports Apple and Google feel like showing you after they get uploaded to their systems. So apps that want more detail have to embed their own crash handlers. Because those handlers are operating in the same memory space as the app which just crashed, their operation is somewhat fraught with peril. At least on iOS, unwinding is straightforward due to mandatory frame pointes, and symbolication is easy because Apple makes iOS symbols available. Android is a cluster fuck when it comes to native code crashes. You may or may not have frame pointers. So unwinding on the device is hit or miss. You also have to deal with both native code and ART (Android Run Time) code. Tracing back through the runtime is non-trivial to impossible. And you will almost definitely not have debugging symbols. So good luck with symbolication. The lack of debugging symbols also make off-device unwinding basically useless, so a minidump often isn't that helpful. I designed and built the mobile crash reporting solution for my company about a decade ago, though we're currently migrating to a SAAS solution. On Android I evolved the solution from minidumps to microdumps to microdumps + on-device unwinding. It also turns out that there's a lot of helpful information sent to logcat and including that with native crash reports can often be more helpful than whatever you can capture from the stack during the crash.
planede 3y ago
Found this for coredumps and minidumps: https://support.backtrace.io/hc/en-us/articles/360040517131-Privacy-Control https://support.backtrace.io/hc/en-us/articles/360040517131-... So there are at least some efforts to do it. IMO it can only ever be best effort, but it doesn't mean that it's not worth doing.

WirelessGigabit 3y ago

>>> frame 5 #5 0x00007f7430627733 in pthread_mutex_lock () from /home/rusty/futex/sysroot/lib/x86_64-linux-gnu/libpthread.so.0 >>> disassemble pthread_mutex_lock ... 0x00007f7430627723 <+211>: mov %rdi,0x8(%rsp) 0x00007f7430627728 <+216>: and $0x80,%esi 0x00007f743062772e <+222>: call 0x7f743062efc0 <__lll_lock_wait> => 0x00007f7430627733 <+227>: mov 0x8(%rsp),%rdi 0x00007f7430627738 <+232>: jmp 0x7f7430627689 <pthread_mutex_lock+57> ... >>> x/w ($rsp + 8) 0x7f7371ff1ce8: 0x08001ea1 I love reading these articles, but sometimes they skip over things that the reader might not understand. In my case I don't understand where x and w come from, and how the result of 0x08001ea1 is invalid (I can see it's not aligned to a 4 byte boundary).

enoent 3y ago
x/w is a gdb command to dereference and print (x) 1 word-sized data (w) at the address given by the expression. So 0x08001ea1 was the unaligned address stored at $rsp + 8, passed as argument to __lll_lock_wait.