4 ms·

Maybe I'm missing something, but they ported from C++ (because 'C++ is bad donchaknow') to Rust and still ran into problems parsing crash dumps? If the dump is

Maybe I'm missing something, but they ported from C++ (because 'C++ is bad donchaknow') to Rust and still ran into problems parsing crash dumps? If the dump is corrupt then just stop trying to parse/make sense of it; it's garbage.

Gankra 4y ago
No we removed many random crashes that the C++ code had. You cannot "simply" discard a crash report if something is slightly off because then you would discard most crash reports. And most debuginfo too. You can't expect "thing that runs when a process may have just experienced memory corruption" and "all builds of your application for all eternity" and "every toolchain you ever built your program with for all eternity" to be even vaguely reliable, because those things are in the past and we're trying to figure out how to fix the bugs people are experiencing in production today. It is a horribly miserable answer to tell your coworkers "yeah sorry I know users are getting thousands of crashes this morning but the crash-dumper didn't sign its name in cursive so I'm gonna refuse to let you read the letter it sent at all". And just an incoherent answer to say "yeah I know this is a stack overflow but it left the stack in a mildly corrupt state so I absolutely refuse to try to even look at the stack and figure anything out about it". Like, that is the entire purpose of a crashreporter, to investigate a program in an invalid state!
- rockdoe 4y ago
  Reminds me of "Your program shouldn't have bugs in it isn't an acceptable position to take for a debugger", from the rr folks. Unfortunately I can't find the source of the quote any more, but it stuck in my mind.
  - Gankra 4y ago
    Yeah computing backtraces in a crashreporter is extremely similar to a debugger in that you need a lot of fudge-factor heuristics and fallback modes for known toolchain bugs or common corruptions.
  - gpm 4y ago
    Speaking of the rr folk, they also had the fascinating point that you can reliably generate a "stack trace" by figuring out which `call` instructions were executed with what values (also other jump instructions I suppose), instead of walking the stack. Thereby skipping the whole "parsing the stack is insanely difficult and unreliable" issue.
    - glandium 4y ago
      FWIW, that's from pernosco, not rr.
      - gpm 4y ago
        I think it's the same people?
        db48x 4y ago
        It is. rr records the trace and the Pernosco converts it into a database that can be queried.
  - khuey 4y ago
    You're probably remembering https://pernos.co/blog/tzcnt-portability/ https://pernos.co/blog/tzcnt-portability/
structural 4y ago
This is the excessively fun part of dealing with crash dumps in general. Many of them are going to be 1% corrupt, 99% fine, and somewhere in them likely has vital information about what caused the corruption. So the entire reason for being for things like rust-minidump are to make enough sense out of files that are known to be corrupt garbage to be able to find bugs.
- tedmielczarek 4y ago
  A lot of the weird bits in the Breakpad codebase were definitely from us finding extremely broken minidumps from Firefox users in the wild and then me tweaking the code to see if we could get something out of it so we had a chance at diagnosing the issue.
- bpye 4y ago
  The worst bugs are often the ones that have trashed some of the stack or ended up with some rubbish register state too, and so being able to try and get some useful information out of a seemingless garbage dump is critical. I imagine a project like Firefox is also big enough to see bad things happening because of incorrect CPU behaviour and bit flips and such too...

Gankra 4y ago

No we removed many random crashes that the C++ code had. You cannot "simply" discard a crash report if something is slightly off because then you would discard most crash reports. And most debuginfo too. You can't expect "thing that runs when a process may have just experienced memory corruption" and "all builds of your application for all eternity" and "every toolchain you ever built your program with for all eternity" to be even vaguely reliable, because those things are in the past and we're trying to figure out how to fix the bugs people are experiencing in production today. It is a horribly miserable answer to tell your coworkers "yeah sorry I know users are getting thousands of crashes this morning but the crash-dumper didn't sign its name in cursive so I'm gonna refuse to let you read the letter it sent at all". And just an incoherent answer to say "yeah I know this is a stack overflow but it left the stack in a mildly corrupt state so I absolutely refuse to try to even look at the stack and figure anything out about it". Like, that is the entire purpose of a crashreporter, to investigate a program in an invalid state!

rockdoe 4y ago
Reminds me of "Your program shouldn't have bugs in it isn't an acceptable position to take for a debugger", from the rr folks. Unfortunately I can't find the source of the quote any more, but it stuck in my mind.
- Gankra 4y ago
  Yeah computing backtraces in a crashreporter is extremely similar to a debugger in that you need a lot of fudge-factor heuristics and fallback modes for known toolchain bugs or common corruptions.
- gpm 4y ago
  Speaking of the rr folk, they also had the fascinating point that you can reliably generate a "stack trace" by figuring out which `call` instructions were executed with what values (also other jump instructions I suppose), instead of walking the stack. Thereby skipping the whole "parsing the stack is insanely difficult and unreliable" issue.
  - glandium 4y ago
    FWIW, that's from pernosco, not rr.
    - gpm 4y ago
      I think it's the same people?
      - db48x 4y ago
        It is. rr records the trace and the Pernosco converts it into a database that can be queried.
- khuey 4y ago
  You're probably remembering https://pernos.co/blog/tzcnt-portability/ https://pernos.co/blog/tzcnt-portability/

structural 4y ago

This is the excessively fun part of dealing with crash dumps in general. Many of them are going to be 1% corrupt, 99% fine, and somewhere in them likely has vital information about what caused the corruption. So the entire reason for being for things like rust-minidump are to make enough sense out of files that are known to be corrupt garbage to be able to find bugs.

tedmielczarek 4y ago
A lot of the weird bits in the Breakpad codebase were definitely from us finding extremely broken minidumps from Firefox users in the wild and then me tweaking the code to see if we could get something out of it so we had a chance at diagnosing the issue.
bpye 4y ago
The worst bugs are often the ones that have trashed some of the stack or ended up with some rubbish register state too, and so being able to try and get some useful information out of a seemingless garbage dump is critical. I imagine a project like Firefox is also big enough to see bad things happening because of incorrect CPU behaviour and bit flips and such too...