3 ms·

That method is pretty much guarnateed to give you completely broken stacks. We (Sentry) use it as a fallback if nothing else works, and the success rate is awfu

That method is pretty much guarnateed to give you completely broken stacks. We (Sentry) use it as a fallback if nothing else works, and the success rate is awful.

loeg 4y ago
Please elaborate on what breaks.
- xuhu 4y ago
  Probably most valid code addresses on the stack are from older calls that are no longer part of the current call chain.
  - userbinator 4y ago
    There's a chance of that, but in practice it's easy to filter them out, since only one of the chains will be complete from the very top of the stack to the current stack pointer. The others will almost certainly be partially overwritten by other stack activity and incomplete, or not match the current stack pointer.
- the_mitsuhiko 4y ago
  The act of stackwalking breaks. You often end up in completely nonsensical stacks halfway through and then all is lost.
  - loeg 4y ago
    Is this just an artifact of old return addresses on the stack not being overwritten?
  - jsolson 4y ago
    That makes sense, as you're fucked by any false positive. If it's a graph of potential stacks, though, wouldn't you eventually find the one that unwinds, if one exists?
    - loeg 4y ago
      GP mentioned in another comment that they don't actually follow the algorithm described by userbinator -- checking the return address for a preceding call -- which would increase the number of false positives a lot.
      - haileys 4y ago
        The thing that makes this tricky is that x86 is a variable length instruction set: you don't know where the preceding call instruction might begin, and you can't decode backwards. You can do it speculatively, but ambiguities are still possible.
        ghusbands 4y ago
        Given that the full technique apparently works well for userbinator, we can assume that this isn't a very significant issue. The chances of a phantom call instruction will be pretty low, and there's a fairly low maximum instruction length.
        loeg 4y ago
        Yes, but checking for valid CALL instructions is probably sufficiently accurate.
userbinator 4y ago
Are you also doing back-disassembly to link the call chain? I've had very good success with it. I agree that without analysing the potential call sites and making sure that they are actually possible, it won't work.
- the_mitsuhiko 4y ago
  Generally what we do when we fall back to scanning depends on the architecture. For x86 you can find the logic here: https://github.com/rust-minidump/rust-minidump/blob/77638ab70b2e2c0ded1317ca175b0fb8a75fd477/minidump-processor/src/stackwalker/x86.rs#L199 https://github.com/rust-minidump/rust-minidump/blob/77638ab7... Since we do post-hoc stack walking our ability to actually look at the assembly is limited. In most cases where we have no CFI we also do not have the binary to begin with, so we're in random memory land.
  - userbinator 4y ago
    If we find such an instruction, we assume it's an ip value that was pushed by the CALL instruction that created the current frame. I'm not familiar with Rust, but if I'm reading it correctly, you're not doing back-disassembly and only checking if an address on the stack is in a code region, which certainly won't work well. we also do not have the binary to begin with Your "minidumps" only contain the stack?
    - js2 4y ago
      Minidumps only contain the stack, yes, along with additional metadata, but not the original executable. This is an old format that I think Microsoft originally designed and that Google extended for the breakpad project. It's used by Chrome and Mozilla also uses it for Firefox. It's essentially a crash reporting binary format. I linked you to it elsewhere in this thread. Sentry is using a rust implementation of the stack walker.

That method is pretty much guarnateed to give you completely broken stacks. We (Sentry) use it as a fallback if nothing else works, and the success rate is awful.

loeg 4y ago

Please elaborate on what breaks.

xuhu 4y ago
Probably most valid code addresses on the stack are from older calls that are no longer part of the current call chain.
- userbinator 4y ago
  There's a chance of that, but in practice it's easy to filter them out, since only one of the chains will be complete from the very top of the stack to the current stack pointer. The others will almost certainly be partially overwritten by other stack activity and incomplete, or not match the current stack pointer.
the_mitsuhiko 4y ago
The act of stackwalking breaks. You often end up in completely nonsensical stacks halfway through and then all is lost.
- loeg 4y ago
  Is this just an artifact of old return addresses on the stack not being overwritten?
- jsolson 4y ago
  That makes sense, as you're fucked by any false positive. If it's a graph of potential stacks, though, wouldn't you eventually find the one that unwinds, if one exists?
  - loeg 4y ago
    GP mentioned in another comment that they don't actually follow the algorithm described by userbinator -- checking the return address for a preceding call -- which would increase the number of false positives a lot.
    - haileys 4y ago
      The thing that makes this tricky is that x86 is a variable length instruction set: you don't know where the preceding call instruction might begin, and you can't decode backwards. You can do it speculatively, but ambiguities are still possible.
      - ghusbands 4y ago
        Given that the full technique apparently works well for userbinator, we can assume that this isn't a very significant issue. The chances of a phantom call instruction will be pretty low, and there's a fairly low maximum instruction length.
      - loeg 4y ago
        Yes, but checking for valid CALL instructions is probably sufficiently accurate.

userbinator 4y ago

Are you also doing back-disassembly to link the call chain? I've had very good success with it. I agree that without analysing the potential call sites and making sure that they are actually possible, it won't work.

the_mitsuhiko 4y ago
Generally what we do when we fall back to scanning depends on the architecture. For x86 you can find the logic here: https://github.com/rust-minidump/rust-minidump/blob/77638ab70b2e2c0ded1317ca175b0fb8a75fd477/minidump-processor/src/stackwalker/x86.rs#L199 https://github.com/rust-minidump/rust-minidump/blob/77638ab7... Since we do post-hoc stack walking our ability to actually look at the assembly is limited. In most cases where we have no CFI we also do not have the binary to begin with, so we're in random memory land.
- userbinator 4y ago
  If we find such an instruction, we assume it's an ip value that was pushed by the CALL instruction that created the current frame. I'm not familiar with Rust, but if I'm reading it correctly, you're not doing back-disassembly and only checking if an address on the stack is in a code region, which certainly won't work well. we also do not have the binary to begin with Your "minidumps" only contain the stack?
  - js2 4y ago
    Minidumps only contain the stack, yes, along with additional metadata, but not the original executable. This is an old format that I think Microsoft originally designed and that Google extended for the breakpad project. It's used by Chrome and Mozilla also uses it for Firefox. It's essentially a crash reporting binary format. I linked you to it elsewhere in this thread. Sentry is using a rust implementation of the stack walker.