4 ms·

(Not the author) I’ve learned to not get my hopes up about the capabilities of external format-parsing libraries when building tools like this (to the point of

by derefr 5y ago

(Not the author) I’ve learned to not get my hopes up about the capabilities of external format-parsing libraries when building tools like this (to the point of often not bothering to evaluate them for fit-for-purpose any more), as they often expose only a high-level fully-decoded representation that’s unsuited to examination of “why” something decoded to what it did. As such, they often won’t be of any help in a situation where you’re trying to use them to diagnose where exactly a corrupted piece of data is going wrong — which is one of the biggest use-cases for such tooling!

jcranmer 5y ago
I've been using object and gimli crates extensively, and I can assure that "expose only a high-level fully-decoded representation" is the exact opposite of what they do. In fact, my biggest criticism (for gimli in particular) is that they lack sufficient high-level representation. The object crate exposes all of the ELF types directly. The one thing it doesn't do is give you this from its format-agnostic object::read::File type, you have to start from object::read::elf::ElfFile instead. As a bonus, it also gives you all of the processor-specific defines so you don't have to look up what the value of, say, the x86-64 relocations are: https://docs.rs/object/0.25.3/object/elf/index.html#constants https://docs.rs/object/0.25.3/object/elf/index.html#constant...
- derefr 5y ago
  My point wasn't about whether you can get into to the nitty-gritty leaf nodes of complex structures in the fully-decoded data; it was about whether the representation it outputs losslessly represents unparseable data elements while still doing best-effort to decode what it can, such that you end up with a representation that was decoded "as much as possible" where the decoded parts can be used to figure out why the non-decoded parts didn't decode, while also not obscuring what the non-decoded parts "say". I'm using "high-level" here to mean "was successfully transformed through all the decoding/lexing/parsing/cross-reference stages", and "low[er]-level" to mean "failed to be transformed by some of those stages." Which is non-normative, I guess, but this sort of "layers of decoded-ness" representation is what you expect from tools like Wireshark or binwalk.
  - jcranmer 5y ago
    Ah, I see what you mean now. Yes, object/gimli are also this kind of low-level. Basically, when you parse the file, you're not actually parsing the file, but you're parsing each element of the structure one bit of a time. So parsing FileHeader will make sure a) the data is correctly aligned [since it's UB in Rust to have underaligned data] and b) that the e_ident bits are actually the magic number for an ELF file. Want to list all the sections? That's when it's actually going to check that a) the section header offset actually points to valid data and b) the number of section headers exist and is sane, but again, it doesn't actually verify that the section headers themselves make any sense whatsoever.

by derefr 5y ago

jcranmer 5y ago

I've been using object and gimli crates extensively, and I can assure that "expose only a high-level fully-decoded representation" is the exact opposite of what they do. In fact, my biggest criticism (for gimli in particular) is that they lack sufficient high-level representation. The object crate exposes all of the ELF types directly. The one thing it doesn't do is give you this from its format-agnostic object::read::File type, you have to start from object::read::elf::ElfFile instead. As a bonus, it also gives you all of the processor-specific defines so you don't have to look up what the value of, say, the x86-64 relocations are: https://docs.rs/object/0.25.3/object/elf/index.html#constants https://docs.rs/object/0.25.3/object/elf/index.html#constant...

derefr 5y ago
My point wasn't about whether you can get into to the nitty-gritty leaf nodes of complex structures in the fully-decoded data; it was about whether the representation it outputs losslessly represents unparseable data elements while still doing best-effort to decode what it can, such that you end up with a representation that was decoded "as much as possible" where the decoded parts can be used to figure out why the non-decoded parts didn't decode, while also not obscuring what the non-decoded parts "say". I'm using "high-level" here to mean "was successfully transformed through all the decoding/lexing/parsing/cross-reference stages", and "low[er]-level" to mean "failed to be transformed by some of those stages." Which is non-normative, I guess, but this sort of "layers of decoded-ness" representation is what you expect from tools like Wireshark or binwalk.
- jcranmer 5y ago
  Ah, I see what you mean now. Yes, object/gimli are also this kind of low-level. Basically, when you parse the file, you're not actually parsing the file, but you're parsing each element of the structure one bit of a time. So parsing FileHeader will make sure a) the data is correctly aligned [since it's UB in Rust to have underaligned data] and b) that the e_ident bits are actually the magic number for an ELF file. Want to list all the sections? That's when it's actually going to check that a) the section header offset actually points to valid data and b) the number of section headers exist and is sane, but again, it doesn't actually verify that the section headers themselves make any sense whatsoever.