2 ms·
> Now suppose you want to debug a compiler backend targetting x86. Good luck, x86 disassembly is an undeciable problem because you don't know where the instruct
by jcranmer 2y ago
> Now suppose you want to debug a compiler backend targetting x86. Good luck, x86 disassembly is an undeciable problem because you don't know where the instructions start. Meanwhile ARM has two kinds of instructions (full-size and half-size, called "thumb"). Thanks to fixed-length instructions and alignment rules, you always know whether you're in thumb or not.
This juxtaposition is (I assume unintentionally) hilarious. ARM is the ISA where you might reasonably see two different ISAs (and thus a disassembler needs to handle both) in the same object file, where x86 only has one, at least since the mid-90's. Yet somehow x86 is undecidable and ARM is straightforward!
In reality, the problem you're talking about for x86 is trivially solvable with what's known as recursive disassembly. You start disassembling from known function locations (since you mention debugging a compiler, that means your binary should be fully symbolized, but finding common function entry points for an unsymbolized binary isn't that difficult of a challenge), and then continue disassembling instructions until you get to various kinds of jump instructions. Then you add branch locations to the list, and rinse and repeat until you're out of new locations to disassemble.