3 ms·

Would be interesting to read about tools used and more technical information on RE process.

by xvilka 8y ago

pjc50 8y ago
I'm not sure there is much standard tooling to a job like this, other than staring at hex dumps a lot. He mentions binwalk, which is a good place to start (see https://reverseengineering.stackexchange.com/questions/17262/binwalk-alternative https://reverseengineering.stackexchange.com/questions/17262... for a good workflow). Other than that it's a question of building job-specific scaffolding as you go. > I decided to “emulate” x86 real mode inside C by translating x86 real mode disassembly into C very directly, modelling segment registers explicitly in C (from the section described as "mentally draining") Once he'd figured out the algorithm: > the particular LZSS format used here turns out to originate from some public domain DOS code which someone posted on a Japanese BBS in 1988 Ah, the glory of the Internet long tail. > After writing the shellcode to facilitate this mode of access, I finally had a way to access the APE's address space. (relating to the use of provided functions to inject code into the APE, allowing inspection of its point of view and boot code) > Although the diagnostic tool was quite helpful, ironically this is not because I ever managed to run it. Neither the DOS, UEFI nor Windows versions have ever worked for me. Instead, the diagnostic tools are useful because they contain various routines to probe APE registers, and then print the contents of these registers along with their names. It's not much information, but it's all I have, and it makes all the difference. Pretty much everything I know about the APE that isn't guessed is from the dry reverse engineering of this diagnostic tool. More reverse engineering heroics, this time on the support tools. Rather like marathon running or the ascent of Everest, a prime attribute for doing the thing is a refusal to give up long after the process has become painful and unrewarding.
viraptor 8y ago
For RE/exploration the most popular tools at the moment are probably radare2, (with various guide) hopper, ida. Maybe ollydbg is you deal with windows. Each has its fans.
MegaDeKay 8y ago
On the topic of tools, the National Security Agency (!) very recently open sourced (!) their tool for reverse engineering under the Apache 2.0 license (!) and it apparently does quite well against the closed source and expensive IDA. Check ghidra out. They are actively pulling in patches from the community, tracking issues, etc. I think this will progress forward quite quickly. https://github.com/NationalSecurityAgency/ghidra https://github.com/NationalSecurityAgency/ghidra
- dfox 8y ago
  Note that ghidra is not that useful.for analyzing things that are not well formed loadable object files, because (in comparison to IDA) it relies on autodetection too much. See for example almost any screenshots of ghidra in mainstream-ish media which show completely nonsencial i386 disassembly interspersed with meaningfull looking autogenerated labels and comments. That is what reliably happens when you feed .NET binary into the thing, as it sees embedded debug info and proceeds to disassemble the .NET bytecode as i386 code...
hlandau 8y ago
There weren't that many off-the-shelf tools involved, other than things like binwalk and disassemblers. My workflow can also be hilariously ghetto at times: I like to output hexdump -C to a file, then annotate that in vim. For example: https://github.com/hlandau/ortega/blob/master/notes/bcm5719_talos.txt https://github.com/hlandau/ortega/blob/master/notes/bcm5719_... Lots of tools were written from scratch. I wrote otgdbg for probing the device; this program has tons of subcommands to let me manipulate the device in various ways, get/set registers, boot a program on the MIPS side from memory, boot a program on the APE side from memory, copy a new image to flash, etc. otgimg examines firmware images and prints information about them, like the MAC addresses in the configuration block, etc. apeimg shows information about APE firmware images and can decompress them. Since the image formats are custom, I had to use linker scripts to build the images, but some fixups could only be done programmatically, like calculating CRC fields. These fixups were done with small C programs which the build system runs afterwards. The APE used a more sophisticated image format with section headers, etc. The fixup program for the APE had to compress some of the sections, etc. before setting the CRCs. These tools are all available in the repository, but most of them link to small amounts of proprietary/reversed code which is automatically scrubbed from the public release. It's not a large amount of code which would need to be replaced, though, if someone wants a tool like otgdbg to probe Broadcom NICs in arbitrary ways. Oh, I should also mention that using clang and lld rather than gcc/binutils made targeting different architectures a breeze. It's long been a bone of irritation to me that you have to recompile gcc to retarget it; with clang, I could target both MIPS and ARM without compiling a new toolchain. https://github.com/hlandau/ortega/blob/master/cc_mips https://github.com/hlandau/ortega/blob/master/cc_mips https://github.com/hlandau/ortega/blob/master/cc_arm https://github.com/hlandau/ortega/blob/master/cc_arm
- markjenkinswpg 8y ago
  Great work Hugo. Thank you!

Would be interesting to read about tools used and more technical information on RE process.

by xvilka 8y ago

Would be interesting to read about tools used and more technical information on RE process.

I'm not sure there is much standard tooling to a job like this, other than staring at hex dumps a lot. He mentions binwalk, which is a good place to start (see https://reverseengineering.stackexchange.com/questions/17262/binwalk-alternative https://reverseengineering.stackexchange.com/questions/17262... for a good workflow). Other than that it's a question of building job-specific scaffolding as you go. > I decided to “emulate” x86 real mode inside C by translating x86 real mode disassembly into C very directly, modelling segment registers explicitly in C (from the section described as "mentally draining") Once he'd figured out the algorithm: > the particular LZSS format used here turns out to originate from some public domain DOS code which someone posted on a Japanese BBS in 1988 Ah, the glory of the Internet long tail. > After writing the shellcode to facilitate this mode of access, I finally had a way to access the APE's address space. (relating to the use of provided functions to inject code into the APE, allowing inspection of its point of view and boot code) > Although the diagnostic tool was quite helpful, ironically this is not because I ever managed to run it. Neither the DOS, UEFI nor Windows versions have ever worked for me. Instead, the diagnostic tools are useful because they contain various routines to probe APE registers, and then print the contents of these registers along with their names. It's not much information, but it's all I have, and it makes all the difference. Pretty much everything I know about the APE that isn't guessed is from the dry reverse engineering of this diagnostic tool. More reverse engineering heroics, this time on the support tools. Rather like marathon running or the ascent of Everest, a prime attribute for doing the thing is a refusal to give up long after the process has become painful and unrewarding.

viraptor 8y ago

For RE/exploration the most popular tools at the moment are probably radare2, (with various guide) hopper, ida. Maybe ollydbg is you deal with windows. Each has its fans.

MegaDeKay 8y ago

On the topic of tools, the National Security Agency (!) very recently open sourced (!) their tool for reverse engineering under the Apache 2.0 license (!) and it apparently does quite well against the closed source and expensive IDA. Check ghidra out. They are actively pulling in patches from the community, tracking issues, etc. I think this will progress forward quite quickly. https://github.com/NationalSecurityAgency/ghidra https://github.com/NationalSecurityAgency/ghidra

dfox 8y ago
Note that ghidra is not that useful.for analyzing things that are not well formed loadable object files, because (in comparison to IDA) it relies on autodetection too much. See for example almost any screenshots of ghidra in mainstream-ish media which show completely nonsencial i386 disassembly interspersed with meaningfull looking autogenerated labels and comments. That is what reliably happens when you feed .NET binary into the thing, as it sees embedded debug info and proceeds to disassemble the .NET bytecode as i386 code...

hlandau 8y ago

There weren't that many off-the-shelf tools involved, other than things like binwalk and disassemblers. My workflow can also be hilariously ghetto at times: I like to output hexdump -C to a file, then annotate that in vim. For example: https://github.com/hlandau/ortega/blob/master/notes/bcm5719_talos.txt https://github.com/hlandau/ortega/blob/master/notes/bcm5719_... Lots of tools were written from scratch. I wrote otgdbg for probing the device; this program has tons of subcommands to let me manipulate the device in various ways, get/set registers, boot a program on the MIPS side from memory, boot a program on the APE side from memory, copy a new image to flash, etc. otgimg examines firmware images and prints information about them, like the MAC addresses in the configuration block, etc. apeimg shows information about APE firmware images and can decompress them. Since the image formats are custom, I had to use linker scripts to build the images, but some fixups could only be done programmatically, like calculating CRC fields. These fixups were done with small C programs which the build system runs afterwards. The APE used a more sophisticated image format with section headers, etc. The fixup program for the APE had to compress some of the sections, etc. before setting the CRCs. These tools are all available in the repository, but most of them link to small amounts of proprietary/reversed code which is automatically scrubbed from the public release. It's not a large amount of code which would need to be replaced, though, if someone wants a tool like otgdbg to probe Broadcom NICs in arbitrary ways. Oh, I should also mention that using clang and lld rather than gcc/binutils made targeting different architectures a breeze. It's long been a bone of irritation to me that you have to recompile gcc to retarget it; with clang, I could target both MIPS and ARM without compiling a new toolchain. https://github.com/hlandau/ortega/blob/master/cc_mips https://github.com/hlandau/ortega/blob/master/cc_mips https://github.com/hlandau/ortega/blob/master/cc_arm https://github.com/hlandau/ortega/blob/master/cc_arm

markjenkinswpg 8y ago
Great work Hugo. Thank you!