5 ms·

I'm curious what feature specifically prompted the NSA to develop their own IDA Pro alternative. I mean, someone somewhere at the NSA must have been trying to d

I'm curious what feature specifically prompted the NSA to develop their own IDA Pro alternative. I mean, someone somewhere at the NSA must have been trying to do something with IDA Pro only to repeatedly fail before the decision was made that whatever the NSA was trying to do warranted developing their own IDA Pro... right? Or perhaps they used IDA Pro so often and grew so frustrated by it that they started their own?

mox1 8y ago
Is it possible that maybe this pre-dates IDA Pro? Ever tried to use IDA Pro on the same project with a co-worker...at the same time?
- souprock 8y ago
  I believe the included decompiler predates the one that goes with IDA Pro, so that could be a motivation. IDA Pro itself is really really old, having once been a 16-bit DOS shareware program, which explains why IDA Pro doesn't use the GUI shortcuts that were standardized much later. IDA Pro still doesn't support collaboration, although there are very broken hacks that attempt to add it. Binary Ninja supports collaboration if you buy the enterprise edition.
tptacek 8y ago
Hex-Rays can be hard to deal with, and the IC deals pretty extensively with large federal contractors like Raytheon, so it's possible they just needed something as capable as IDA that they could roll out across all their suppliers to use as a common toolchain and interchange format. But it's also possible this is just sort of a labor of love type thing.
- comex 8y ago
  There’s a third possibility: they wanted a piece of software they could customize to meet their needs. Admittedly, for simple bugfixes and the like, Hex-Rays’ support is known for being quite responsive (as suits the small number of customers). There’s also a quite large (albeit poorly-documented and crash-prone) SDK, which can handle a wide variety of needs and has gained functionality over time. But if you want to add new functionality that can’t be implemented using the SDK? (That includes just about anything related to the decompiler.) You’re at the mercy of Ilfak’s priority list. Well, mere mortals are, at least; the NSA has enough money that it might be able to set up some sort of special contract with Hex-Rays, but I suspect that’s easier said than done. From that perspective, the ideal is what the NSA ended up with, a codebase whose development is fully in-house. Notably, though, second best would be to just have access to the source code of an existing tool, so you can at least make your own patches if necessary, even if you’re not in control of the codebase’s overall direction. Did the NSA ever seek that in IDA’s case, and could they have obtained it if they did? I don’t know the answer to either question… but source access certainly isn’t offered to typical customers. In general I’m surprised that “paid + source access for customers” isn’t a more popular model of software development. From my perspective, which admittedly is very different from the NSA’s, I was never very interested in low-cost IDA competitors like Hopper or Binary Ninja, but I’m very excited about Ghidra. Why? Partly because it’s a more full-fledged competitor in terms of feature set, I admit – but the competitors I mentioned are bound to narrow the gap over time. Partly because of cost: I myself am at a point where I could justify the $600/y for Binary Ninja’s commercial edition, or even the order-of-magnitude-higher cost of the Hex-Rays decompilers, without wincing too badly. but I believe that reverse engineering should be accessible to beginners and amateurs. (Piracy is a partial solution, including in IDA’s case, but some people don’t like to do that.). But the main reason I’m excited about Ghidra is that I have the source code. As a concrete example, I’ve spent a good amount of time reverse engineering software for the Nintendo Wii and Wii U. Both consoles have a main CPU based on the PowerPC architecture, but with a custom ISA extension for an extremely barebones version of SIMD. Well, both Hex-Rays and Ghidra support PowerPC decompilation (although that’s a relatively recent development), but unsurprisingly, neither of them have full support for that ISA extension. IDA actually does have built-in support for disassembling it, but AFAIK not for decompiling; Ghidra doesn’t seem to support it at all (but I may just need to configure it properly). What can I do? Well, in practice, nothing, because I don’t care about the Wii U anymore. But if Ghidra had been released a few years ago, I’m pretty sure I would have gone and implemented support for the extension myself; I haven’t looked at Ghidra’s source yet, but since it already supports other vector ISAs, it probably wouldn’t be that hard. With IDA, I was stuck. The SDK supports adding custom instruction sets for disassembly, but the decompiler SDK is so limited that supporting them there is either impossible or at least would be a huge hack. And that’s just one of many customizations I‘ve wanted over the years. Some of them are probably easier said than implemented, but at least now I can put that to the test!
  - alexozer 8y ago
    I'm also in the situation where I'd like to try to reverse binaries for the custom PowerPC chips in Gamecube/Wii/Wii U. I'm wondering if it might be easier to rewrite the assembly such that the SIMD instructions are rewritten in terms of standard instructions, and then just use that. I hope it's possible.
  - occamrazor 8y ago
    > In general I’m surprised that “paid + source access for customers” isn’t a more popular model of software development. Microsoft has a "shared source" agreement with governments.
  - PeterisP 8y ago
    It's worth to note NSA has quite specific needs. If you need some very specific small functionality by next monday for $1million, then that development can be arranged. However, NSA could also reasonably want that their targets (who have extensive capabilities of their own, likely including insiders in various companies) can't find out that NSA needs that very specific small functionality by next monday. They may not care if that functionality becomes available to the general public sometime in the next year (preferably in a more general manner that covers the reasonable/common usecases instead of just the one NSA had at the moment), but leaking the information that you needed (and thus probably used) X at time Y often isn't acceptable. At the very least, they'd need every single employee who works on that feature or can see that this feature was developed to be vetted by them i.e. to have a security clearance; and that also requires the company to have the appropriate processes and infrastructure for separate, secret codebases and builds that can't be seen by uncleared people. And that is something many companies can't or don't want to provide.
- smokeyj 8y ago
  Alternative take. Want to secure national infrastructure? Release the tools to do it for free. Viva la open-source revolution
thornjm 8y ago
A few reasons I can think of: 1. Collaborative. 2. supporting classified proprietary architectures (think missile chips or something) 3. The intermediate representation (architecture independent representation of code) can be integrated in to many other classified tools. Maybe for automated analysis for example.
- perlgeek 8y ago
  4. Managing licenses is a huge PITA, presumably especially in environments with lots of classified information.
  - janekm 8y ago
    Good point... "we need a site license. No, I can't tell you for how many employees, that's classified. No, I can't tell you who we are, that's classified. No, I can't tell you what we are working on, that's classified. Hello? Hello? Darn they hung up again..."
    - ktjfi 8y ago
      I think all three letter agencies create front companies for this kind of stuff?
      - rasz 8y ago
        https://www.battelle.org/ https://www.battelle.org/
    - jeffreyrogers 8y ago
      When I worked for a hedge fund we had to deal with this sort of thing (not classified obviously, but wanted exemptions from certain things), but it was actually pretty easy to deal with. They just charge you more to get custom terms.
- pbhjpbhj 8y ago
  Operational reasons? They found a compromisable worker was employed there, or they somehow put modifications in IDA making the software compromisable and so not safe for them?
yalogin 8y ago
May be the NSA probably had this before IDA came about. It’s the NSA after all. They are now releasing it because it’s not a competitive edge anymore and can be used as a recruiting tool.
- smolsky 8y ago
  I don't think that is possible. I used early versions of IDA Pro on MS-DOS in 1996... and it had the core analysis and interactive disasm mode then. Ghidra seems to be based on this very concept, all the way down to the "XREF" labels. Even the sub-panels in the UI look the same: Imports, Exports, Symbols...

mox1 8y ago

Is it possible that maybe this pre-dates IDA Pro? Ever tried to use IDA Pro on the same project with a co-worker...at the same time?

souprock 8y ago
I believe the included decompiler predates the one that goes with IDA Pro, so that could be a motivation. IDA Pro itself is really really old, having once been a 16-bit DOS shareware program, which explains why IDA Pro doesn't use the GUI shortcuts that were standardized much later. IDA Pro still doesn't support collaboration, although there are very broken hacks that attempt to add it. Binary Ninja supports collaboration if you buy the enterprise edition.

tptacek 8y ago

Hex-Rays can be hard to deal with, and the IC deals pretty extensively with large federal contractors like Raytheon, so it's possible they just needed something as capable as IDA that they could roll out across all their suppliers to use as a common toolchain and interchange format. But it's also possible this is just sort of a labor of love type thing.

comex 8y ago
There’s a third possibility: they wanted a piece of software they could customize to meet their needs. Admittedly, for simple bugfixes and the like, Hex-Rays’ support is known for being quite responsive (as suits the small number of customers). There’s also a quite large (albeit poorly-documented and crash-prone) SDK, which can handle a wide variety of needs and has gained functionality over time. But if you want to add new functionality that can’t be implemented using the SDK? (That includes just about anything related to the decompiler.) You’re at the mercy of Ilfak’s priority list. Well, mere mortals are, at least; the NSA has enough money that it might be able to set up some sort of special contract with Hex-Rays, but I suspect that’s easier said than done. From that perspective, the ideal is what the NSA ended up with, a codebase whose development is fully in-house. Notably, though, second best would be to just have access to the source code of an existing tool, so you can at least make your own patches if necessary, even if you’re not in control of the codebase’s overall direction. Did the NSA ever seek that in IDA’s case, and could they have obtained it if they did? I don’t know the answer to either question… but source access certainly isn’t offered to typical customers. In general I’m surprised that “paid + source access for customers” isn’t a more popular model of software development. From my perspective, which admittedly is very different from the NSA’s, I was never very interested in low-cost IDA competitors like Hopper or Binary Ninja, but I’m very excited about Ghidra. Why? Partly because it’s a more full-fledged competitor in terms of feature set, I admit – but the competitors I mentioned are bound to narrow the gap over time. Partly because of cost: I myself am at a point where I could justify the $600/y for Binary Ninja’s commercial edition, or even the order-of-magnitude-higher cost of the Hex-Rays decompilers, without wincing too badly. but I believe that reverse engineering should be accessible to beginners and amateurs. (Piracy is a partial solution, including in IDA’s case, but some people don’t like to do that.). But the main reason I’m excited about Ghidra is that I have the source code. As a concrete example, I’ve spent a good amount of time reverse engineering software for the Nintendo Wii and Wii U. Both consoles have a main CPU based on the PowerPC architecture, but with a custom ISA extension for an extremely barebones version of SIMD. Well, both Hex-Rays and Ghidra support PowerPC decompilation (although that’s a relatively recent development), but unsurprisingly, neither of them have full support for that ISA extension. IDA actually does have built-in support for disassembling it, but AFAIK not for decompiling; Ghidra doesn’t seem to support it at all (but I may just need to configure it properly). What can I do? Well, in practice, nothing, because I don’t care about the Wii U anymore. But if Ghidra had been released a few years ago, I’m pretty sure I would have gone and implemented support for the extension myself; I haven’t looked at Ghidra’s source yet, but since it already supports other vector ISAs, it probably wouldn’t be that hard. With IDA, I was stuck. The SDK supports adding custom instruction sets for disassembly, but the decompiler SDK is so limited that supporting them there is either impossible or at least would be a huge hack. And that’s just one of many customizations I‘ve wanted over the years. Some of them are probably easier said than implemented, but at least now I can put that to the test!
- alexozer 8y ago
  I'm also in the situation where I'd like to try to reverse binaries for the custom PowerPC chips in Gamecube/Wii/Wii U. I'm wondering if it might be easier to rewrite the assembly such that the SIMD instructions are rewritten in terms of standard instructions, and then just use that. I hope it's possible.
- occamrazor 8y ago
  > In general I’m surprised that “paid + source access for customers” isn’t a more popular model of software development. Microsoft has a "shared source" agreement with governments.
- PeterisP 8y ago
  It's worth to note NSA has quite specific needs. If you need some very specific small functionality by next monday for $1million, then that development can be arranged. However, NSA could also reasonably want that their targets (who have extensive capabilities of their own, likely including insiders in various companies) can't find out that NSA needs that very specific small functionality by next monday. They may not care if that functionality becomes available to the general public sometime in the next year (preferably in a more general manner that covers the reasonable/common usecases instead of just the one NSA had at the moment), but leaking the information that you needed (and thus probably used) X at time Y often isn't acceptable. At the very least, they'd need every single employee who works on that feature or can see that this feature was developed to be vetted by them i.e. to have a security clearance; and that also requires the company to have the appropriate processes and infrastructure for separate, secret codebases and builds that can't be seen by uncleared people. And that is something many companies can't or don't want to provide.
smokeyj 8y ago
Alternative take. Want to secure national infrastructure? Release the tools to do it for free. Viva la open-source revolution

thornjm 8y ago

A few reasons I can think of: 1. Collaborative. 2. supporting classified proprietary architectures (think missile chips or something) 3. The intermediate representation (architecture independent representation of code) can be integrated in to many other classified tools. Maybe for automated analysis for example.

perlgeek 8y ago
4. Managing licenses is a huge PITA, presumably especially in environments with lots of classified information.
- janekm 8y ago
  Good point... "we need a site license. No, I can't tell you for how many employees, that's classified. No, I can't tell you who we are, that's classified. No, I can't tell you what we are working on, that's classified. Hello? Hello? Darn they hung up again..."
  - ktjfi 8y ago
    I think all three letter agencies create front companies for this kind of stuff?
    - rasz 8y ago
      https://www.battelle.org/ https://www.battelle.org/
  - jeffreyrogers 8y ago
    When I worked for a hedge fund we had to deal with this sort of thing (not classified obviously, but wanted exemptions from certain things), but it was actually pretty easy to deal with. They just charge you more to get custom terms.
pbhjpbhj 8y ago
Operational reasons? They found a compromisable worker was employed there, or they somehow put modifications in IDA making the software compromisable and so not safe for them?

yalogin 8y ago

May be the NSA probably had this before IDA came about. It’s the NSA after all. They are now releasing it because it’s not a competitive edge anymore and can be used as a recruiting tool.

smolsky 8y ago
I don't think that is possible. I used early versions of IDA Pro on MS-DOS in 1996... and it had the core analysis and interactive disasm mode then. Ghidra seems to be based on this very concept, all the way down to the "XREF" labels. Even the sub-panels in the UI look the same: Imports, Exports, Symbols...