4 ms·

The question I have is how does something like this end up undetected in the officially signed update? I would imagine internally there would be a source contro

by lovecg 6y ago

The question I have is how does something like this end up undetected in the officially signed update? I would imagine internally there would be a source control system with auditable commits, and the commit hash would be signed with the executable — is that not standard practice?

gauravphoenix 6y ago
Nope. Theoretically it should be happening but it rarely happens. In fact, given the sophistication of this particular adversary, they would have just compromised the build server(and resign the binary) and I doubt anyone goes to the length of verifying the build server builds against some reference.
- numbsafari 6y ago
  Maybe we should be, in response to knowing the viability and use of CI hacks? Ensure you have a reproducible build, then randomly build on a different machine and compare file signatures of the results. Do it every so often with a “clean room” machine. Probably no need to run parallel infra that’s just as likely to be hacked.
jcims 6y ago
While it’s absolutely not a requirement, my first guess would be an inside job. They hire for locations all over the world, and once you’re in it would be much easier to spend your time finding and/or creating the scenario for this functionality to get into the build without detection.
- thu2111 6y ago
  No way. So far there have been zero APT attacks traced back to inside jobs. That's not how modern spy agencies work anymore. Code signing hardly means much. All you have to do is compromise CI to make an organisation sign a binary that isn't based on reviewed code. How hard is that? I think for the vast majority of us we have to admit our CI would be very easy to compromise. Especially if you can get administrator credentials as easily as these guys apparently can. But even without that, CI executes and runs arbitrary code that's intended to only come from trusted individuals. Step 1. Find a third party package or library that is being downloaded by a package manager onto a CI box. Step 2. Compromise the open source dev's laptop by phishing them or something else really basic. Step 3. Put a back door into that package. Step 4. Wait for it to be downloaded to and executed on SolarWinds CI. Step 5. Insert a rootkit into CI. Now that rootkit can just systematically insert the final back door into any build of that file it sees using link time editing. Every build will have the back door and the owners will eventually sign the binary and push it out to production. In many orgs CI can actually request signing of any binary it wants, so you don't even have to do that. You can just hack the CI machine, grab the credential used to authenticate to the HSM, sign your preferred binary and then ensure users are downloading it.
  - numbsafari 6y ago
    And so perhaps the question that should be keeping the rest of us up at night this week is this: What if this exploit came through a compromised third-party dependency (eg npm or pypy or deb or rpm or docker image) that made its way into SolarWinds CI system? Is that something the rest of us now face?
    - lovecg 6y ago
      That’s a very good point. The issues with Npm in particular have been studied: https://news.ycombinator.com/item?id=21111911 https://news.ycombinator.com/item?id=21111911
  - xorcist 6y ago
    > So far there have been zero APT attacks traced back to inside jobs. That means we're not finding them, not that they don't exist. Considering how easy it is to create plausible external attack vectors from the inside, I would be surprised if it didn't exist. As you said, intelligence agencies have had people on the inside of important organisations since forever. That's what they do. It's not like everyone decided inside access is useless, it's more likely it has become easier to not burn insiders by them having to work hands on.
  - webmaven 6y ago
    > Step 2. Compromise the open source dev's laptop by phishing them or something else really basic. Step 2 can also be: Buy the open source package from the maintaining dev.
    - Aperocky 6y ago
      Hence the most valuable open source package ever: isOdd.
  - rcxdude 6y ago
    The attack you describe is plausible but very visible (your backdoor code will be visible on the package manager and generally not easy to take down). I doubt it's the kind of thing that is likely to be used as part of this kind of attack, which wants to remain invisible for as long as possible.
octoberfranklin 6y ago
Codesigning creates a ton of false sense of security. It has its place, but it has become massively overused.
papito 6y ago
I believe the SolarWinds build system was compromised, sneaking in the malicious code into a legitimately signed build.

by lovecg 6y ago

gauravphoenix 6y ago

Nope. Theoretically it should be happening but it rarely happens. In fact, given the sophistication of this particular adversary, they would have just compromised the build server(and resign the binary) and I doubt anyone goes to the length of verifying the build server builds against some reference.

numbsafari 6y ago
Maybe we should be, in response to knowing the viability and use of CI hacks? Ensure you have a reproducible build, then randomly build on a different machine and compare file signatures of the results. Do it every so often with a “clean room” machine. Probably no need to run parallel infra that’s just as likely to be hacked.

jcims 6y ago

While it’s absolutely not a requirement, my first guess would be an inside job. They hire for locations all over the world, and once you’re in it would be much easier to spend your time finding and/or creating the scenario for this functionality to get into the build without detection.

thu2111 6y ago
No way. So far there have been zero APT attacks traced back to inside jobs. That's not how modern spy agencies work anymore. Code signing hardly means much. All you have to do is compromise CI to make an organisation sign a binary that isn't based on reviewed code. How hard is that? I think for the vast majority of us we have to admit our CI would be very easy to compromise. Especially if you can get administrator credentials as easily as these guys apparently can. But even without that, CI executes and runs arbitrary code that's intended to only come from trusted individuals. Step 1. Find a third party package or library that is being downloaded by a package manager onto a CI box. Step 2. Compromise the open source dev's laptop by phishing them or something else really basic. Step 3. Put a back door into that package. Step 4. Wait for it to be downloaded to and executed on SolarWinds CI. Step 5. Insert a rootkit into CI. Now that rootkit can just systematically insert the final back door into any build of that file it sees using link time editing. Every build will have the back door and the owners will eventually sign the binary and push it out to production. In many orgs CI can actually request signing of any binary it wants, so you don't even have to do that. You can just hack the CI machine, grab the credential used to authenticate to the HSM, sign your preferred binary and then ensure users are downloading it.
- numbsafari 6y ago
  And so perhaps the question that should be keeping the rest of us up at night this week is this: What if this exploit came through a compromised third-party dependency (eg npm or pypy or deb or rpm or docker image) that made its way into SolarWinds CI system? Is that something the rest of us now face?
  - lovecg 6y ago
    That’s a very good point. The issues with Npm in particular have been studied: https://news.ycombinator.com/item?id=21111911 https://news.ycombinator.com/item?id=21111911
- xorcist 6y ago
  > So far there have been zero APT attacks traced back to inside jobs. That means we're not finding them, not that they don't exist. Considering how easy it is to create plausible external attack vectors from the inside, I would be surprised if it didn't exist. As you said, intelligence agencies have had people on the inside of important organisations since forever. That's what they do. It's not like everyone decided inside access is useless, it's more likely it has become easier to not burn insiders by them having to work hands on.
- webmaven 6y ago
  > Step 2. Compromise the open source dev's laptop by phishing them or something else really basic. Step 2 can also be: Buy the open source package from the maintaining dev.
  - Aperocky 6y ago
    Hence the most valuable open source package ever: isOdd.
- rcxdude 6y ago
  The attack you describe is plausible but very visible (your backdoor code will be visible on the package manager and generally not easy to take down). I doubt it's the kind of thing that is likely to be used as part of this kind of attack, which wants to remain invisible for as long as possible.

octoberfranklin 6y ago

Codesigning creates a ton of false sense of security. It has its place, but it has become massively overused.

papito 6y ago

I believe the SolarWinds build system was compromised, sneaking in the malicious code into a legitimately signed build.