5 ms·

> The source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your org

> The source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation. Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build. Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.

rockwotj 5y ago
Yeah our org does the same thing and I'd say this is the gold standard (or at least mine) but sadly I'm assuming most companies don't do this, not everyone runs on docker nor has a build pipeline like this.
- yjftsjthsd-h 5y ago
  Sure, but everyone should be able to have their built process embed that info into build artifacts, whether it's by passing GCC a DEFINE flag that ends up in --version output, passing nasm a data structure embedded at X bytes into the binary, a version.txt in the output zip, an autogenerated comment inlined in each HTML/JS/CSS file, whatever... somehow, you can embed build info.
jiggawatts 5y ago
This happened last week, for real with ${dayjob} customer. Put yourself in their shoes for a second. Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive. Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1] So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent. Now what? No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago. A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server. There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use. Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on. [1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir. [2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)
- arpa 5y ago
  > Now what? Goodbye and thanks for all the fish. Run!
- kitd 5y ago
  So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent. Now what? The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.
  - jiggawatts 5y ago
    That provides incomplete protection even for this Log4j vulnerability, and similarly trivial "self help patching" may not be available for future vulnerabilities.
  - emn13 5y ago
    This generalizes to "apply a vulnerability-specific tricky auto-fix to a binary it was never tested on". That might be a reasonable solution if there's nothing better (and since speed is of the essence, there might not be). But it's not exactly a reliable solution, and not every vulnerability will have hacks like that available, nor is it really clear to what extent they fix the problem; it's pretty easy to miss something using this kind of approach, or to have some future deploy bot restore the vulnerability in normal operation. And hacks like this can break things, both directly (the patch breaks actually used functionality unexpectedly), and indirectly (the fact that the software was patched at all breaks sanity checks or other cleverness like binary diffing or whatever). I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.
- Macha 5y ago
  > No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago. Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it. If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host. Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.

rockwotj 5y ago

Yeah our org does the same thing and I'd say this is the gold standard (or at least mine) but sadly I'm assuming most companies don't do this, not everyone runs on docker nor has a build pipeline like this.

yjftsjthsd-h 5y ago
Sure, but everyone should be able to have their built process embed that info into build artifacts, whether it's by passing GCC a DEFINE flag that ends up in --version output, passing nasm a data structure embedded at X bytes into the binary, a version.txt in the output zip, an autogenerated comment inlined in each HTML/JS/CSS file, whatever... somehow, you can embed build info.

jiggawatts 5y ago

This happened last week, for real with ${dayjob} customer. Put yourself in their shoes for a second. Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive. Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1] So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent. Now what? No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago. A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server. There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use. Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on. [1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir. [2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)

arpa 5y ago
> Now what? Goodbye and thanks for all the fish. Run!
kitd 5y ago
So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent. Now what? The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.
- jiggawatts 5y ago
  That provides incomplete protection even for this Log4j vulnerability, and similarly trivial "self help patching" may not be available for future vulnerabilities.
- emn13 5y ago
  This generalizes to "apply a vulnerability-specific tricky auto-fix to a binary it was never tested on". That might be a reasonable solution if there's nothing better (and since speed is of the essence, there might not be). But it's not exactly a reliable solution, and not every vulnerability will have hacks like that available, nor is it really clear to what extent they fix the problem; it's pretty easy to miss something using this kind of approach, or to have some future deploy bot restore the vulnerability in normal operation. And hacks like this can break things, both directly (the patch breaks actually used functionality unexpectedly), and indirectly (the fact that the software was patched at all breaks sanity checks or other cleverness like binary diffing or whatever). I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.
Macha 5y ago
> No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago. Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it. If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host. Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.