9 ms·
I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an opera
by jiggawatts 5y ago
I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime. Windows, Linux, Java, and .NET are all commonly updated, but "modules" aren't.
This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide.
Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.
If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?
I had to help an organisation with log4j that had on the order of 5K distinct executables/applications across 3K servers on two clouds and three on-prem networks. At least with log4j it's a simple matter of finding JAR files and scanning their contents. They're just zip files! With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.
Introspection and post-shipping updates for security are mandatory. Rust and Go like to pretend they aren't, because they originate from organisations that are in total, end-to-end control of the software on their own network. Google famously uses a monorepo and can build everything they run from scratch in short order. The shortcuts they can take with their post-release management will never work for ordinary organisations. Never.
So let this be a lesson: Log4j was made for a language that at least allowed us all to find the issue and fix it ourselves. We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.
- c-cube 5y agoIf you still have the source code for your rust code, it contains the lock file. Look in there and you'll see all dependencies and their version. Why do you want to look at the binary instead?
- tatersolid 5y agoIn commercial software the customers don’t have the source files…
- Perseids 5y agoWell, we're in a thread that started with > I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime. If there is the possibility to plan ahead from a sufficiently strong bargaining position, you will not end up in the situation you described. The seller should either continue to support it with a good SLA, or you get the source code with a turn-key build system if the seller discontinues its support. Or you switch out the software if the SLA ends. Should you not be able to agree on such terms, I don't think you would have enough influence to tell the seller to write their software in such a way that dependencies are easy to hotfix/upgrade.
- Spooky23 5y agoROFL. Ok, after putting the unicorns herd to bed, an event happens and congratulations, you now own a license to PeopleSoft v.whatever source code. Oh yeah, you also got the management platform for your network provider too. Now what?
- yjftsjthsd-h 5y agoIt's called code escrow and it's a real thing even if it's news to you. Although yes, it's usually an insurance plan you want to never ever need because using it will suck.
- jiggawatts 5y agoThe source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation. If it came from an external organisation, things are exponentially harder. Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options. IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken. Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.
- staticassertion 5y agoThis sounds like an ops problem. It shouldn't be hard to tell what's running in your production environment and tie that back to a specific commit.
- jethro_tell 5y agoIt shouldn't be hard for a dev to update a dependency which should trigger an auto build and roll your pipeline. Shouldn't be hard for an upstream vendor to patch and have that pulled to kick off a new build. IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.
- bigiain 5y ago> IDK where this idea that no one can know what's in a dependency tree is coming from I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody. How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or… And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?
- nitrogen 5y agoWith languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update. ELF binaries are just collections of another sort. You can scan for symbol names or assembled instructions common to a dependency, and you might be able to update with an LD_PRELOAD library that patches the symbol table. Just look at what game modders have accomplished without access to source code.
- spockz 5y agoBesides this, who actually scans the deployed binaries? Just check which version of the binary was deployed, backtrack to the source, and get the dependency tree from there. Problem solved. (Except for shaded dependencies…, but scanning for that should happen at build time anyway.)
- lazide 5y agoGood luck doing that on the custom software some vendor provided you 5 years ago and since has gone out of business.. and that’s a GOOD scenario for a lot of enterprise software. Sometimes it’s 20 years.
- Too 5y agoLesson learned. Never buy closed source. Imagine the vulnerability would have been in the vendor code instead of in the log4j jar. You’d. still need a way to fix and rebuild it. Note that this doesn’t have to mean a public MIT repo on GitHub. You can have more restrictive gray box licenses between your b2b partners only.
- tormeh 5y agoIf it's a 20 year old closed source application, there probably aren't any exploits in the wild. You can bet there are exploits for log4j.
- hamilyon2 5y agoWe need some sort of escrow for source code of closed-source product. Whenever there is practical need to have access to source code, it should be available.
- staticassertion 5y agoI think you could take this further. There's another "log4j" vulnerability out there right now, I guarantee it. It might be exploited, it might not be. But there's no patch. What are you doing about it? With log4j you have a brutal combination of: 1. RCE 2. Exposure RCE happens frequently but often attackers don't have an easy time getting to the exploitable code. With log4j it's trivial - every app can be owned. But here are some questions: 1. Why do those apps have the ability to make network requests? 2. For the apps that need to make those requests internally, why aren't those over mTLS? 3. For the apps that need to make them externally, why isn't that going through egress proxying? 4. Why are there credentials spewed all over your environment variables across your services? Are they short lived? It's actually not super hard to have a worst-case scenario vulnerability like log4j be not that bad for your organization. A bit of hardening and even if something like this happens you're in a good position to wait, monitor, and patch.
- querulous 5y agoit's definitely time for a language with a capabilities based security model that allows scope to the function/method level, at minimum not just for security either. i can't count the number of times i've traced a performance problem or bug to a piece of code that was doing some io when it didn't need to
- staticassertion 5y agoAgreed. In a capabilities system you'd see that, for some reason, your logger requires all sorts of absurd capabilities for no reason.
- hyiltiz 5y agoHaskell comes to mind?
- twunde 5y agoRegarding your questions, they make sense in the context of modern software built in the last 5 years. However, most large enterprises have a long tail of legacy applications built years ago. These may be inherited from a combination of acquisitions, from vendors that have since gone under, some from. The common factor is that the original developers are long gone, and they may be supported using a skeleton crew or not at all. Documentation if it ever existed is likely lost. What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises
- Perseids 5y ago> We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that. I find it quite ironic to praise the forward-thinking of the company that has been instrumental to bring us into this mess via their vision of loading dependencies at runtime from an online repository. Sun imagined that the code and its configuration does not have to worry about how to fulfill its dependencies, but instead JNDI [1] could magically fetch the appropriate objects from wherever, don't worry. I'd say I would even find it ironic to praise the Java ecosystem, which, due to its excessive over-engineering, has produced best practice frameworks where fully documented behavior has the latent potential to just be catastrophically exploitable by accident, as nobody is able to reasonably understand how all of it plays together. I have a hard time to imagine how you could enable loading of external code in Rust or Go by accident using run-of-the-mill logging frameworks. [1] https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Interface https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Inte...
- watersb 5y agoI'm writing this from memory, with checking any sources. IIRC: Designs like JNDI were widespread in the IT software industry at the time. The academic prototype for all this was CORBA, the Common Object Request Broker Architecture. An object-oriented was to transmit data between systems running on separate processes, CPUs, remote servers. CORBA was a next-gen RPC (Remote Procedure Call) framework, the data format on the wire closely follows the format for C structs using Sun-RPC, but was way better because these CORBA things were discoverable and described at run-time, rather than compile-time libraries. That's right: RPC was initially a Sun UNIX thing. Microsoft needed a way to do this, so they had COM, the Component Object Model, for run-time discovery and linking (late binding).. Microsoft got COM to reach across network connections, "DCOM", for Distributed systems. Meanwhile, corporate IT was still pulling text out of databases and shoving it into C structs. It sucked. Microsoft's object model was better, and Java was better than better, because you didn't have to license it to use it. Well, you did; it was like an MIT License. Reason why IBM writes huge Java systems. And why Oracle owns Java. For some value of ownership. Yay objects!
- bboreham 5y ago
- xyproto 5y agoYou can make dynamically linked executables with both Go and Rust. The compilers have no blame in this.
- butlerm 5y agoSure. But you have a slight problem with ABI stability, at least in Rust, if you ever want to compile an ABI compatible replacement library. Until that problem is resolved, if you want to make a useful replaceable object you need to make a C style foreign function interface and use that, something that isn't exactly convenient for Rust to Rust use, even if rather helpful for making Rust libraries usable from other languages. C++ has a very similar issue, and has had it for years, albeit one mostly resolved by the advent of stable C++ ABI standards.
- Spooky23 5y agoPerhaps everyone is stupid. It’s funny because when I worked in .gov IT, we were all characterized as stupid donkeys by because we weren’t able to just use NPM, etc to do cool kid stuff. Good luck getting security people with the power and brass balls to do that in any company.
- wbl 5y agoGo binaries will contain the commit hash, which contains all the information about what it was compiler with.
- pabs3 5y agoBundling/embedding/vendoring/monorepos and other names for embedded code copies really need to stop being common practice in the tech world.
- spc476 5y agoWhat should be done instead?
- pabs3 5y agoDepend on your dependencies instead of embedding them. Specify version ranges instead of pinning dependencies. Report issues in your dependencies and contribute back fixes for them.
- mtrower 5y ago... hear back from upstream that your fix, developed against version X (that you pinned and use), does not apply to HEAD. Pull HEAD and realize there have been major changes which conflict with your platform, use case, or any number of your other dependencies. Sometimes you can get away with just depending, and that’s nice, but other times there are strong reasons to embed. At that point, you own the code, and can actually do the things you need to do. (Of course, it follows that you own the code - and need to then actually do the things you need to do.) Both approaches have their advantages, and both have potential for technical debt. Which approach offers the most advantage for the least debt will depend on the individual circumstance; I don’t feel that you can really advocate for one over the other in a general sense.
- MereInterest 5y agoWhen HeartBleed came out, there was a single dynamically linked library to be replaced. If you had a vulnerable version of libopenssl, then you replace the .so or .dll with a patched version, restart any applications that were using the old version, and you're set. This works because the binary is compiled against the dynamic library, using the system-provided library. It isn't statically compiled, because even though that frees the developers from worrying about library versions, it makes the sysadmin's job tracking library versions be much harder. It isn't vendored, for the same reason.
- fulafel 5y agoYou can, and people do, scan native binaries for these things. It's called SCA (software composition analysis).
- Macha 5y ago> If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files? Our software scans take 3 forms: 1. Searching for file names 2. Running test exploits that have a payload that just logs the vulnerability with a central monitoring system 3. Dependency resolution and checks against a DB (our solution _already_ supports Cargo.lock files. Rust (and Go, C++, etc.) only prevent option 1. Thanks to bundlers it also can miss it in JS, or shaded fatjars it can also fail kn Java too. So it's not a new challenge
- wynck 5y ago> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime. This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide. Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days. I think this attitude is one of the main issues for this kind of incidents in larger organisations. I'm not sure why you would be willing to lay blame at your colleagues for this. Most of the time, development teams don't tend to like some governance over the code and the dependencies they are pulling in. Not just because they know of course what they are doing, but also because they are under pressure to deliver features. That is what matters for business. Also, convincing management that budget is needed for correct tooling to track all stuff deployed; is also not as straight forward as you seem to suggest. In this case, the problem goes even beyond just the code of your own dev teams. This is embedded in countless software packages deployed all over your organisation. Same here, people want to buy and use whatever they want. And all processes to keep some form of control over it, are mostly seen as overhead. And I'm sure they are lots of "security" people who are just producing documents and policies which are complete detached from reality. But developers who consider security completely as somebody else his responsibility, are a problem as well. The only good think I see coming from this mess, is that security teams probably will get the means to try to get more control and insight over this. For the coming weeks/months at least. After that, everyone in management of dev will be forgotten about it. But something tells me the security people who are working on this right now, won't.
- fnord77 5y ago> With languages that output a single binary by default such as Rust and Go, we would have been screwed. no, best practices include checking in your Cargo lock file