7 ms·
Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF
by jacobyoder 4y ago
Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version.
We had to spend a week back and forth with
"no, we're not vulnerable".
"But we see log4j is right there in the a file in the directory".
"No, it's not vulnerable and it's not being used and we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency"
"But we see log4j is right there in the a file in the directory".
Also, there was no budget to do any sort of rewrite/refactor/etc anyway.
- hedora 4y agoOnce you realize this sort of shallow, automated audit exists to grease the wheels of some security theater operation, this sort of back and forth makes sense. False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.
- tekchip 4y agoSo somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then. I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh. But by in large automation and the problems inherent are required.
- hedora 4y agoYou could set your build up to not auto-download questionable stuff from untrustworthy machines on the Internet.
- paulmd 4y agoThe problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept. In one of my past jobs I was an early-mover on doing a lot of our ops on Linux and the audit tools had no concept of the backport security model whatsoever. If you were running on some kind of LTS distro rather than a current distro, it would see "gosh you're 3 minor versions behind, you have a ton of unpatched vulnerabilities!", but in actuality you didn't, because the fixes got backported into security releases (the "31" in something like "3.1.22~ubuntu31" for example). But the tool was dumb and it just had a table that said "anything under 3.4.14 is vulnerable" even when it wasn't necessarily. OP's "log4j 1.x is not vulnerable to this the first place" is a similar thing where either someone forgot to put in a value for "min vulnerable version" or the tool doesn't understand the concept of min-version at all. To be fair that can be genuinely tricky in java, best-case you are reading manifest files to try and pick out a version, but some legacy stuff doesn't have manifests, especially very legacy fat-jar stuff. Yes, that stuff didn't go away, it's still out there in places! And to be clear this was a long-running issue... it wasn't a "oh our tool doesn't understand LTS, I get it" and they left me alone... this was every month they'd be back at me for a new list of utility vulnerabilities even though I repeatedly explained to them that I had set up apt-get to auto-upgrade every night and reboot, so we were running whatever the latest security backports were available, and that just like the last 10 times this was even more false-positives. And again, this isn't just "we have to run it by you no matter what" and they leave you alone either. What security really wanted was for me to run windows and manually install the updates and get back in line with everyone else, even though that would have been a less secure outcome than my locked-down linux boxes. But this wasn't my day-job at the company so to speak, it was helping out another project with some ops and it needed to be as low-effort as possible (and to be fair their requirements weren't steep, auto-patch-and-reboot was fine by them and never caused any breakage during my tenure there). The root problem is that these sorts of tools aren't a substitute for an actual security culture, they're often a symptom of a compliance requirement and the whole thing turns into a box-checking exercise. Someone is on their ass about this because of contractual requirements or PCI requirements, and they bought the cheapest thing off the shelf that would check the box, and they are implicitly showing you here that they don't even understand how to interpret the output of that.
- 4y ago
- horsawlarway 4y agoI was in this industry for a while (I did 5 years full time for a security company). > I will concede there probably are some firms out there acting poorly that way. This is a hilarious take. Here's mine: The vast majority of these firms are here for liability. They do not provide security, they provide security theater so that if/when something goes wrong, the client can claim to have followed best practices, and that it's not their fault. This style of theater is RAMPANT in the industry. Literally most of them come in with some version of a shitty automated tool, often with false positives in the 95%+ range, and then you check off checkboxes to make legal happy, and ensure that your insurance (or your customer's insurance) will pay if you get compromised. This is not limited to small companies, but is instead mostly how large banks, credit firms, and large enterprises work. The entire damn show is for liability, and half of these "Security professionals" can't do anything other than read the message coming out of their tool. They are utterly incompetent when it comes to applying logic to figure out whether a specific use of a "risky" tool/language feature a genuine problem, vs a god damned fucking regex match from their tool on something completely unrelated. I went in as a naive young dev, I came out with ZERO respect for this industry, and a healthy dose of skepticism around anything these folks are saying. Security researchers? Generally fine (although there's a new breed of them that simply submits inane non-vulnerabilities over and over to attempt to get bug bounties from large companies) Security advice from open source devs? Generally top notch, listen to it. Security advice from that contractor your company paid? Expect to have 95% false positives, and they'll miss the 2 places you know actually have issues. But it's ok because you'll check all the boxes on their sheet and legal is happy. --- I'm waiting for insurance companies to wise up and stop covering companies who are breached. Prices are already shooting way up, since it turns out security theater does a very bad job stopping real threats, and that's what this industry is right now. Might was well be the TSA of software, gonna frisk you real good, and then flunk every fucking test.
- kriro 4y agoThank you, that is a very interesting take. I've always played with the idea of pivoting into the security industry because playing hack the box and the like is something I do in my free time and enjoy. Maybe I'll just do some bug bounties or something for fun instead. I think it's a bit different in Europe though at least anecdotally there seems to be more code audit and the like (more product security) instead of pen testing (more network/infrastructure security).
- spc476 4y agoBack in the late 90s, I was working at a small web hosting company (take note). One day, a 500+ page report of a recent PCI compliancy check landed on my desk. It was nothing but "OMFG! DOMAIN1 RESPONDED TO PING! YOU WILL BE PWOWNED! OMFG! DOMAIN1 HAS DNS RECORDS! YOU WILL BE PWONED! OMFG! DOMAIN1 HAS A WEB SERVER RUNNING! YOU WILL BE PWONED!". Over and over again. For every domain we have. Complete and utter garbage report. Better---just summarize the IPs scanned and report back which services were found running on said IPs. Then in an appendix, list why each service is (or might be) problematic. "Ping? Attackers might be able to figure out your network topology and that is bad because blah blah blah blah." But 500 pages of this automatic breathless garbage? Utter trash.
- plmpsu 4y agoConfigure your build tool to exclude the transitive dependency.
- thayne 4y agoI thought Log4j 1.1 was vulnerable, just to a lesser extent, and there wasn't a lot of information on it because 1.x is end of life
- bragr 4y agoLog4j 1.x is vulnerable, but to different things and not as badly.
- cesarb 4y ago> Log4j 1.x is vulnerable, but to different things And the most common use of log4j 1.x (logging to the console or to a file, with a simple configuration) uses none of the vulnerable parts.
- Natsu 4y agoEOL libraries are themselves vulnerabilities. It's not really good to even have this reachable on the classpath, in cases like the grandparent, you should look at removing the class from inside the JAR (or removing the JAR itself) if possible to ensure that the vulnerable code cannot get called under any circumstances.
- hyperman1 4y agoSee https://logging.apache.org/log4j/1.2/ https://logging.apache.org/log4j/1.2/ . Plenty of vulnerabilities. Just different ones
- Danidada 4y agoSomeone said "hey log4j 1.2 wasn't that bad" after the log4shell vuln and came up with the idea of reload4j which is just log4j 1.2 but with its known vulnerabilities, bugs and performance issues fixed. Complete feature freeze besides that. Considering that a log tool shouldn't have that many features and that log4j 1.2 was being used by a lot of companies (just see maven central stats) there is no need to add more features to reload4j, which I find kind of cool
- Nursie 4y agoBeen in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this is a vulnerability" "Do you understand this isn't part of the thing you're concerned about" "I am only concerned about shipping a product without vulnerabilities" "Well this isn't one, because it's a false positive, here's the CVE, here's what it applies to, here's the link to this package on maven central, it's not part of netty, back off" "But it's insecure, the tool says so" We went round and round for about 3 hours until I told the guy (an infosec 'pro') to leave me alone until he'd figured out how to do his job... he came back to me the next morning for another 3 hour slack argument along the exact same lines at which point I told him to leave me alone permanently and communicate through management if he had anything to say. I'm not massively impressed with infosec as a profession after a few similar encounters.
- d4mi3n 4y agoSecurity engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so. That said, I've seen the inverse problem: engineering staff that either don't understand how their dependencies are managed (not unreasonable for less experienced teams, NPM for example has a tendency for huge dependency trees and this can be a hard problem to manage for a big project) or whom are for a plethora of reasons incentivized to push back on requests for work they don't perceive as important. The middle ground here requires trust between both parties (InfoSec and Engineering org functions). Sadly, not all security programs are well managed and not all engineering teams have mature practices. If you have time or capacity, you can get a lot of leverage here by educating your security analysts on how your technology works (e.g. how the build system functions, how your languages of choice share and bundle code). Trust can be built between engineering and infosec teams by educating where possible. Done well such practices can up-level both sides of these discussions and save you stress in the long run.
- dspillett 4y ago> We had to spend a week back and forth with We've had similar out of a pen test that noted we were using nginx for some of our internal dashboards/tools and the version running (1.18 IIRC) was officially EOLed upstream so a high security issue. They initially wouldn't listen to the argument that we use stable/LTS Debian/Ubuntu release only, and those hold functionality stable (which is one of the key reasons to use them) and backport security updates where relevant. We pointed them at the package changelogs and they were convinced that there was still a vulnerability not patched but for some reason didn't want to tell us which, when we finally got that out of them it turned out to be one that was introduced in a later version so was never relevant in the first place for the one the stable repositories included. You would think a penetration testing company would have a clue about something so common…
- raesene9 4y agoIf it was an unauthenticated test, I'm not surprised. Raising findings off of banners is notoriously error prone, but pentesters often err on the side of raising things that are dubious rather than leaving them out. If it was a credentialed review, you'd have hoped their scanners would plugin to the appropriate debian/ubuntu security database and give you a less false positive prone set of findings (although with debian based stuff you still have the problem of scanners reporting the "unfixed" set sometimes)
- allochthon 4y agoTypically there's a way to suppress specific warnings in systems like these. In your company's situation, I would look at moving away from a scanning system if it didn't allow overrides like this.
- alfalfasprout 4y agoSo far this is the best approach I've found. The scanning tools rarely include that ability but if you build tooling around them you can maintain exclusion lists, for particular vulnerabilities, library/version pairs, etc. Unfortunately it does mean there's no getting around having someone manually deal with false positives.
- MilStdJunkie 4y agoYup, that's us. Same deal exactly. They already stoppered our entire team for eight months with a tools audit a few years ago, so I hung that dead fish above the conversation when this came up. "Uh, ok, don't want to do that, sooooo . . recommendations?" "Let's isolate it, toss stuff at it, and see if it lights up" "Sounds good to me. FIXED"
- sporkland 4y agoLog4j-api isn't vulnerable and the slf4j-log4j bridge depends on it. Log4j-core is. All of our internal security scanners failed to make this distinction so we had to post filter ourselves.
- dotnet00 4y agoI recall that NVIDIA had to release an update to their CUDA Toolkit around when the Log4j vulnerability was announced. They weren't using it but the file was being distributed, presumably for a similar reason. I'm guessing they might've had similar complaints coming in.
- cmckn 4y ago> we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency If you use Maven, just exclude that transitive dependency from the explicit dependency. But yeah, dumb warnings are dumb. <dependency> <groupId>com.example</groupId> <artifactId>my-dependency</artifactId> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency>
- EdwardDiego 4y agoBingo.
- bostik 4y agoWe (read: I) face this on an ongoing basis. We can't upgrade the entire components that bundled up log4j for various reasons, starting from licensing rules. So we made the decision to strip out the entire JndiLookup class from every project that uses java. Clients do various scans, and rely on dumb version string matching and/or banner grabbing. We have to routinely point them to our VERY detailed and explicit log4j response document, and carefully explain to them that their scanners are relying on insufficient detection methods. Security teams are quite content with us giving them detailed explanation about the false positive. And then they forget or deliberately choose to ignore the lesson and the next time they run their scans, get the same false positives again. Even supposedly state-of-the-art security tools, to this day, refuse to actually verify their detections.