23 ms·

Log4j: The pain just keeps going

259 points by CrankyBear 4y ago

neals 4y ago
As a small time hobby / small business server administrator (enthusiast) , what's the best practice here?
- AndrewDucker 4y ago
  Make sure you have the time to patch/update your applications when security issues are raised.
  - LaGrange 4y ago
    Whoa whoa whoa, slow down, that would mean somebody in the organization would have to spend _valuable senior developer time_ on reading things. Yes, that's jaded, but that's what I'd expect. So far I've yet to feel like a job _encouraged me_ to pay attention to security updates, accessibility standards, or anything like that. And it's quite hard to keep myself motivated to do that, when it's pretty clear most software products are disposable anyway.
    - AndrewDucker 4y ago
      You can automate some of this. BlackDuck scans your dependencies (if you're using Maven or similar) to check what versions of dependencies you're using and if there are known CVEs for them. And there are plenty of competitors. But you still need to patch and test when you find out there's an issue.
    - thechao 4y ago
      I hired a guy because he loves to be (technically) correct about everything, and he does so by reading (and REREADING) specs. Gives my team a superpower.
- shagie 4y ago
  Put a dependency check in your builds. Raise build errors when something is bad. Have it build on a regular basis (could be every month)... and then fix things. https://owasp.org/www-project-dependency-check/ https://owasp.org/www-project-dependency-check/ And some examples - https://jeremylong.github.io/DependencyCheck/dependency-check-maven/ https://jeremylong.github.io/DependencyCheck/dependency-chec...
- kevinmgranger 4y ago
  As an alternative / addition to the other suggestions, maintain an internal artifact store that syncs with upstream stores, but enforce that vulnerable versions aren't present. Block access to the upstream artifact stores so builds have to use the internal one.
- mikeweiss 4y ago
  If want an extremely strong mitigating control that you can design into your architecture..... allow internet egress in your production environment only to a list of domains/ips in an allowlist. This or control egress with an explicit proxy. If vulnerable systems have no path to the internet at large then it's extremely difficult for attackers to know if a system is vulnerable to log4j and even harder to actually use it to exfiltrate data.
- brabel 4y ago
  Every case is different... but for the log4shell case in particular: * newer JVM versions protected against the main attack (JNDI), so if you were on any JDK version released in the last couple of years, you wouldn't be vulnerable. * Java serialization filters [1], which are highly recommended to be setup in any Java application, would also have prevented the attacks. * not including large libraries with lots of unused features could have prevented this. In this case, log4j has support for a dozen or so things that should've been opt-in (and I think they are now) but were enabled by default in every installation. In summary: actually paying attention to possible attacks and ensuring you're defended against them, and reducing the surface area for attacks makes your application much safer. [1] https://docs.oracle.com/javase/10/core/serialization-filtering1.htm#JSCOR-GUID-3ECB288D-E5BD-4412-892F-E9BB11D4C98A https://docs.oracle.com/javase/10/core/serialization-filteri...
WesolyKubeczek 4y ago
Sometimes people ask me why I'm so skeptical about software delivery techniques involving bundling and static compilation. This is a thing that people sometimes ask.
- 0xbadcafebee 4y ago
  It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "replacing a jar in a zip file".
  - skybrian 4y ago
    The situation for Go seems to have gotten a bit better with Go 1.18. Go binaries now include the versions of all the modules in them, and this can be listed with the "go version" command. That won't help you rebuild the binary, but at least you can find them.
    - jewel 4y ago
      That's fantastic. Since Go has gone all-in on static compiling, it'd be awesome if they made it possible to update a dependency without recompiling (or even having the source code for) the rest of the program. In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.
      - er4hn 4y ago
        Wouldn't that require the ABI, or at least the API, to be the same across versions of the affected library? That feels like it may be possible with enough tooling, but still very hard to enforce.
        4y ago
mrbonner 4y ago
At this point I’m asking if it is ok to go nuclear: just slab aspectJ compiler as the last step and do comptime patch the log4j remote logging API. Have anyone thought of this? Btw who would use remote logging to begin with?
- paulmd 4y ago
  > Btw who would use remote logging to begin with? It's not about remote logging like "we log to somewhere else" but rather about including information from remote sources (specifically JDNI) in your log messages. If you are ingesting logs from a bunch of microservices, it would obviously be desirable to know which instance you're in, and just include that in the logger pattern. Or if you have multiple containers in a single tomcat instance, you might want to have the log message include which container it's coming from (especially for the console). Multi-container tomcat used to be VERY common in pet-style legacy java deployments - it's much slimmer on memory, although it does come with the caveat that GC pauses affect all containers. But that container/server information isn't built into the jar, it's runtime data, and the easiest source for getting that information is... connecting to the tomcat server via JNDI and pulling instance information. But oops, they populated the JDNI string values after the message was already built, meaning that any log message that included a JDNI handle would be executed automatically on its own. And JNDI is basically a whole inner-platform, including the ability to classload arbitrary code from remote sources, because "lol why not, it's java enterprise code, we're building the future here!", so basically any user-controlled log message could include a JDNI link to arbitrary code and it would be downloaded and run. (the JDNI being built after the rest of the message was populated was obviously the wrong place to do it for exactly this reason... that is really the core problem here, user-controlled JDNI is absolutely not acceptable given the capabilities, but, it's not like I've never written bad code... there but by the grace of god, go I. Although again, that one is pretty obviously a problem just conceptually.) Incidentally though "remote logging" really isn't inherently insane as a concept, it's conceptually reasonable that some kinds of important messages might want to be logged into Kafka or JDBC, and 15 years ago there wasn't the overwhelming consensus on "write to disk first, ingest it later" being the right way to get it there. If you want to be really really sure that certain super-important messages are committed, and you don't mind the latency, why not just poke it directly? Then you never have to worry about logs getting dropped on disk/etc and not making it there. And by running it through log4j you don't have to do anything special at an application level, you just define certain packages and message levels as being important and you don't have to re-invent the wheel on the message-building framework/etc. And sure log4j ended up having vulnerabilities but your personal implementation may have vulnerabilities too - how sure are you about your string handling and escaping, could a message spoof what looks like a valid log entry? that's a common one. It just turns out that JDNI was a vastly bigger attack surface than initially realized...
upupandup 4y ago
It's probable that there are more log4j type of exploits thats just waiting to be discovered. It's also becoming hard to tell which are private enterprises vs state sponsored cyber attack. It just seems like there's no solution. If Java is this bad imagine how much more attack surface a npm heavy project. Not only that the seeming hardware knowledge of state sponsored attackers seem sophisticated enough to gain access at the vendor level, which forces us to question even the most trusted methodologies and dependency management.
- hedora 4y ago
  We could learn to write auditably correct software. Reflections on Trusting Trust explains why writing correct software is inadequate: https://dl.acm.org/doi/10.1145/358198.358210 https://dl.acm.org/doi/10.1145/358198.358210
- commandlinefan 4y ago
  > seems like there's no solution There is - don't depend on anybody else's software. It just means that software development will take longer and cost more.
  - Beltalowda 4y ago
    This assumes that your own software is more secure.
    - waynesonfire 4y ago
      for what it's worth, the log4j vulnerability was due to some obscure feature that nobody uses. unlikely you'd even write such code.
  - upupandup 4y ago
    but the cost of rewriting everything that you are currently using through dozens of dependencies and third party libs you are going to go broke. Unless you have the budget like the DOD which has its own coding language I believe, you are ngmi Not to mention you would be introducing attack surfaces that wont have enough attention because its only being used by your team. At least with a widely used lib, vulnerabilities are caught, published (very important) and constantly raising vigilence. here you would have to perform your own threat mitigation and have bounty programs.
jacobyoder 4y ago
Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and forth with "no, we're not vulnerable". "But we see log4j is right there in the a file in the directory". "No, it's not vulnerable and it's not being used and we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency" "But we see log4j is right there in the a file in the directory". Also, there was no budget to do any sort of rewrite/refactor/etc anyway.
- hedora 4y ago
  Once you realize this sort of shallow, automated audit exists to grease the wheels of some security theater operation, this sort of back and forth makes sense. False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.
  - tekchip 4y ago
    So somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then. I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh. But by in large automation and the problems inherent are required.
    - hedora 4y ago
      You could set your build up to not auto-download questionable stuff from untrustworthy machines on the Internet.
    - paulmd 4y ago
      The problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept. In one of my past jobs I was an early-mover on doing a lot of our ops on Linux and the audit tools had no concept of the backport security model whatsoever. If you were running on some kind of LTS distro rather than a current distro, it would see "gosh you're 3 minor versions behind, you have a ton of unpatched vulnerabilities!", but in actuality you didn't, because the fixes got backported into security releases (the "31" in something like "3.1.22~ubuntu31" for example). But the tool was dumb and it just had a table that said "anything under 3.4.14 is vulnerable" even when it wasn't necessarily. OP's "log4j 1.x is not vulnerable to this the first place" is a similar thing where either someone forgot to put in a value for "min vulnerable version" or the tool doesn't understand the concept of min-version at all. To be fair that can be genuinely tricky in java, best-case you are reading manifest files to try and pick out a version, but some legacy stuff doesn't have manifests, especially very legacy fat-jar stuff. Yes, that stuff didn't go away, it's still out there in places! And to be clear this was a long-running issue... it wasn't a "oh our tool doesn't understand LTS, I get it" and they left me alone... this was every month they'd be back at me for a new list of utility vulnerabilities even though I repeatedly explained to them that I had set up apt-get to auto-upgrade every night and reboot, so we were running whatever the latest security backports were available, and that just like the last 10 times this was even more false-positives. And again, this isn't just "we have to run it by you no matter what" and they leave you alone either. What security really wanted was for me to run windows and manually install the updates and get back in line with everyone else, even though that would have been a less secure outcome than my locked-down linux boxes. But this wasn't my day-job at the company so to speak, it was helping out another project with some ops and it needed to be as low-effort as possible (and to be fair their requirements weren't steep, auto-patch-and-reboot was fine by them and never caused any breakage during my tenure there). The root problem is that these sorts of tools aren't a substitute for an actual security culture, they're often a symptom of a compliance requirement and the whole thing turns into a box-checking exercise. Someone is on their ass about this because of contractual requirements or PCI requirements, and they bought the cheapest thing off the shelf that would check the box, and they are implicitly showing you here that they don't even understand how to interpret the output of that.
bgro 4y ago
When are they going to release Log5j? Or even Log4k to address these issues?
- hedora 4y ago
  The log4j author already did this about a decade ago. No one could understand log4j back then, so the replacement is much, much simpler, and API compatible for 99+% of use cases. Few projects seem to have bothered moving to it, probably becuase log4j is such a collosal pain to set up or modify.
  - bxparks 4y ago
    What is that replacement? java.util.logging?
    - jffry 4y ago
      I believe that replacement is Logback [1] + SLF4J, according to the intro section on Wikipedia's log4j article [2] [1] https://logback.qos.ch/reasonsToSwitch.html https://logback.qos.ch/reasonsToSwitch.html [2] https://en.wikipedia.org/wiki/Log4j https://en.wikipedia.org/wiki/Log4j
  - qw 4y ago
    I think Spring Boot defaults to Logback these days.
  - hinkley 4y ago
    > No one could understand log4j back then If you're scoffing at this, I want you to do an experiment. Imagine yourself at 26, when you just knew enough to be dangerous. Or as the person you're currently mentoring/would like to mentor. Now go into your application code, set a breakpoint on one of the moderately complex `logger.info()` calls, run your application and step into log4j. All the way down, until it writes to disk. If you don't say "what the fuck" out loud at least once, you're a soulless monster and should go live in a cave. If your application has a pretty good architecture, you may well discover that the code leading up to the `logger.info()` call is substantially less complex than the code below it. Just to write some text to the end of a single-writer file. It chaps my ass that log4js, which claims specifically not to be a javascript port of log4j, has the same pattern of layers and weird indirection (some of which are avoided by monkey patching, which is in many respects even worse from a legibility standpoint, and absolutely from a discoverability standpoint). Much of this is chasing after not implementing the same 1 line function 6 times, which is made weirder by the fact that you have to write individual unit tests for each of the log levels, so you've already written 95% of the duplicated code. Logging frameworks are a nearly canonical example of DRY as an antipattern.
hyperman1 4y ago
I never understood where log4j 2 was supposed to fit in the ecosystem. * We had log4j 1, which was good enough for most use cases and everywhere. * We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design. * We had java.util.logging, which while braindead was builtin to Java and hence available everywhere. *We had commons logging, which was never supposed to escape from tomcat, but got overused everywhere anyway. All of this was more logging frameworks than was healthy for 1 language, and there are some more smaller ones. Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why?
- shagie 4y ago
  > Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why? slf4j itself isn't a logging framework. It's a facade to logging frameworks. Simple Logging Facade for Java ( https://www.slf4j.org https://www.slf4j.org ) It needs a logging framework behind it - log4j, log4j2, logback, commons, JUL. The question is "why do log4j2?" Logback went from the log4j1.x path ( https://logback.qos.ch https://logback.qos.ch ) Log4j2 has a lot of features that weren't present when the project started ( https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 ). There is a licensing difference between Logback (LGPL) and Log4jx (Apache Commons).
  - Aperocky 4y ago
    > Log4j2 has a lot of features Apache projects in general have this feature creep seeping into every single one of their products that I have used. It's just hard to trust apache projects now.
  - kelnos 4y ago
    > It needs a logging framework behind it - log4j Minor nit: log4j doesn't implement the slf4j interface, though. If you do want log4j to do your logging, but use slf4j in your application, you need to include a "bridge" library that adapts slf4j to log4j. And, conversely, if you have dependencies that use log4j, but want things to end up going through your slf4j-supporting backend (say, logback), you need to exclude the "real" log4j jar and include a different library that implements log4j's interface and forwards calls to slf4j.
BrainVirus 4y ago
If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down. "Use a build tool to do dependency checks!" And then what? "Only update libraries if they have vulnerabilities." So, should this be a manual process where I have to dig through obscure warnings every time I build something? "No, you can automate it!" Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build process reliant on yet another online service. "But it's not a hard dependency, you can still build if the 3d party service is off." So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this? Etc. People with clever advice don't seem to think it through. We're fighting fragile complexity of too many tools ducktaped together by ducktaping more tools to the whole setup. Again and again and again.
- AndyMcConachie 4y ago
  The ease with which software remains mutable after deployment is both a blessing and a curse. This is particular and intrinsic to software and other digital artifacts. Things based in matter do not benefit from this blessing or suffer from this curse.
- wyager 4y ago
  > The current software engineering paradigm has no meaningful answer to this The answer is to stop using garbage software written in garbage languages. Of course the current environment is too high-time-preference for this to be economical in many industries. We have lightweight-to-heavyweight formal systems which can almost completely or literally completely eliminate issues of the type seen with log4j, but due to the marginally higher up-front cost of using such systems, only firms with some combination of high stakes, low time preference, and foresight end up using them.
vanattab 4y ago
Does anyone know if log4net the .net version of log4j is effected?
- xtracto 4y ago
  Hey, English is not my first language. Is "effected" here the right word to use or would "affected" be the correct term? I see English speaking people use these two interchangeably so much online that I am starting to doubt what I learned in my English courses.
  - eatsyourtacos 4y ago
    "Affected" is the correct term here. People do use those wrong all the time. Honestly it's not always that they don't know which one (well ok it probably is, but doesn't have to be), but I find typing comments online my brain interchanges some things like that even though I know which one is correct.
  - hello_erich 4y ago
    “affected” should have been used. It is a common mistake to confuse the two.
  - aoetalks 4y ago
    It should be “affected”
  - macintux 4y ago
    "Affected" is correct. Fortunately most of the time you can use the wrong word and still be understood. Even with English as my first language, and as a life-long pedant, I still have to stop and think each time I use one or the other.
  - teddyh 4y ago
    Native English speakers mostly learned English by sound alone, so many such speakers regularly mix up words which sound very similar, like it’s/its, effected/affected, site/sight, etc. Non-native speakers have instead frequently learned by mostly reading, and may therefore instead be lacking in correct pronunciation, but have no problem in distinguishing “it’s” and “its”, for example.
lcfcjs 4y ago
Foobar8568 4y ago
Long live to SaaS products, no longer our problems hahaha... Going to cry in a corner.
croes 4y ago
If I understand this correctly the log4j problem wasn't a bug but Java feature at the wrong place
alexbakker 4y ago
I'm seeing this as well. While the amount of traffic has certainly decreased compared to the first couple of days after the CVE was announced, https://log4shell.tools https://log4shell.tools is still being used by people every day.
ChrisMarshallNY 4y ago
> Welcome to how we use open source code. That’s a big reason Software Bills of Materials (SBOM) have become essential for securing the software supply chain. I tend to do this, but it's pretty damn easy for me. I have a "2-level-dependency" rule that I generally go by, these days. i.e., Each dependency can have only one dependency, and that dependency can't have any others. The main way that I deal with it, is to write all my own dependencies, and I don't count toolsets, standard libraries, or platform frameworks (for better, or for ill). I really, really want dependencies to work. They are a great idea, and I have been promoting modular architecture for decades. It's just that we are in a "wild west" situation, where we can't trust dependencies. I'm tired of hearing the phrase "It's a solved problem," with a reference to a dependency. I don't have the answer. I just have my own rule, which elicits scorn from most modern programmers (I'm painted as a luddite, refusing to "get with the times," or as a "tinfoil-hatter"). WFM. YMMV.
- jeroenhd 4y ago
  I like the "2-level-dependency" rule, but what do you do when one of your dependencies gains more dependencies? Do you stop updating and start migrating to something else, or maybe maintain a fork of your own? It's a very nice concept, but I kind of have my doubts about maintainability.
  - ChrisMarshallNY 4y ago
    I use my own dependencies. I've written many of them. Some of them, go "deeper," but only in dependencies that I authored. Here's the dependency manifest from the app I'm working on now (From my localization file): // MARK: - // MARK: - DO NOT TRANSLATE BELOW THIS LINE - // MARK: - "SLUG-VERSION-BMLT" = "BMLTiOSLib: 1.4.2"; "SLUG-VERSION-KEYCHAINSWIFT" = "KeychainSwift: 20.0.0"; "SLUG-VERSION-LGVCLEANTIME" = "LGV_Cleantime: 1.3.5"; "SLUG-VERSION-UICLEANTIME" = "LGV_UICleantime: 1.1.1"; "SLUG-VERSION-AUTOFILL" = "RVS_AutofillTextField: 1.3.0"; "SLUG-VERSION-GCD" = "RVS_BasicGCDTimer: 1.5.0"; "SLUG-VERSION-CHECKBOX" = "RVS_Checkbox: 1.2.0"; "SLUG-VERSION-OBSERVER" = "RVS_GeneralObserver: 1.1.0"; "SLUG-VERSION-GST" = "RVS_Generic_Swift_Toolbox: 1.10.1"; "SLUG-VERSION-MB" = "RVS_MaskButton: 1.2.0"; "SLUG-VERSION-PP" = "RVS_Persistent_Prefs: 1.3.1"; "SLUG-VERSION-UKT" = "RVS_UIKit_Toolbox: 1.1.5"; "SLUG-VERSION-WHITEDRAGON" = "White Dragon SDK: 3.2.2"; The only one of those that I didn't write, was KeychainSwift[0]. It makes dealing with the Keychain easy, and is a very simple dependency. If it went off the rails, I'd write something like it, myself. All the others, are in my own repos, as top-shelf-quality open-source modules. [0] https://github.com/evgenyneu/keychain-swift https://github.com/evgenyneu/keychain-swift
craigching 4y ago
> We are all in this together. The last line from the article and my favorite from Brazil which seems fitting?
aorth 4y ago
Surprised nobody mentioned yet: reload4j https://reload4j.qos.ch/ https://reload4j.qos.ch/ It's a fork of log4j 1.2.17 by the original author that fixes a boat load of issues with 1.2.x and is a drop in replacement for log4j. Very useful if you can't upgrade to log4x 2.x.
kazinator 4y ago
That's what you get for not minding that a logging library needs over 300,000 lines of code.
danhab99 4y ago
I feel like if we as the open source community would require commit signing we would be in a safer position. Crypto signing doesn't block malware from being introduced, but it would make it harder to sneek under our noses. Currently in open source, you really don't know where your code is coming from and who worked on it. Git's "commit author" fields don't require any proof of identity, that's what GPG is for.

259 points by CrankyBear 4y ago

neals 4y ago

As a small time hobby / small business server administrator (enthusiast) , what's the best practice here?

AndrewDucker 4y ago
Make sure you have the time to patch/update your applications when security issues are raised.
- LaGrange 4y ago
  Whoa whoa whoa, slow down, that would mean somebody in the organization would have to spend _valuable senior developer time_ on reading things. Yes, that's jaded, but that's what I'd expect. So far I've yet to feel like a job _encouraged me_ to pay attention to security updates, accessibility standards, or anything like that. And it's quite hard to keep myself motivated to do that, when it's pretty clear most software products are disposable anyway.
  - AndrewDucker 4y ago
    You can automate some of this. BlackDuck scans your dependencies (if you're using Maven or similar) to check what versions of dependencies you're using and if there are known CVEs for them. And there are plenty of competitors. But you still need to patch and test when you find out there's an issue.
  - thechao 4y ago
    I hired a guy because he loves to be (technically) correct about everything, and he does so by reading (and REREADING) specs. Gives my team a superpower.
shagie 4y ago
Put a dependency check in your builds. Raise build errors when something is bad. Have it build on a regular basis (could be every month)... and then fix things. https://owasp.org/www-project-dependency-check/ https://owasp.org/www-project-dependency-check/ And some examples - https://jeremylong.github.io/DependencyCheck/dependency-check-maven/ https://jeremylong.github.io/DependencyCheck/dependency-chec...
kevinmgranger 4y ago
As an alternative / addition to the other suggestions, maintain an internal artifact store that syncs with upstream stores, but enforce that vulnerable versions aren't present. Block access to the upstream artifact stores so builds have to use the internal one.
mikeweiss 4y ago
If want an extremely strong mitigating control that you can design into your architecture..... allow internet egress in your production environment only to a list of domains/ips in an allowlist. This or control egress with an explicit proxy. If vulnerable systems have no path to the internet at large then it's extremely difficult for attackers to know if a system is vulnerable to log4j and even harder to actually use it to exfiltrate data.
brabel 4y ago
Every case is different... but for the log4shell case in particular: * newer JVM versions protected against the main attack (JNDI), so if you were on any JDK version released in the last couple of years, you wouldn't be vulnerable. * Java serialization filters [1], which are highly recommended to be setup in any Java application, would also have prevented the attacks. * not including large libraries with lots of unused features could have prevented this. In this case, log4j has support for a dozen or so things that should've been opt-in (and I think they are now) but were enabled by default in every installation. In summary: actually paying attention to possible attacks and ensuring you're defended against them, and reducing the surface area for attacks makes your application much safer. [1] https://docs.oracle.com/javase/10/core/serialization-filtering1.htm#JSCOR-GUID-3ECB288D-E5BD-4412-892F-E9BB11D4C98A https://docs.oracle.com/javase/10/core/serialization-filteri...

WesolyKubeczek 4y ago

Sometimes people ask me why I'm so skeptical about software delivery techniques involving bundling and static compilation. This is a thing that people sometimes ask.

0xbadcafebee 4y ago
It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "replacing a jar in a zip file".
- skybrian 4y ago
  The situation for Go seems to have gotten a bit better with Go 1.18. Go binaries now include the versions of all the modules in them, and this can be listed with the "go version" command. That won't help you rebuild the binary, but at least you can find them.
  - jewel 4y ago
    That's fantastic. Since Go has gone all-in on static compiling, it'd be awesome if they made it possible to update a dependency without recompiling (or even having the source code for) the rest of the program. In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.
    - er4hn 4y ago
      Wouldn't that require the ABI, or at least the API, to be the same across versions of the affected library? That feels like it may be possible with enough tooling, but still very hard to enforce.
      - 4y ago

mrbonner 4y ago

At this point I’m asking if it is ok to go nuclear: just slab aspectJ compiler as the last step and do comptime patch the log4j remote logging API. Have anyone thought of this? Btw who would use remote logging to begin with?

paulmd 4y ago
> Btw who would use remote logging to begin with? It's not about remote logging like "we log to somewhere else" but rather about including information from remote sources (specifically JDNI) in your log messages. If you are ingesting logs from a bunch of microservices, it would obviously be desirable to know which instance you're in, and just include that in the logger pattern. Or if you have multiple containers in a single tomcat instance, you might want to have the log message include which container it's coming from (especially for the console). Multi-container tomcat used to be VERY common in pet-style legacy java deployments - it's much slimmer on memory, although it does come with the caveat that GC pauses affect all containers. But that container/server information isn't built into the jar, it's runtime data, and the easiest source for getting that information is... connecting to the tomcat server via JNDI and pulling instance information. But oops, they populated the JDNI string values after the message was already built, meaning that any log message that included a JDNI handle would be executed automatically on its own. And JNDI is basically a whole inner-platform, including the ability to classload arbitrary code from remote sources, because "lol why not, it's java enterprise code, we're building the future here!", so basically any user-controlled log message could include a JDNI link to arbitrary code and it would be downloaded and run. (the JDNI being built after the rest of the message was populated was obviously the wrong place to do it for exactly this reason... that is really the core problem here, user-controlled JDNI is absolutely not acceptable given the capabilities, but, it's not like I've never written bad code... there but by the grace of god, go I. Although again, that one is pretty obviously a problem just conceptually.) Incidentally though "remote logging" really isn't inherently insane as a concept, it's conceptually reasonable that some kinds of important messages might want to be logged into Kafka or JDBC, and 15 years ago there wasn't the overwhelming consensus on "write to disk first, ingest it later" being the right way to get it there. If you want to be really really sure that certain super-important messages are committed, and you don't mind the latency, why not just poke it directly? Then you never have to worry about logs getting dropped on disk/etc and not making it there. And by running it through log4j you don't have to do anything special at an application level, you just define certain packages and message levels as being important and you don't have to re-invent the wheel on the message-building framework/etc. And sure log4j ended up having vulnerabilities but your personal implementation may have vulnerabilities too - how sure are you about your string handling and escaping, could a message spoof what looks like a valid log entry? that's a common one. It just turns out that JDNI was a vastly bigger attack surface than initially realized...

upupandup 4y ago

It's probable that there are more log4j type of exploits thats just waiting to be discovered. It's also becoming hard to tell which are private enterprises vs state sponsored cyber attack. It just seems like there's no solution. If Java is this bad imagine how much more attack surface a npm heavy project. Not only that the seeming hardware knowledge of state sponsored attackers seem sophisticated enough to gain access at the vendor level, which forces us to question even the most trusted methodologies and dependency management.

hedora 4y ago
We could learn to write auditably correct software. Reflections on Trusting Trust explains why writing correct software is inadequate: https://dl.acm.org/doi/10.1145/358198.358210 https://dl.acm.org/doi/10.1145/358198.358210
commandlinefan 4y ago
> seems like there's no solution There is - don't depend on anybody else's software. It just means that software development will take longer and cost more.
- Beltalowda 4y ago
  This assumes that your own software is more secure.
  - waynesonfire 4y ago
    for what it's worth, the log4j vulnerability was due to some obscure feature that nobody uses. unlikely you'd even write such code.
- upupandup 4y ago
  but the cost of rewriting everything that you are currently using through dozens of dependencies and third party libs you are going to go broke. Unless you have the budget like the DOD which has its own coding language I believe, you are ngmi Not to mention you would be introducing attack surfaces that wont have enough attention because its only being used by your team. At least with a widely used lib, vulnerabilities are caught, published (very important) and constantly raising vigilence. here you would have to perform your own threat mitigation and have bounty programs.

jacobyoder 4y ago

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version. We had to spend a week back and forth with "no, we're not vulnerable". "But we see log4j is right there in the a file in the directory". "No, it's not vulnerable and it's not being used and we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency" "But we see log4j is right there in the a file in the directory". Also, there was no budget to do any sort of rewrite/refactor/etc anyway.

hedora 4y ago
Once you realize this sort of shallow, automated audit exists to grease the wheels of some security theater operation, this sort of back and forth makes sense. False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.
- tekchip 4y ago
  So somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then. I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh. But by in large automation and the problems inherent are required.
  - hedora 4y ago
    You could set your build up to not auto-download questionable stuff from untrustworthy machines on the Internet.
  - paulmd 4y ago
    The problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept. In one of my past jobs I was an early-mover on doing a lot of our ops on Linux and the audit tools had no concept of the backport security model whatsoever. If you were running on some kind of LTS distro rather than a current distro, it would see "gosh you're 3 minor versions behind, you have a ton of unpatched vulnerabilities!", but in actuality you didn't, because the fixes got backported into security releases (the "31" in something like "3.1.22~ubuntu31" for example). But the tool was dumb and it just had a table that said "anything under 3.4.14 is vulnerable" even when it wasn't necessarily. OP's "log4j 1.x is not vulnerable to this the first place" is a similar thing where either someone forgot to put in a value for "min vulnerable version" or the tool doesn't understand the concept of min-version at all. To be fair that can be genuinely tricky in java, best-case you are reading manifest files to try and pick out a version, but some legacy stuff doesn't have manifests, especially very legacy fat-jar stuff. Yes, that stuff didn't go away, it's still out there in places! And to be clear this was a long-running issue... it wasn't a "oh our tool doesn't understand LTS, I get it" and they left me alone... this was every month they'd be back at me for a new list of utility vulnerabilities even though I repeatedly explained to them that I had set up apt-get to auto-upgrade every night and reboot, so we were running whatever the latest security backports were available, and that just like the last 10 times this was even more false-positives. And again, this isn't just "we have to run it by you no matter what" and they leave you alone either. What security really wanted was for me to run windows and manually install the updates and get back in line with everyone else, even though that would have been a less secure outcome than my locked-down linux boxes. But this wasn't my day-job at the company so to speak, it was helping out another project with some ops and it needed to be as low-effort as possible (and to be fair their requirements weren't steep, auto-patch-and-reboot was fine by them and never caused any breakage during my tenure there). The root problem is that these sorts of tools aren't a substitute for an actual security culture, they're often a symptom of a compliance requirement and the whole thing turns into a box-checking exercise. Someone is on their ass about this because of contractual requirements or PCI requirements, and they bought the cheapest thing off the shelf that would check the box, and they are implicitly showing you here that they don't even understand how to interpret the output of that.

bgro 4y ago

When are they going to release Log5j? Or even Log4k to address these issues?

hedora 4y ago
The log4j author already did this about a decade ago. No one could understand log4j back then, so the replacement is much, much simpler, and API compatible for 99+% of use cases. Few projects seem to have bothered moving to it, probably becuase log4j is such a collosal pain to set up or modify.
- bxparks 4y ago
  What is that replacement? java.util.logging?
  - jffry 4y ago
    I believe that replacement is Logback [1] + SLF4J, according to the intro section on Wikipedia's log4j article [2] [1] https://logback.qos.ch/reasonsToSwitch.html https://logback.qos.ch/reasonsToSwitch.html [2] https://en.wikipedia.org/wiki/Log4j https://en.wikipedia.org/wiki/Log4j
- qw 4y ago
  I think Spring Boot defaults to Logback these days.
- hinkley 4y ago
  > No one could understand log4j back then If you're scoffing at this, I want you to do an experiment. Imagine yourself at 26, when you just knew enough to be dangerous. Or as the person you're currently mentoring/would like to mentor. Now go into your application code, set a breakpoint on one of the moderately complex `logger.info()` calls, run your application and step into log4j. All the way down, until it writes to disk. If you don't say "what the fuck" out loud at least once, you're a soulless monster and should go live in a cave. If your application has a pretty good architecture, you may well discover that the code leading up to the `logger.info()` call is substantially less complex than the code below it. Just to write some text to the end of a single-writer file. It chaps my ass that log4js, which claims specifically not to be a javascript port of log4j, has the same pattern of layers and weird indirection (some of which are avoided by monkey patching, which is in many respects even worse from a legibility standpoint, and absolutely from a discoverability standpoint). Much of this is chasing after not implementing the same 1 line function 6 times, which is made weirder by the fact that you have to write individual unit tests for each of the log levels, so you've already written 95% of the duplicated code. Logging frameworks are a nearly canonical example of DRY as an antipattern.

hyperman1 4y ago

I never understood where log4j 2 was supposed to fit in the ecosystem. * We had log4j 1, which was good enough for most use cases and everywhere. * We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design. * We had java.util.logging, which while braindead was builtin to Java and hence available everywhere. *We had commons logging, which was never supposed to escape from tomcat, but got overused everywhere anyway. All of this was more logging frameworks than was healthy for 1 language, and there are some more smaller ones. Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why?

shagie 4y ago
> Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why? slf4j itself isn't a logging framework. It's a facade to logging frameworks. Simple Logging Facade for Java ( https://www.slf4j.org https://www.slf4j.org ) It needs a logging framework behind it - log4j, log4j2, logback, commons, JUL. The question is "why do log4j2?" Logback went from the log4j1.x path ( https://logback.qos.ch https://logback.qos.ch ) Log4j2 has a lot of features that weren't present when the project started ( https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 ). There is a licensing difference between Logback (LGPL) and Log4jx (Apache Commons).
- Aperocky 4y ago
  > Log4j2 has a lot of features Apache projects in general have this feature creep seeping into every single one of their products that I have used. It's just hard to trust apache projects now.
- kelnos 4y ago
  > It needs a logging framework behind it - log4j Minor nit: log4j doesn't implement the slf4j interface, though. If you do want log4j to do your logging, but use slf4j in your application, you need to include a "bridge" library that adapts slf4j to log4j. And, conversely, if you have dependencies that use log4j, but want things to end up going through your slf4j-supporting backend (say, logback), you need to exclude the "real" log4j jar and include a different library that implements log4j's interface and forwards calls to slf4j.

BrainVirus 4y ago

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware. If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues. The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down. "Use a build tool to do dependency checks!" And then what? "Only update libraries if they have vulnerabilities." So, should this be a manual process where I have to dig through obscure warnings every time I build something? "No, you can automate it!" Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build process reliant on yet another online service. "But it's not a hard dependency, you can still build if the 3d party service is off." So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this? Etc. People with clever advice don't seem to think it through. We're fighting fragile complexity of too many tools ducktaped together by ducktaping more tools to the whole setup. Again and again and again.

AndyMcConachie 4y ago
The ease with which software remains mutable after deployment is both a blessing and a curse. This is particular and intrinsic to software and other digital artifacts. Things based in matter do not benefit from this blessing or suffer from this curse.
wyager 4y ago
> The current software engineering paradigm has no meaningful answer to this The answer is to stop using garbage software written in garbage languages. Of course the current environment is too high-time-preference for this to be economical in many industries. We have lightweight-to-heavyweight formal systems which can almost completely or literally completely eliminate issues of the type seen with log4j, but due to the marginally higher up-front cost of using such systems, only firms with some combination of high stakes, low time preference, and foresight end up using them.

vanattab 4y ago

Does anyone know if log4net the .net version of log4j is effected?

xtracto 4y ago
Hey, English is not my first language. Is "effected" here the right word to use or would "affected" be the correct term? I see English speaking people use these two interchangeably so much online that I am starting to doubt what I learned in my English courses.
- eatsyourtacos 4y ago
  "Affected" is the correct term here. People do use those wrong all the time. Honestly it's not always that they don't know which one (well ok it probably is, but doesn't have to be), but I find typing comments online my brain interchanges some things like that even though I know which one is correct.
- hello_erich 4y ago
  “affected” should have been used. It is a common mistake to confuse the two.
- aoetalks 4y ago
  It should be “affected”
- macintux 4y ago
  "Affected" is correct. Fortunately most of the time you can use the wrong word and still be understood. Even with English as my first language, and as a life-long pedant, I still have to stop and think each time I use one or the other.
- teddyh 4y ago
  Native English speakers mostly learned English by sound alone, so many such speakers regularly mix up words which sound very similar, like it’s/its, effected/affected, site/sight, etc. Non-native speakers have instead frequently learned by mostly reading, and may therefore instead be lacking in correct pronunciation, but have no problem in distinguishing “it’s” and “its”, for example.

lcfcjs 4y ago

Foobar8568 4y ago

Long live to SaaS products, no longer our problems hahaha... Going to cry in a corner.

croes 4y ago

If I understand this correctly the log4j problem wasn't a bug but Java feature at the wrong place

alexbakker 4y ago

I'm seeing this as well. While the amount of traffic has certainly decreased compared to the first couple of days after the CVE was announced, https://log4shell.tools https://log4shell.tools is still being used by people every day.

ChrisMarshallNY 4y ago

> Welcome to how we use open source code. That’s a big reason Software Bills of Materials (SBOM) have become essential for securing the software supply chain. I tend to do this, but it's pretty damn easy for me. I have a "2-level-dependency" rule that I generally go by, these days. i.e., Each dependency can have only one dependency, and that dependency can't have any others. The main way that I deal with it, is to write all my own dependencies, and I don't count toolsets, standard libraries, or platform frameworks (for better, or for ill). I really, really want dependencies to work. They are a great idea, and I have been promoting modular architecture for decades. It's just that we are in a "wild west" situation, where we can't trust dependencies. I'm tired of hearing the phrase "It's a solved problem," with a reference to a dependency. I don't have the answer. I just have my own rule, which elicits scorn from most modern programmers (I'm painted as a luddite, refusing to "get with the times," or as a "tinfoil-hatter"). WFM. YMMV.

jeroenhd 4y ago
I like the "2-level-dependency" rule, but what do you do when one of your dependencies gains more dependencies? Do you stop updating and start migrating to something else, or maybe maintain a fork of your own? It's a very nice concept, but I kind of have my doubts about maintainability.
- ChrisMarshallNY 4y ago
  I use my own dependencies. I've written many of them. Some of them, go "deeper," but only in dependencies that I authored. Here's the dependency manifest from the app I'm working on now (From my localization file): // MARK: - // MARK: - DO NOT TRANSLATE BELOW THIS LINE - // MARK: - "SLUG-VERSION-BMLT" = "BMLTiOSLib: 1.4.2"; "SLUG-VERSION-KEYCHAINSWIFT" = "KeychainSwift: 20.0.0"; "SLUG-VERSION-LGVCLEANTIME" = "LGV_Cleantime: 1.3.5"; "SLUG-VERSION-UICLEANTIME" = "LGV_UICleantime: 1.1.1"; "SLUG-VERSION-AUTOFILL" = "RVS_AutofillTextField: 1.3.0"; "SLUG-VERSION-GCD" = "RVS_BasicGCDTimer: 1.5.0"; "SLUG-VERSION-CHECKBOX" = "RVS_Checkbox: 1.2.0"; "SLUG-VERSION-OBSERVER" = "RVS_GeneralObserver: 1.1.0"; "SLUG-VERSION-GST" = "RVS_Generic_Swift_Toolbox: 1.10.1"; "SLUG-VERSION-MB" = "RVS_MaskButton: 1.2.0"; "SLUG-VERSION-PP" = "RVS_Persistent_Prefs: 1.3.1"; "SLUG-VERSION-UKT" = "RVS_UIKit_Toolbox: 1.1.5"; "SLUG-VERSION-WHITEDRAGON" = "White Dragon SDK: 3.2.2"; The only one of those that I didn't write, was KeychainSwift[0]. It makes dealing with the Keychain easy, and is a very simple dependency. If it went off the rails, I'd write something like it, myself. All the others, are in my own repos, as top-shelf-quality open-source modules. [0] https://github.com/evgenyneu/keychain-swift https://github.com/evgenyneu/keychain-swift

craigching 4y ago

> We are all in this together. The last line from the article and my favorite from Brazil which seems fitting?

aorth 4y ago

Surprised nobody mentioned yet: reload4j https://reload4j.qos.ch/ https://reload4j.qos.ch/ It's a fork of log4j 1.2.17 by the original author that fixes a boat load of issues with 1.2.x and is a drop in replacement for log4j. Very useful if you can't upgrade to log4x 2.x.

kazinator 4y ago

That's what you get for not minding that a logging library needs over 300,000 lines of code.

danhab99 4y ago

I feel like if we as the open source community would require commit signing we would be in a safer position. Crypto signing doesn't block malware from being introduced, but it would make it harder to sneek under our noses. Currently in open source, you really don't know where your code is coming from and who worked on it. Git's "commit author" fields don't require any proof of identity, that's what GPG is for.