8 ms·
Log4Shell Still Has Sting in the Tail
- richbell 4y ago> When it was first revealed in early December 2021, the Log4Shell bug was described as one of the most severe security vulnerabilities ever. > The Apache Software Foundation, which maintains the open-source tool, quickly released a patch... Apache horribly mismanaged this and did not release a patch until it was already widely known and being exploited in the wild. They also messed up and had to release several subsequent patches to actually fix the vulnerability. This greatly contributed to "[the] mad scramble to plug the gaps". You can watch the chaos unfold by reading through the GitHub PR. Remember, this was disclosed to Apache in November. https://github.com/apache/logging-log4j2/pull/608#issuecomment-990094911 https://github.com/apache/logging-log4j2/pull/608#issuecomme...
- sdf4j 4y agoSaying that "Apache mismanaged" it when talking about addressing a vuln is quite awkward. Probably fueled by lack of understanding of what the foundation does. The fact that every project have to have apache stamped in their name doesn't help either.
- richbell 4y agoSo is saying "The Apache Software Foundation... quickly released a patch", but the average person doesn't care about the inner workings of the foundation because, as you mentioned, everything is stamped with "Apache". It may be technically incorrect, but most people wouldn't understand if you got into the minutiae.
- wongarsu 4y agoThey might be able to understand, but they wouldn't care. If there's a bug in Microsoft Office there's no discussion which engineer was responsible. Microsoft put their name on it and are thus first recipient of praise and blame alike. Apache Log4J2 is no different.
- layer8 4y agoApache provides infrastructure and some general organizational rules for projects, but the people who actually maintain any given Apache project are largly independent and autonomous, they are not "Apache".
- richbell 4y agoYou're right, but this misses the point of my comment. It's a distinction without a difference for most people, and trying to clarify the structure of the foundation to executives and senior leaders (technical and non-technical) only makes their perception of any Apache-branded or open source projects worse.
- layer8 4y agoMy point is one shouldn't bring up Apache in the first place when talking about a project like Log4j. It's similarly irrelevant as "hosted on GitHub".
- richbell 4y ago> My point is one shouldn't bring up Apache in the first place when talking about a project like Log4j. It's similarly irrelevant as "hosted on GitHub". That isn't comparable. - the name of the project is "Apache Log4J" - the tag-line is "*Apache Log4j* 2 is an upgrade to Log4j that provides significant improvements over its predecessor..." - it's hosted under the Apache organization - the documentation, release notes, and bug tracker is under the apache.org domain. - the artifacts are published under the org.apache namespace - etc. No competent person in the industry is going to think that GitHub is associated with the projects it hosts. There are also hundreds of projects stewarded by Eclipse, Red Hat, and other similar organizations that avoid this confusion by using sensible branding. What you're saying is more equivalent to "no reasonable person would think that Vitamin Water contains vitamins and is healthy". https://www.businessinsider.com/coca-cola-glacau-vitaminwater-misleading-advertising-lawsuit-2014-10 https://www.businessinsider.com/coca-cola-glacau-vitaminwate...
- 4y ago
- cogman10 4y agoIt was a complete misfeature from start to finish. They put in an "attempt to evaluate strings" feature. Literally as bad as someone doing a `console.log(eval(input))`. Not only is it that bad, but also it's something with just a tiny bit of inconvenience you can get with either String.format or the likes of slf4j's formatting ( `logger.info("{} value", foo.get());` ) They took the worst case for an injection attack and applied it everywhere.
- richbell 4y agoNot to mention that `log.info(eval(input))` made it effectively impossible to determine if you were compromised. If you weren't vulnerable you'd see "eval(input)" in your log, however, if you were vulnerable you wouldn't see anything.
- sys42590 4y agoIn such cases I'd normally apply Hanlon's razor ( https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor ) but this security gap is really huge and rather obvious if you have an informed look at the code. So this is one of the cases where I'd personally would not be surprised if in the far future someone admits on their death bed that malice was involved.
- jerf 4y agoThis particular bug has manifested in many places now, and in all those cases I find it plausible that it was a confluence of several independent teams (as in, not even working on the same code) adding features until, whoops, it turns out the combination is catastrophic. You find it obvious after the fact, but I don't think it is always obvious at the time. It's really easy for someone to see a feature like "interpolate strings better than ever" in their string library, and say to themselves, "Hey, that sounds good, let's turn it on!" without realizing that "better than ever" means "with arbitrary lookup from a string into a class of some sort", and then chain a few things together and hey presto you've got an arbitrary code execution vulnerability. I personally think allowing arbitrary string -> class lookup for all classes in the runtime is automatically a security code smell at best if not outright antipattern, but I don't think this is at all well understood yet. (While the general pattern is hard to avoid, you should have to register all classes/structs/values/whatever you want to automatically load from somehow. This prevents things like "I can use the OS object to execute arbitrary shell code" from automatically and unexpectedly creeping in.) Dynamic languages are all but based on such a capability, with their "eval" support. It's probably still going to be decades before this is generally understood "programmer wisdom", partially because the particular confluence of features that enables this bug is not common. It's just that when it does happen, it's catastrophic. But it's not actually that common.
- deleted 4y ago[deleted]
- bob1029 4y agoWe dropped an entire feature from our product because of this crap. We had a sidecar java process that handled certain PDF conversion tasks, but the lack of in-house java wizardry meant that we couldn't trust that small binary with regard to a spiraling vulnerability situation. Our customers were very nervous at the time. Making something work in Java is easy. Proving it is secure is not. It is almost certain that we over-reacted and shot ourselves in the foot with this, but our customers were happy with the way in which we responded. Better safe than sorry was the theme throughout.
- geodel 4y ago> Making something work in Java is easy. Proving it is secure is not. Ok, so which language it is easy to prove secure?
- eli 4y agoProve secure to a corporate procurement process. Not a formal logic proof. Any use of Java will trigger additional diligence work to prove it does not use vulnerable log4j. So in this sense almost anything else is better.
- bob1029 4y agoNone are. But, we do have sufficient resources to justify and defend decisions made with our primary development language. In this context, "prove" is more like "convince".
- orwin 4y agoI'm not a cybersecurity expert, but i follow security news as i'm often working alongside secops teams. When i read here about Log4J, is was a real "Holy shit moment". It was so easy to use, it could've been a nebula exercise. I contacted my PO asap, we found out that we had no inventory of what our VM were and what java versions were installed (worked at a bank btw). We then contacted the secops team, who just found out, and triggered all alarms. It was all hands on deck time. Quite a fun two weeks tbh.
- dylan604 4y agoWhy would your parole officer be concerned about some java vuln? Not familiar with the term PO in the way you are actually using it. What management level does that represent?
- asplake 4y agoProduct Owner (from Scrum)? Likely an org where dev teams must get permission from product people, who own the backlog. Less objectionably it might be the leader of a technical product.
- dylan604 4y agothanks. i guess it's just another of those buzzword bingo terms.
- orwin 4y agoIt means "product Owner". Basically someone who know the inside of the applications you're presenting, understand the tech debt and your release cycles, and discuss with the users about new features and potential issues. It's basically the same managment level as the tech lead/lead dev. I understand this sound like scrum/agile gibberish, but honestly this was like having a technical manager/salesman who could also help you debug stuff, probably the most usefull post of "agile at scale"
- freeqaz 4y ago(Note: I'm the person that coined the term "Log4Shell") You may be surprised when I tell you what the Apache Software foundation's yearly budget is. You'd think for software that is used by practically every Fortune 500 company and most governments, it would be something reasonable. Maybe a few hundred million dollars a year to pay for a reasonable full-time staff, right? It turns out... it's about $2 million a year. (Wikipedia[0]) This helps explain to me why the devs of Log4j directly uploaded the file "JNDIExploit.java" (the POC) to GitHub while they were patching. (Here is a full analysis about what happened[1].) They're not security people. They're volunteers working on this in addition to their full-time job. What kind of brave soul wants to trudge through and maintain log4j in their spare time for zero compensation? I appreciate the people that are capable of doing that, but I think they are rare! This whole entire vulnerability was eye opening for everybody and I have actually spent the last year building tooling on GitHub to help fix the problems that Log4Shell exposed. If you have 2 seconds to try that out or just Star the repo[2], it would be very helpful! 0: Log4j revenue https://en.wikipedia.org/wiki/The_Apache_Software_Foundation https://en.wikipedia.org/wiki/The_Apache_Software_Foundation 1: "How to Discuss and Fix Vulnerabilities in Open Source" https://www.lunasec.io/docs/blog/how-to-mitigate-open-source/ https://www.lunasec.io/docs/blog/how-to-mitigate-open-source... 2: GitHub project building better dependency patching tools https://github.com/lunasec-io/lunasec https://github.com/lunasec-io/lunasec
- MuffinFlavored 4y ago> What kind of brave soul wants to trudge through and maintain _____ for zero compensation? Can't this sentence be changed for almost any open source project?
- freeqaz 4y agoAbsolutely, but if I had to rank Open Source projects by "enjoyment level to work on", I would rank a 10+ year old Java project primarily used by enterprise companies quite low on my list! Compare that to something like Bun.js[0] which is "sexy" and written in a "cool" programming language (Zig). Or Wasp[1] which is built with Haskell and is trying to define a new programming language designed to make common dev patterns less painful. Those projects are naturally going to soak up smart people that have extra energy to share because they hate their day job but need to pay their bills. (imo) Who is left that wants to bang their head against a legacy codebase like Log4j? Maybe somebody that feels there is "clout" to be had from it? (Spitballing here, I honestly don't know!) 0: https://github.com/oven-sh/bun https://github.com/oven-sh/bun 1: https://github.com/wasp-lang/wasp https://github.com/wasp-lang/wasp
- adra 4y agoI wanna give props to all the teams who have developed great features to mitigate these types of security incidents. Props to GitHub and Amazon for both producing great sets of vulnerability scanners. Amazon gets bonus points for finding a shaded dependency embedded in a jar released though one of our docker images.
- mbfg 4y agoA couple of clarifications that i expect everyone knows, but still should be said. In all practical sense, no one "at Apache" has released any remediations for log4j. There's just a bunch of random people, probably like 3 around the world, who released these patches. "how reliant many companies are on third-party and open source code over which they have little control or visibility" - that's crap. They have complete visibility, and a good deal of control. Companies just choose not to spend money doing so. Which is true of their own code, as well. What the hell is "log4jshell" that is a completely stupid name. It confers some meaning that isn't there.
- richbell 4y ago> What the hell is "log4jshell" that is a completely stupid name. It confers some meaning that isn't there. It's a pun on Log4J and describes the vulnerability: all you need to get reverse shell is for the application to log input. What meaning do you feel it confers that isn't there? (The trend of subsequent vulnerabilities being dubbed "...4Shell" is infuriating, though.)
- freeqaz 4y agoI apologize for accidentally creating a meme here. (I got angry when I saw the recent "ProxyNotShell" vulnerability in the media...) It is an interesting exercise in the power of language though. The name "Log4Shell" was literally picked because there wasn't a CVE and people running searches for "log4j RCE" were seeing a vuln in Log4j v1 from 2016. There needed to be a unique identifier that people could search for and tag on Twitter. The name just... sort of got picked arbitrarily while I was very tired after doing many hours of security research and reading through Enterprise Java code lol. (You're right that "shell" was used to express RCE, ala Shellshock or other major vulns.) I think the subsequent vulns (Spring4Shell, Text4Shell) just use the format of the name to be more catchy and ride the coat tails of Log4Shell's notoriety. (Like a meme!)
- richbell 4y agoYour blog posts were instrumental in response efforts, and it was honestly a great name (in response to a real problem, as you've mentioned), so I'll forgive you. ;) > I think the subsequent vulns (Spring4Shell, Text4Shell) just use the format of the name to be more catchy and ride the coat tails of Log4Shell's notoriety. (Like a meme!) 100%. The problem is that it's not only lazy (like how every controversy is X-gate), it also makes executives and senior leaders shit their pants. Log4Shell was a serious wake-up call for many companies and forced them to take security and software seriously. Subsequent clout-chasing or memed vulnerabilities that evoke its name risk creating "alarm fatigue" — "The Boy Who Cried '4Shell'", so to speak. https://en.wikipedia.org/wiki/Alarm_fatigue https://en.wikipedia.org/wiki/Alarm_fatigue
- rhapsodic 4y ago[dead]
- diffxx 4y agoThe big problem here is really the library distribution system for java. Most jvm based projects use binary dependencies that are mostly sourced from maven central. A build tool could automatically block downloads of, or construction of a classpath including, binaries with known vulnerabilities. But then you might find yourself with a broken build that you can't fix yourself leaving you at the mercy of the developers of an upstream dependency that you inadvertently depend on (like log4j). I think the situation would be a lot better if java projects defaulted to source dependencies instead of binary dependencies and build tools were good at supporting remote binary caching so that you could build your own cache of binary artifacts rather than rely on a central one (so that not everyone in your org needs to burn cpu rebuilding libraries with each update). Then even if you indirectly depend on log4j, you could just grep your codebase for log4j, bump the version, rebuild everything and move on with your life. This also could make life easier for maintainers since downstream users might just fix things for them.
- mkleczek 4y agoI will say it again: please, reconsider JEP411. SecurityManager helps: https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/#what-does-that-mean-for-elasticsearch https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasti...
- thangalin 4y agoLogging is a cross-cutting concern, and most code bases employ a logger such that it violates the DRY principle. Any particular logger shouldn't be difficult to replace in a system. The issue is that developers conflate "logging" with "something happened that must be known." The former being a text message, the latter being an event. The advantage of using an event is that it decouples how the logging transpires from the information about the event. When done well, this can isolate logging to a single line of code in the code base, which is far easier to maintain than countless repetitious logger.debug(...) statements. I've blogged about this to mixed reception: https://dave.autonoma.ca/blog/2022/01/08/logging-code-smell/ https://dave.autonoma.ca/blog/2022/01/08/logging-code-smell/
- panny 4y agoLog4shell raised something else that isn't being discussed much: Staying up to date on latest versions of dependencies is not always a good idea. Log4j 1.x didn't have the log4shell vulnerability. Log4j 2.x did. Those who stayed behind on 1.x never had the log4shell issue, even though the log4shell issue raised some other new vulnerability issues on the old code. Worse yet, when that happened, the Apache project wouldn't reopen the 1.x branch to allow for fixes, necessitating a fork called reload4j. What could have been an easy version update overriding old versions became more troublesome. Now one must add exclusions for the old log4j to any dependency that has log4j 1.x transitively, and then declare the reload4j dependency to replace them. It is a very suboptimal solution which requires constant vigilance instead of a one and done fix.... all in an apparent effort to force dependency consumers to update to their new version which added major vulnerabilities.
- Beltalowda 4y ago> bring to light how reliant many companies are on third-party and open source code over which they have little control or visibility All the log4j code is open. All the discussion are open. All the commit log is open. This is the case for the overwhelming majority of "open source code" projects that many companies. It's hard to see how more "control or visibility" could be reasonably enacted for a project like log4j. All you have to do is actually get involved instead of being an apathetic consumerist and then start crying when it's not all perfect.