6 ms·
I have taken some flak over the years for rolling my own basics like logging, telemetry, tracing, authentication, session management, etc... Incidents like thi
by bob1029 5y ago
I have taken some flak over the years for rolling my own basics like logging, telemetry, tracing, authentication, session management, etc...
Incidents like this remind me that it is sometimes OK to ignore those who constantly order you to "vendor it out" over some notion of principled development excellence wherein one never reinvents a single hypothetical wheel. Arguably, writing text to a log file on disk is a simple thing you dont want to keep implementing over and over. But, it is also a simple thing that doesn't take a whole lot to do correctly in many use cases.
DIY (in aggregate) would represent one of the largest barriers to any attacker producing a wormable/DDOS exploit of this log4j-style vulnerability. Imagine ten thousand plus unique implementations for how to get a string to disk and I can almost assure you that not one of them would contain an attack vector like this. Even if one of them did, the other 99.999% of implementations would be unaffected and nothing would have made the headlines.
Depending on the code paths that others have developed is the biggest liability in all of software engineering. We are going to have to get better at evaluating this cost relative to the benefits. This doesnt stop with Log4j. There are other things more ubiquitous and more complicated out there still. IMO, the best we can do is to stop producing new software that is exposed in these ways.
- ineedasername 5y agoI'm not sure that an argument from monoculture is the right way to go here. It's merely a form of security by obscurity, and probably shouldn't be the dominant factor in whether or not you take some amount of mythical man hours to roll your own code vs. adopting a piece of very well established OSS that has had countless eyes on it over the years.
- hpoe 5y agoGiven many of the engineers i have worked with and much more importantly the PMs and Managers that work over them, I would expect vulnerabilities like this to proliferate all over the place if people were to roll their own logging and telematry and especially authentication. I mean SQL injection is such an easy known mitigation yet is still on the OWASP top 10 even after so many years.
- ineedasername 5y agoYes, people still forget to ; delete from comments where id = 29544262 */-- sanitize their inputs. ...still there?
- mypalmike 5y agoThe humor here is that sanitizing inputs is not the proper mitigation.
- mro_name 5y agobut? It think it's far deeper than mitigation, it's the solution. Edit: I apologize for getting 'sanitation' wrong. Don't do it.
- jdlshore 5y agoThe proper solution to SQL injection is parameterized queries, not input sanitization, to my knowledge.
- deleted 5y ago[deleted]
- drunkpotato 5y agoThe irony here is that if you use the log4j equivalent of parameterized queries, parameterized logging strings, you're still vulnerable to this CVE, even if you did everything right.
- AnotherGoodName 5y agoFor databases you simply parametize the inputs so that code is code and data is data and there's no mixing of the two. Sanitization is a defence of last resort when you simply can't separate code and data. Usually used for user content on the web since HTML has no formal mechanism to separate code and data because the angled brackets that do this separation are also valid user input. But databases do have a way to separate the query from the data. Parametize your queries.
- Johnny555 5y agoOn the other hand, you don't know what vulnerabilities are hidden in your own code and just haven't been discovered yet, and you're one fuzzing attack away from a vulnerability being discovered/exploited.
- tester756 5y agoWhat if there are none? It's not like library code is some magical stuff that only senior principal ninja 10x developers can comprehend Not everything is don't roll your own crypto.
- bob1029 5y ago> Not everything is don't roll your own crypto. Even for crypto, I have had people give me trouble for daring to use Microsoft's public & documented implementations of SHA256 and CSPRNG to derive session secrets. It's almost as if directly interacting with any crypto primitives is tantamount to rolling your own crypto these days.
- dharmab 5y agoBut splat 20 lines of obtuse openssl commands into a bash script and no one bats an eye!
- ineedasername 5y ago>What if there are none? The problem is that "There are no bugs" approaches an unproveable statement as a codebase grows into even a modest size. It doesn't matter if you're a 10x developer or not, you aren't always going to immediately see all the implications and edge cases of a few hundred++ (at least) moving pieces the first time around with new code, and each additional piece adds a non-linear amount of complexity. That doesn't mean you never roll your own version of something, it just means that you should either be very very good or have a lot of experience (or both) when making that decision for anything that's non-trivial. (And deciding what is/isn't trivial is its own decision too)
- mro_name 5y agoyou're advocating diversity against monoculture. Only nature has muse enough for such.
- noizejoy 5y agoI feel pretty good about betting on nature in the long term. Life in general has a pretty decent track record of surviving, even when complex life forms don’t. The argument of monoculture vs diversity may actually be a short term vs long term advantage discussion. But it’s been too long since philosophy classes, and not enough of them in any case. :-)
- jl6 5y agoWell it sure does suck to find out that you have a RCE vulnerability in your application thanks to a dependency, but I’m not sure that means it was wrong to decide to use Log4j based on the information available at the time. Having to rewrite a bunch of library code in-house is a significant burden, without a guarantee you’ll get it any more right than the Log4j developers did. I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing. The better lesson is to learn how to contain the behaviour of your dependencies so you don’t end up with ridiculous situations like backend database servers being permitted to make JNDI calls to the public internet.
- mumblemumble 5y agoTo take the example of log4j, I can see this going two ways. On the one hand, I've worked at a couple places that used homegrown logging, and it's been fine. They covered all the features I use/want out of a logging framework, in only a couple hundred lines of code. I suspect that both logging frameworks even had lower TCO than their respective platforms' name brand loggers, because the greater simplicity paid recurring dividends across the whole organization. I suspect, sometimes, that arguments against DIY being too complex have an unstated major premise that you need to implement a significant fraction of the off-the-shelf library's features. In my estimation, it's more like 10%, and then the code is only maybe 2-3%, because you also don't need any of the code for managing how the features you do need interact with all the features you don't need. On the other hand, it's not like not using something like log4j yourself is any protection from things like this. We don't use log4j where I work, but I've still had my hands full dealing with all the other software we use that relies on it.
- shoto_io 5y agoExactly. Judging a decision solely by the outcome is called "resulting" and should be avoided.
- bob1029 5y ago> I don’t think this event is sufficiently bad to overturn the principle that code reuse is a good thing. Are we open to the idea that code reuse is a delicate spectrum of possibilities? I am also curious what your threshold would be for something that is sufficiently bad to force a review of these principles. Last I checked 10.0 is the highest CVSS score available.
- neximo64 5y agoHow do you know yours isn't vulnerable?
- tfehring 5y agoOn the other hand, if you roll your own logging but import a dependency for something else (you generally can't roll your own everything) and that dependency uses log4j, isn't that strictly worse from a security perspective than just using log4j everywhere? Whatever the probability of any given logging utility exposing an RCE exploit, the probability that none of your logging utilities expose an RCE exploit decays exponentially to zero as the number of unique utilities you're using increases.
- zwegner 5y agoI don't think that's necessarily (or even probably) true in this case, since the vulnerability only exists where attacker-controlled data gets logged. If a library is just using log4j for some diagnostic messages with no user data, there's no issue, right?
- bob1029 5y ago> if a library is just using log4j for some diagnostic messages with no user data, there's no issue, right? Depends on the diagnostic messages and how you use the library. If exceptions are being logged with any level of detail, and an attacker has the ability to provoke exceptions while varying error message content, then the thing that seems safe was just made quite unsafe. The way I would look at it - If log4j methods are being invoked on any logical threading context which is serving a potential attacker, then you have something you need to investigate for safety. This is not a complete scope, but it is the most immediate because of the feedback loop being instant for an attacker. Other flavors of this attack could involve nightly log shipping and other batched procedures being poisoned with nasty messages, but these have longer cycles and will take more time to compromise.
- zwegner 5y ago> If exceptions are being logged with any level of detail, and an attacker has the ability to provoke exceptions while varying error message content, then the thing that seems safe was just made quite unsafe. Right, I would certainly include that in "user data" though. I was mainly responding to the idea that using two logging libraries is strictly worse than one: when log4j is only used for logging safe data, and another library elsewhere, this is better than using log4j everywhere.
- deleted 5y ago[deleted]
- raxxorrax 5y agoTo be fair, log4j offers more than to write a file on the disk. Logging is also quite relevant for performance as these I/O operations are a bottleneck in many applications. I think we cannot prevent bugs like this, but mitigation has to become easier.