4 ms·
Ironically, on a previous team I had switched our log4j2 log formats over to %m{nolookups} like 8 months ago... I didn't realize the whole jndi issue, what we r
by grogers 5y ago
Ironically, on a previous team I had switched our log4j2 log formats over to %m{nolookups} like 8 months ago... I didn't realize the whole jndi issue, what we ran into was the O(n^2) behavior of its string substitution.
While deploying an ancillary change, our jvms started locking up for minutes on end. What was happening was that we were logging customer input, and the change caused it to run certain things in parallel, which ended up logging the data multiple times. Normally the extra logging didn't matter but one customer had data like "${foo} ${bar} ${baz} ...". Even when the ${foo} portion is replaced wothout modification, this triggers quadratic behavior. So we were already potentially vulnerable to the DOS but it was rare enough that we never got locked up until logging the string multiple times, which then overflowed log4js internal buffer and blocked worker threads.
You can try this yourself by just logging a string like "${}${}${}..." And in fairly short order it starts taking forever. I'm very glad the fix in 2.15 is to disable lookups by default.
I hope that in the time after I left, the security org at the big tech company I worked at and reported this to (as I thought it was - a dos vector, not the complete pwnage it actually was) forced teams to switch to nolookups. Otherwise a lot of people had a bad week forcing updates through...
- kaeso 5y ago> So we were already potentially vulnerable to the DOS [...] > the security org at the big tech company I worked at and reported this to I'm confused about these two statements, because I did not find any recent CVEs for log4j in the DoS category, nor related to format lookup (other than CVE-2021-44228 of course). Perhaps I misread it, but are you basically saying that (after you reported the issue to them internally) the security team at your previous company could not successfully report a DoS vulnerability in the default configuration of a widely used (by them, at least) Apache library and make sure a CVE got assigned to track it? If so, it would be interesting to know where the CVE/vuln-reporting chain broke, possibly to reduce the blast radius for similar future cases. Hypothetically speaking, a CVE in March for a DoS in a problematic design/feature could have resulted in flipping the default setting earlier. Instead of chasing live RCE in the wild in December.
- cerved 5y agono they're saying they discovered the behavior of their Log4j that was using interpolation was so slow that is had the potential of causing a DDoS at their company