5 ms·

I'm not saying you, as an engineer for those companies, should be the one to donate your time and energy toward the problem. We all have competing priorities,

by bshipp 5y ago

I'm not saying you, as an engineer for those companies, should be the one to donate your time and energy toward the problem. We all have competing priorities, as do the maintainers of those FLOSS packages. I'm saying that your company's CTO, especially one with a very large companies, could likely identify two or three engineers who they pull into a meeting and say "reach out to these guys and get them whatever they need. Here's my cell, call me the moment you need the plane or additional resources." Seriously, if a CTO has a budget of a few hundred million dollars and thousands of dedicated employees, how hard is it to throw a few crumbs to the open source community to change this situation from being one of a burden on a volunteer effort to, instead, one where they feel like they're in the middle of an international event where their knowledge and services are vital to keeping the internet alive? Again, I'm exaggerating, but you see where I'm going with this. It's a missed opportunity for some seriously great PR out of a seriously bad situation.

UncleMeat 5y ago
This situation is fairly urgent, but I think you might not realize just how many people a CTO at one of these companies manages. There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed. I've only seen this happen a very few times and they were for problems that were way more serious and complex. But that is not to say that nothing is being done. At Google, at least, there are organized efforts staffed with plenty of people that are trying to solve the much much much bigger problem of "secure all of our open source dependencies and all future dependencies" rather than the individual problem of "secure this one dependency." And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.
- teruakohatu 5y ago
  > And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community. Google's Project Zero is both very helpful and gets them A LOT of PR, both tech and mainstream.
  - UncleMeat 5y ago
    GPZ isn't oncall for urgent bugfixes and, while a truly excellent project filled with great people, isn't the core team responsible for safe imported code.
- Aeolun 5y ago
  > There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed. If all your IT projects have an RCE vulnerability that’s relatively easy to exploit, that should keep you up at night.
  - UncleMeat 5y ago
    The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago? The dirty secret is that an absolutely enormous amount of code is vulnerable and that the solution to software security is not "fix RCEs as they are discovered as fast as possible." If having RCEs keeps you up at night, then I don't believe that there is a single engineer at almost any company in the world that interfaces with the internet that should be able to sleep. The actual solutions here are at a more abstract layer than individual vulns.
    - Aeolun 5y ago
      No, the fact of having these vulnerabilities is not a problem (I mean, obviously, but at the level you describe). The problem is having them be known to the world. Especially with a level of publicity like this.
    - cesarb 5y ago
      > The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago? A week ago, this vulnerability might have been known at most to a few three-letter agencies. Today, every two-bit script kiddie will be trying to exploit it. It's not hard to see how the situation has changed.
raggi 5y ago
As with earlier comments this seems to oversimplify the problem of throwing people at a problem. Adding people to a project puts more pressure on the current maintainers, to authenticate, validate, train and support newcomers. Apache has processes for this, and project maintainers pointed people in that direction repeatedly (e.g. https://github.com/apache/logging-log4j2/pull/608#issuecomment-990217634 https://github.com/apache/logging-log4j2/pull/608#issuecomme..., https://github.com/apache/logging-log4j2/pull/608#issuecomment-990264908 https://github.com/apache/logging-log4j2/pull/608#issuecomme...). The Apache foundation receives funding from a large number of organizations already: https://www.apache.org/foundation/thanks.html https://www.apache.org/foundation/thanks.html Perhaps the right question to ask here is: what did Apache do to help their members in this event? You can ask this question of the Apache foundation independently, without adding pressure on the project maintainers at this time.
jcranmer 5y ago
Some time ago, I saw this suggestion from a disaster relief specialist directed towards those who want to help with disaster relief: the best thing you can do after a disaster is stay away. Taking yourself to the disaster zone very often at best consumes scarce resources from those present to manage the disaster trying to bring you up to speed and at worst creates new problems that need solving. It's not hard to extend this to the kind of software security flaw here. If I'm a developer on a package with a critical security vulnerability that needs to be fixed now, sending me extra developers who know absolutely nothing about the code I'm working on isn't going to be helpful--it's just going to waste my time trying to bring them up to speed (or more often, telling them to just go away). If I actually need help, I'll ask the people who I know can help me; trying to sift through unsolicited help to figure out who actually has the skills to do so would take too much time. So think hard about what help Google et al could actually be providing to help log4j here. If you have to resort to clear exaggeration to find examples... maybe that's a sign that there actually isn't all that much that they could be doing that would actually be helpful.
- bshipp 5y ago
  > So think hard about what help Google et al could actually be providing to help log4j here. I think you make a perfectly valid point and one that shouldn't be overlooked. How about this: "Here's a $100K and an isolated penthouse suite down the road rented for the month where you can focus on fixing the problem and not be interrupted by screaming children. Here's a phone number if you need to delegate any specific tasks to additional teams." Incentive to help. No added pressure. Just one practical example.
  - zaphirplane 5y ago
    You didn’t explain why it’s a perfect valid point It doesn’t seem reasonable for Johnson & Johnston at a valuation of 1/2 trillion to free load You are kind of talking about greater good, perhaps those charitable donations should go to medical research or homeless shelters rather than reducing the burden on for profit companies
    - 5y ago
juanbyrge 5y ago
I am guessing FANG engineers (and most engineers in general) would unanimously suggest "delete this ridiculous , ill-conceived JNDI integration ASAP. If people want JNDI integration, use a custom opt-in log4j appender. Don't let this shit be enabled by default". Yet that may not sit well with the log4j folks.
- flatiron 5y ago
  Log4j taking user input to run and execute code is crazy. You don’t think it “sits well”?
deleted 5y ago
[deleted]
ramblerman 5y ago
This is almost laughably naive. The irony is that you are criticizing poor IT leadership, by offering a suggestion that is beyond poor. "Let's get 3 of your best guys in touch with the guys over at log4j...." As if you can just wire up some engineers to reach out to each other on unrelated projects and magic will happen.
watwut 5y ago
Throwing new people who never seen your project at the mainteners during critical situation will just slow them down. Adding random new people generally does not immediately speeds up the process, but it is actively harmful in the middle of crisis.

by bshipp 5y ago

UncleMeat 5y ago

This situation is fairly urgent, but I think you might not realize just how many people a CTO at one of these companies manages. There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed. I've only seen this happen a very few times and they were for problems that were way more serious and complex. But that is not to say that nothing is being done. At Google, at least, there are organized efforts staffed with plenty of people that are trying to solve the much much much bigger problem of "secure all of our open source dependencies and all future dependencies" rather than the individual problem of "secure this one dependency." And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.

teruakohatu 5y ago
> And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community. Google's Project Zero is both very helpful and gets them A LOT of PR, both tech and mainstream.
- UncleMeat 5y ago
  GPZ isn't oncall for urgent bugfixes and, while a truly excellent project filled with great people, isn't the core team responsible for safe imported code.
Aeolun 5y ago
> There are going to be "OSS fires" more or less constantly so "some major OSS project has a bad vuln" is not the sort of thing that gets a CTO at a company like Google or Facebook out of bed. If all your IT projects have an RCE vulnerability that’s relatively easy to exploit, that should keep you up at night.
- UncleMeat 5y ago
  The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago? The dirty secret is that an absolutely enormous amount of code is vulnerable and that the solution to software security is not "fix RCEs as they are discovered as fast as possible." If having RCEs keeps you up at night, then I don't believe that there is a single engineer at almost any company in the world that interfaces with the internet that should be able to sleep. The actual solutions here are at a more abstract layer than individual vulns.
  - Aeolun 5y ago
    No, the fact of having these vulnerabilities is not a problem (I mean, obviously, but at the level you describe). The problem is having them be known to the world. Especially with a level of publicity like this.
  - cesarb 5y ago
    > The RCE existed prior to this disclosure. If I can't sleep today, why should I have been able to sleep a week ago? A week ago, this vulnerability might have been known at most to a few three-letter agencies. Today, every two-bit script kiddie will be trying to exploit it. It's not hard to see how the situation has changed.

raggi 5y ago

As with earlier comments this seems to oversimplify the problem of throwing people at a problem. Adding people to a project puts more pressure on the current maintainers, to authenticate, validate, train and support newcomers. Apache has processes for this, and project maintainers pointed people in that direction repeatedly (e.g. https://github.com/apache/logging-log4j2/pull/608#issuecomment-990217634 https://github.com/apache/logging-log4j2/pull/608#issuecomme..., https://github.com/apache/logging-log4j2/pull/608#issuecomment-990264908 https://github.com/apache/logging-log4j2/pull/608#issuecomme...). The Apache foundation receives funding from a large number of organizations already: https://www.apache.org/foundation/thanks.html https://www.apache.org/foundation/thanks.html Perhaps the right question to ask here is: what did Apache do to help their members in this event? You can ask this question of the Apache foundation independently, without adding pressure on the project maintainers at this time.

jcranmer 5y ago

Some time ago, I saw this suggestion from a disaster relief specialist directed towards those who want to help with disaster relief: the best thing you can do after a disaster is stay away. Taking yourself to the disaster zone very often at best consumes scarce resources from those present to manage the disaster trying to bring you up to speed and at worst creates new problems that need solving. It's not hard to extend this to the kind of software security flaw here. If I'm a developer on a package with a critical security vulnerability that needs to be fixed now, sending me extra developers who know absolutely nothing about the code I'm working on isn't going to be helpful--it's just going to waste my time trying to bring them up to speed (or more often, telling them to just go away). If I actually need help, I'll ask the people who I know can help me; trying to sift through unsolicited help to figure out who actually has the skills to do so would take too much time. So think hard about what help Google et al could actually be providing to help log4j here. If you have to resort to clear exaggeration to find examples... maybe that's a sign that there actually isn't all that much that they could be doing that would actually be helpful.

bshipp 5y ago
> So think hard about what help Google et al could actually be providing to help log4j here. I think you make a perfectly valid point and one that shouldn't be overlooked. How about this: "Here's a $100K and an isolated penthouse suite down the road rented for the month where you can focus on fixing the problem and not be interrupted by screaming children. Here's a phone number if you need to delegate any specific tasks to additional teams." Incentive to help. No added pressure. Just one practical example.
- zaphirplane 5y ago
  You didn’t explain why it’s a perfect valid point It doesn’t seem reasonable for Johnson & Johnston at a valuation of 1/2 trillion to free load You are kind of talking about greater good, perhaps those charitable donations should go to medical research or homeless shelters rather than reducing the burden on for profit companies
  - 5y ago

juanbyrge 5y ago

I am guessing FANG engineers (and most engineers in general) would unanimously suggest "delete this ridiculous , ill-conceived JNDI integration ASAP. If people want JNDI integration, use a custom opt-in log4j appender. Don't let this shit be enabled by default". Yet that may not sit well with the log4j folks.

flatiron 5y ago
Log4j taking user input to run and execute code is crazy. You don’t think it “sits well”?

deleted 5y ago

[deleted]

ramblerman 5y ago

This is almost laughably naive. The irony is that you are criticizing poor IT leadership, by offering a suggestion that is beyond poor. "Let's get 3 of your best guys in touch with the guys over at log4j...." As if you can just wire up some engineers to reach out to each other on unrelated projects and magic will happen.

watwut 5y ago

Throwing new people who never seen your project at the mainteners during critical situation will just slow them down. Adding random new people generally does not immediately speeds up the process, but it is actively harmful in the middle of crisis.