27 ms·

I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Unless their specific personal/professional projects a

• iJohnDoe 5y ago
  Great comment. I think this speaks to overall what’s happening in the world today.
• thrdbndndn 5y ago
  I knew you're deliberately exaggerating but isn't it a little bit over the top? That ("...private jets..") doesn't happen because the solution isn't exactly the hard part, and the unpaid original maintainers are doing them anyway.
  • bshipp 5y ago
    I admit to a certain level of exaggeration but, at the same time, we are talking literal peanuts to a large company. They could spend a million dollars and it'd be a rounding error on their balance sheet. In all seriousness, taking actions like I identified above would cost the companies virtually nothing but result in huge long-term benefits by signaling to the rest of the open source world that "we love your work and will be right beside you helping if the chips are down." This is, of course, not a suitable compensation model for popular open source projects. Thats a separate conversation. But it would at least be something.
    • dylan604 5y ago
      if this was a difficult task to fix, maybe support to ensure the devs can properly focus on the task would be a valid thing. however, this sounded like something not very difficult to fix. it will take longer for all of the end users to deploy fixes in their envs that it took for the patch to become available. molehill meet mountain.
    • antonvs 5y ago
      The compensation model is the problem. It doesn't mesh well with the way corporations function. If you don't charge for your product or services, corporations have no standard mechanism for addressing that some other way. Open source contracts essentially state that companies can use the product with no relevant obligations, so they do. The "huge long term benefit" you claim can't be reliably translated onto a balance sheet, so it isn't. And when companies do get involved, it's often explicitly for their direct commercial benefit, like Amazon's ElasticSearch distribution. There's also a race to the bottom aspect to it. If an open source package e.g. charges for commercial use, something more free is likely to replace it.
      • jancsika 5y ago
        > The compensation model is the problem. It doesn't mesh well with the way corporations function. If you don't charge for your product or services, corporations have no standard mechanism for addressing that some other way. FSF: Pardon me, here's a little process that encourages sharing so that everyone can... Corporations: gnawing on a giant proprietary turkey leg Doesn't Mesh Well nom nom nom No Standard Mechanism nom nom nom ** Corporations: yelling through a mouthful of turkey What in sam hell is Linux why does IBM have a billboard about Linux why don't we have Linux? Underling: Sir that's open source, our standard licensing mechanism wouldn't... Corporations: belching through a mouthful of turkey change the goddamned mechanism I want linux and pass me that mayo
        • antonvs 5y ago
          > Pardon me, here's a little process that encourages sharing But they're following the process. What the comment I replied to was saying was that they should go beyond it. I'm pointing out that the lack of standard processes for doing that is what prevents it from happening. That's just the reality of the situation. Yelling at clouds isn't going to change that.
  • joshjdr 5y ago
    For argument’s sake, at least, I don’t consider anything suggested here as definitively “over-the-top”. It may seem (or be) unrealistic in practice (for reasons I don’t know), but the suggestion is far from unconscionable— it may, in fact, be the lowest cost solution to what could cost mega-corps billions in current (and potential future) fines/liabilities. To the extent it sounds like an exaggeration, I think that embodies the point of the comment— there are some (almost unreconcilable) concerns that impact the interplay of corporations and open source development.
  • gilgad13 5y ago
    As you said, the solution isn't the hard part. The reason that large companies aren't deploying their own solutions for this issue isn't that their engineers engineers that are incapable of developing their own solutions, but because then they would have to carry that patch forever, and if a problem was found with their particular solution they would be on the hook for it. And yes, I do think this, "but everyone else is doing the same thing so it isn't really our fault" attitude is a problem.
• rackjack 5y ago
  This is a problem in open source: everybody wants the fruits of labor without paying for it. The log4j vulnerability is what happens when you don't pay for it.
  • reidrac 5y ago
    That's right. It is open source and, when it breaks, you get to keep both pieces.
    • dylan604 5y ago
      but being open source, you can pull out the crazy glue and stick them back together. if you feel generous, you can submit your crazy glue solution to the devs. if they like it, they can then make it part of the package.
  • deleted 5y ago
    [deleted]
  • coliveira 5y ago
    That's why I don't contribute to open source that is used by big corps. I don't like the idea of working for free for the benefit of billionaires.
  • skybrian 5y ago
    On the other hand, I expect that most people running log4j didn't know about or want this feature. What should they pay for something they don't want? Maybe it makes more sense to fund system-wide efforts?
  • mbrodersen 5y ago
    You give away your work with a price tag of $0. So people/companies pay $0 for it. What’s hard to understand about that?
• MattGaiser 5y ago
  Are data breaches actually treated as all that seriously? For all the talk about cyber security, there seems to generally be little investment. It appears to be viewed as more of a reputational concern than an operational one. A past organization of mine had a data breach (the kind that ended up making the news everywhere). A few people left (probably making it worse with all the turnover there), but I would be surprised if anything really changed in that organization.
  • willis936 5y ago
    A data breach isn't the primary concern here. This exploit allows full pwnage of a system and could take down entire networks for as long as it takes to rebuild them.
  • twunde 5y ago
    If the company is in healthcare or finance, yes. Otherwise the typical answer is no. Most companies just load up on cyber insurance and call it a day. That said, reputational concern, is a big thing for companies. Take Dropbox for example. Early on they suffered several security breaches, and had a bad reputation around security. They've since built out a fairly large security program, in part because bad security can block deals, especially in the enterprise space. I'll note that there's been more investment in security the last 4-5 years. Most B2B companies do a SOC2, and early on, so there tends to be a baseline of competence.
  • deleted 5y ago
    [deleted]
  • PeterisP 5y ago
    This is not really about data breaches. The first widely spread automated attacks seem to drop cryptominers, however, we should expect that (if it's not already happened) within a week or so this will get used as the entry point for ransomware attacks, since it gives attackers a solid way of getting of code execution into company servers for anyone who has not solved this issue.
• throw_log4jfang 5y ago
  > You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would (...) mitigate the problems. FAANG engineer here, and one who had to work extra hours to redeploy services with the log4j vulnerability fix. I'm not sure you understand the scope and constraints of this sort of problem. Log4j's maintainers have a far more difficult and challenging job than FANGs or any other consumer of a FLOSS package, who only need to consider their own personal internal constraints, and if push comes to shove can even force backwards-incompatible changes. The priority of any company, FANG or not, is to plug their own security holes ASAP. Until that's addressed the thought of diverting resources to fix someone else's security issues doesn't even register on the radar. I mean, are you willing to spend your weekend working around the clock to fix my problems? Why do you expect others like me to do that, then? Instead I'm spending a relaxing weekend with my family with the confort of knowing my service is safe. Why wouldn't I?
  • 2muchcoffeeman 5y ago
    > I mean, are you willing to spend your weekend working around the clock to fix my problems? Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it? As opposed to some guys with jobs who have a project on the side. The big guys could even do something like offer to pay the maintainers and maybe they can take leave or something. I agree with both sentiments. The big guys are under no obligation to fix an issue in some library they happen to use. But the log4j guys are under even less obligation when they do it in their spare time. Everyone should enjoy their weekends.
    • throw_log4jfang 5y ago
      > Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it? No, I'm not getting paid. What leads you to believe in that? My targets are defined yearly and are very well defined, and patching random FLOSS projects is not one of them. And what leads you to believe that others, such as my boss, don't have their own milestones to meet, and instead take random FLOSS requests from random people on the internet? A FANG is not a magical entity where any engineer can drop everything they're doing at the drop of a hat to work on external projects, let alone one whose only possible outcome is at best total indifference and at worse we get the company to own a problem affecting everyone for no reason whatsoever.
• phkahler 5y ago
  >> I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Agreed, while reading it I also disagreed at this point: >> the maintainers of log4j would have loved to remove this bad feature long ago, but could not because of the backwards compatibility promises they are held to. Nobody is holding them to anything. If they want to remove an old feature, go right ahead. If those using it think it's that important they can fork the project and maintain it themselves. Oh right, that would take effort or money.
  • duxup 5y ago
    Did they want to remove it because of security concerns? If so, I really wouldn’t hold someone to any backwards compatibility promise if security is a concern.
    • miohtama 5y ago
      My favorite way to dissect any open source issues: How much you paid for it? Money back guaranteed. If you paid $0 then it you are guaranteed to get $0 money back in the case of an issue.
  • omegalulw 5y ago
    > Nobody is holding them to anything I don't get this argument. Part of sharing your work is making sure what you put out is actually helpful to people. If they remove features people really like, then the library won't be as helpful - so it's perfectly fine for the OG devs to maintain this feature. The same thing with "scrambling" to fix - that could be because a sword is hanging over your head, or because you care about the people who use your work. Thinking this way, I can perfectly see them working hard to fixing this bug.
    • manquer 5y ago
      It is still a choice you make, do you want to be nice and do everything anyone asks of you ? Or stick to your principles and build only what you think is right? Developers are social creatures like anyone else and like validation and recognition from peers, that is understandable but not an excuse. It is no different from being able to say no when your boss ask you to build something you don't believe in or is against your morals, it is tough but our choice nonetheless. Their choice then to support this feature, and now to patch it. They could have no either time, no point in complaining about it after making the choice.
• sorry_outta_gas 5y ago
  I'm not sure about Amazon but Google project's zero and openfuzz teams seem to be doing a lot of good work when it comes to open-source security -- more would be nice always Personally I'd like something like a security health card/metric on opensource libaries that we could tie into CI systems/pull requests or something in the past there were so few libarries it wasn't as daunting I'd be able reason about stuff like libpng, libttf ..etc and think about them or even support them but now some projects are massive hodgepodges of thousands upon thousnads of packages
• dehrmann 5y ago
  > You'd think...these multi-billion dollar companies...would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. For the general case, the problem is that a reporter might report the vulnerability to the open source project, then the project needs to keep it a secret while they make a fix. There isn't a great way to leverage these stakeholders. It's obviously different for something like Android that is open source, but clearly Google.
• phendrenad2 5y ago
  What? It was already fixed. You just need to update. There's no need for the world's top fintech programmers to hack it out on a mountaintop somewhere. Also, the reason the maintainers are rushing to fix it is: they're worried about losing "market share". Having been in open-source circles for a long time, maintainers care GREATLY about how many users they have. They just like watching their download stats go up every year. Even if it beings them no financial rewards. It's a sort of addiction.
  • bshipp 5y ago
    > maintainers care GREATLY about how many users they have. They do. Until they don't. That inevitable day when they get yelled at in a github issue thread by a user who didn't bother reading the documentation, while staring at their kid in the living room playing video games and start wondering to themselves "why am I doing this hobby in my spare time again?" Mild dopamine hits to affirmation-addicted programmers is not the sturdiest foundation upon which to build enterprise-grade software libraries.
    • manquer 5y ago
      Partly it is because it looks good on the resume, improves job prospects, help build a consulting business or get keynote offers or write books etc. Many serious contributors eye all these aveneues to leverage their popularity, it is just not affirmation, it is also ability to monetize it
• ralph84 5y ago
  > I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. If the RCE had been responsibly disclosed instead of via tweets and PR comments, maybe there wouldn't have had to be so much scrambling. And indeed maybe ASF could have found corporate OSPOs to help with remediation. There are lots pixels being spilled on how the users of open source software should be paying for it (?), but I haven't seen much criticism of the vulnerability not being responsibly disclosed.
  • weaksauce 5y ago
    to the best of my knowledge it was discovered via a minecraft exploit and I don't think minecraft players are generally the "responsible disclosure" kinda people.
    • dogma1138 5y ago
      TBH it’s also quite mind boggling that this level of an RCE was used to hack Minecraft servers of all things. I wonder if this wasn’t intended for something bigger and they just got caught when testing this out in the field.
      • enkid 5y ago
        Or they didn't know what they had. People spend a surprising large amount of time trying to cheat at video games. They could have stumbled onto it in this context without realizing the much larger picture.
      • shagie 5y ago
        Minecraft servers were one part of it... Minecraft clients where another. If you sent a message to everyone with a client (that logged that message), everyone with a client would at least ping back to the ldap server. The issue where this was introduced was: https://issues.apache.org/jira/browse/LOG4J2-313 https://issues.apache.org/jira/browse/LOG4J2-313 That's from 2013 (Minecraft was 2 years old then) On the other hand, nothing with it really happened until someone asked if it was a security vulnerability - https://github.com/apache/logging-log4j2/pull/608#issuecomment-989762094 https://github.com/apache/logging-log4j2/pull/608#issuecomme... And then all hell broke loose. Simply said, this was a feature that was intended to make it easier for one company to use their structure with ldap lookups for where/how to log. The author of the change did what many people encourage others to do when working with open source "here's some code that I wrote, I'm contributing it back upstream." If this was part of something "bigger", it sat quiet for the better part of a decade.
• kccqzy 5y ago
  > You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. Google doesn't even use log4j. What are you talking about? The spirit of open source does not dictate that the richest companies automatically shoulder the burden of maintenance of projects they do not even use. Google already has initiatives like Summer of Code that help open source projects it does not use, and I think it's perfectly fine to draw the line there. > divert the best engineers they had So the lessons from the mythical man-month are forgotten here. At this point I don't think adding more manpower helps.
  • pabs3 5y ago
    The Android SDK depends on log4j, so they definitely do use it.
  • xena 5y ago
    Google voice was vulnerable to this, so I think this means they use log4j somewhere but I'm not an expert.
• wpietri 5y ago
  > I'm still flabbergasted that the original maintainers are rushing around trying to patch these problems. Unless their specific personal/professional projects are at risk they have no responsibility to hurry and fix a thing. Sorry, but what's the hard part to understand? Open source maintainers end up in this position because they are nice, helpful people who like using computers to solve problems for others. People who spend years on a project and then see a bigger problem arise don't suddenly turn that off. With the bigger problem, they'll want to work harder, not just hoist a middle finger and go binge Netflix without a care in the world. But I totally agree with you on the CTOs, etc. I don't expect random programmers who like working on logging to also be good at solving complicated sociotechological problems around paying for global infrastructure. But it boggles my mind that none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this. If not out of community spirit or social responsibility, then out of pure self interest.
  • bshipp 5y ago
    > none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this Indeed. Each of them has had to spend the last few days madly trying to fix this problem to avoid exposing exposing their infrastructure. Each has been, in some way, replicating the wheel to do so. I'm curious how many will actually submit their findings to the original OSS so others can learn from their experience? There's always resources to put a fire but rarely enough to install a sprinkler system.
    • wpietri 5y ago
      For sure. And this was a richly predictable fire. Not this specific problem, of course. But there have been enough fires like this in the past that it makes no sense for companies to act like it won't happen again and again.
  • ratww 5y ago
    > they'll want to work harder, not just hoist a middle finger. There is a perfectly healthy, acceptable, middle ground between those two extremes, however.
• dagmx 5y ago
  An influx of pull requests is also equally difficult for open source projects. Anything sufficiently at scale needs a set of maintainers that the commercial tech companies would then collaborate with to get the PRs going. Otherwise if everyone's just panicking and rushing to submit PRs, they'll inundate the maintainer. There's also no guarantee that even the best engineers at these companies are intimately familiar with the project, and might introduce regressions or other vulnerabilities in the process. Anyway I do agree companies should be working with OSS devs, but it shouldn't be rushed or knee jerk. It should be collaborative and measured.
• ohazi 5y ago
  > They should have been buried in useful pull requests. Drive-by pull requests during a highly visibile emergency are rarely useful.
  • manquer 5y ago
    True, but that thoughtfulness is not what is stopping major companies from contribution is it ?
• deleted 5y ago
  [deleted]