17 ms·

Xz: A microcosm of the interactions in open source projects

526 points by transpute 3y ago

gladiatr72 3y ago
Funny. I was just saying the same thing to one of my partners just 4 hours ago. Culpability also must be laid at RedHat's feet for sanctioning the practice of side loading libraries into such a critical service's address space. Their drive to cellularize systems management has overtaken their common sense. The idea that they could not be bothered to answer the call of the sole maintainer of a library used in such a critical path in his time of need is, well, astonishing.
- dralley 3y ago
  1) Debian includes this downstream patch, also. 2) A potential explanation for "why now" is that systemd DID prevent these dependencies from loading automatically in a patch one month ago [0], and the patches to lzma enabling the backdoor merged a few days later, followed by (as we know) an immediate and somewhat heavy push to get distros to upgrade driven by sockpuppets. It could be a total coincidence, or it could be that the attacker jumped to pull the trigger before the window of vulnerability started closing on them [0] https://github.com/systemd/systemd/pull/31550#issuecomment-1972737923 https://github.com/systemd/systemd/pull/31550#issuecomment-1... I think it's a bit cheap to blame systemd here, and systemd does not equate directly to Red Hat either.
  - 1oooqooq 3y ago
    I believe they were talking about sideloading a binary tarball instead of building and repackaging. Choosing a distro is nothing but choosing where you place your trust. I can understand debian cutting corners here and there. But RH have little excuses with the money they make. Yet, even a superficial analysis, show them to be less trustworthy than the anime-avatars maintaining gentoo or arch.
    - dralley 3y ago
      Arch was doing the same..
fredamaral 3y ago
awkward, but heard very recently that open source is not "vc backable, go away". maybe it will change now, after the infrastructure pillars of the modern world ruins in front of those many saas/ai/web3/cloud/whatever investors
- creshal 3y ago
  Why should VCs back oss infrastructure directly? It doesn't help the VCs and only creates perverse incentives for the oss projects. The SaaS/cloud/etc. companies themselves should fund the projects they depend on. They actually know what those are and don't have to force their own monetization/growth models onto the projects.
  - 331c8c71 3y ago
    > The SaaS/cloud/etc. companies themselves should fund the projects they depend on. There is a bit of a free rider problem it seems.
    - FridgeSeal 3y ago
      Noooooooo you can’t ask them to pay! You’re supposed to do this for the good of the community only! They ought to be able to take whatever you do and resell at their leisure, after all you made it open source!
      - cherryteastain 3y ago
        And don't forget that GPL is bad, only BSD and MIT are the real free licenses because they let us take the code and sell it on without forcing us to contribute back.
  - fredamaral 3y ago
    definitely. not talking about the vc specifically, but the investment chain. a typical free rider, directly or indirectly.
  - 3y ago
forgotpwd16 3y ago
>Our no-longer-reasonable requestor also offers a suggestions. Notice there is no offer to actually help. Help in maintainship how? Patches had already been made and were awaiting to be reviewed and merged. This was up to the maintainer to do and requestor couldn't help with it.
- ribasushi 3y ago
  > Help in maintainship how? Pay.
  - amszmidt 3y ago
    That assumes the maintainer wants to be paid. There are plenty of us who maintain FLOSS projects that do it for other reasons and any monetary exchange would burden us, since it might pressure one that this is now a job and you have to execute on tasks - there are enough headaches handling other things as it is.
    - kamma4434 3y ago
      Agreed - I already have a job where people come asking. If developing OSS came with obligations, I’d do something else instead.
      - creato 3y ago
        That’s of course a fine approach for you. I think a big part of the problem in this case is not with the maintainer, but with the critical software that took a dependency on this hobby project (in the maintainer’s own words).
        kamma4434 3y ago
        Agreed.
  - yunohn 3y ago
    Huh? So anyone who asks/says something in OSS, should just throw money into the ring to have an opinion?
SuperHeavy256 3y ago
So the first step of this huge mess was: a social engineering attack. Attacking a tired, burnt-out open source project developer and peer pressuring him into giving more control of the repo to the attacker.
- aborsy 3y ago
  Enabled by customers who don’t pay or donate.
  - grumpyprole 3y ago
    Are these customers or additional attackers who have never posted before and will never post again?
    - kijin 3y ago
      Tired maintainers have no way to distinguish one from the other, that's the problem. Even if we say "no payment, no customer," it won't prevent determined attackers from paying significant amounts of laundered money in order to be treated as customers.
      - Ekaros 3y ago
        Also one thing that is easy to come across at this type of work is money. I mean in the cases where someone is injecting backdoors or vulnerabilities. Might not be for individuals or criminal groups. But once agencies and corporations get involved, the sums are trivial...
  - rini17 3y ago
    On the other hand, accepting significant amounts of money causes overhead (accounting, taxation). Plus it reinforces the psychological obligation and it's not fun anymore. Thus many maintainers avoid it.
- makapuf 3y ago
  Well before that: letting a central lib and project to be maintained by a single tires burned out project dev (see xkcd2347)
feverzsj 3y ago
Considering how kind(naive) our open source projects maintainers are, backdoors are probably already everywhere.
- lionkor 3y ago
  theyre much harder to add if people use a buildsystem from the last, idk, 20 years
  - feverzsj 3y ago
    Didn't Jia Tan add a single '.' in cmake script to disable the landlock?
    - astrange 3y ago
      That was some sort of theoretical security loosening, it didn't contribute to the backdoor.
- redox99 3y ago
  I don't think this kind of backdoor is that widespread. But intentional vulnerabilities for sure are.
WesolyKubeczek 3y ago
“Community desires more” At which point if “the community” consists of one guy doing everything and a couple whiners making demands, fuck it.
- resource_waste 3y ago
  As someone who has a community who follows my stuff and demands more.... I somewhat get it. I became the central spot for this particular product. I get occasional emails from people saying the did this same thing, but on a smaller scale. I have scale, so I can do things much bigger than anyone else. I have a million people using this, vs a startup who has ~100. (Btw, I'm barely profitable, I was selling to frugal people...)
- kijin 3y ago
  Well, in this case it's even worse. The "community" here in all likelihood was overrun with sock puppets of a malicious state-level actor.
  - WesolyKubeczek 3y ago
    I also don’t really understand the notion of a community around a fucking data compression program. “Bro, do you even compress?” Is there a community of ½" plumbing pipe enthusiasts?
- keybored 3y ago
  I by default distrust any statement that claims to speak on behalf of some community.
- noobermin 3y ago
  This line in light of what we know now makes me think that while OP has a point about open source, this thread and others like it definitely were sockpuppets trying to goad Collin into transferring maintainership quickly.
sumanthvepa 3y ago
The Jigar Kumar account should be treated with extreme suspicion. It seems like it was part of a social engineering attack.
- armada651 3y ago
  Given how all participants in the discussion were very focused on transferring ownership to the attacker I'd treat everyone in the entire e-mail thread with suspicion. I would not be surprised if Dennis Ens, the one who started the thread in the first place, was also a sock puppet of Jia Tan.
  - WesolyKubeczek 3y ago
    The word that springs to mind is “campaigning”. If persons campaigning are nobodies as their contribution is concerned, whatever they are pushing for should be considered noise and put straight into the bin.
- WhyNotHugo 3y ago
  I don’t agree with speculating like this, but I also can help wonder how many participants in this thread are the same person.
kimixa 3y ago
I do sometimes wonder if by trying to be "nice" to users and try to see the best intentions of commenters, many developers waste huge amounts of mental energy. For context, I've really only worked on "fun" side projects, namely emulators and game remakes, where I've explicitly avoided any mention of donations or similar. Both as it's intended to be a distraction from my job, not become part of it. And generally avoid as many issues as possible around said money distribution within the project, and possible copyright problems that such things often skirt. And those sort of projects tend to be extremely limited in progress not by total interest, but skilled interest with the ability to contribute. I wish as many as 1% of the regular discussion participants in the community could contribute to that level. There's a natural belief that unless someone is egregiously out of line, all discussion around a project from users is allowed, even encouraged. But many people just can't seem to help themselves making "suggestions" (or, another reading, "demands"), and that can have a significant impact on motivation of volunteers. I believe the vast majority are from based in good intentions, wanting a "better" result, but arguably are mostly self-defeating in that I find the discussion around much of these things to be draining and demotivating. I want to have "fun" coding game remakes, not defusing drama in discord threads. But it's expected to have a "community" for all such projects, and not exclude non-direct contributors. Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men. But I sometimes wonder if allowing this sort of open community is better in the long run.
- TeMPOraL 3y ago
  > Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men. There's this weird trend that's been happening for some time now that tries to make non-fully-open groups look wrong, but honestly, has anything ever actually been done by such open groups? As far as I can tell, "closed community" is a necessary (but not sufficient) condition for any kind of quality creative output (this includes individual projects as single-person closed group). Having a core of creators surrounded by distinct group of fans and secondary contributors is a natural organization that forms spontaneously.
  - lukan 3y ago
slavik81 3y ago
I'm starting to feel that one of the lessons here is that individuals invited into trusted positions should be identifiable. Jia Tan is not a real person. We don't know who they are, so there is no way to hold them accountable.
- armada651 3y ago
  As much as I support the privacy of maintainers I do feel like this is necessary to raise the bar for these types of social engineering attacks. However that would've likely done little to impede this attack if it is backed by organized crime or by a state as is being speculated. It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air.
  - sedatk 3y ago
    Such identity checks would also hurt honest contributors who want to hide their identity for other, legitimate reasons.
  - avsteele 3y ago
    It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air. Can you provide a historical example of when something like that has happened? It sounds like hyperbole to say that it is 'easy' (even for a state) to impersonate a real-life identity or create one for a professional developer with a multi-year work history.
- fulafel 3y ago
  Reputation works with pseudonymous identities too, and those have security upsides (eg can't be as easily pressed into service of others by extortion or rubber hose). And of course privacy is a value in itself.
  - dist-epoch 3y ago
    You can build multiple pseudonymous identities in parallel. If one is burned, it doesn't matter, you still have 9 other.
MASNeo 3y ago
Sadly, this is how it works in many large social systems, left unchecked. The “reasonable consumers” drain the kind contributors until they cannot do this anymore. The curse of kindness strikes in software and other industries alike.
- goku12 3y ago
  This is all due to the ongoing attempt to convert FOSS into an obligation. The original idea behind FOSS was to code to scratch your itch and then leave it out there for anyone to find, modify and use. Instead we now treat FOSS code like commercial projects with performance goals and expectations. Why? My guess is that this new culture is promoted to extract free labor from hobby developers. Take for example, Github. It has a bot to close stale issues after a given interval of inactivity. But what if the issue was valid and unresolved? What's the problem in leaving it open for someone interested to take it up? But what GH wants to promote is a culture where open issues are considered as bad performance on the maintainer's side. It indirectly prompts the maintainers to take open issues too seriously.
  - kqr 3y ago
    > Why? Well, also a lot of people get into FOSS for altruistic reasons. They want to pay it forward and make the world a better place. For these people, applying good operations principles allows them to more efficiently do good -- just like it allows a commercial company to deliver more end-user value for lower cost. Running an efficient operation in FOSS isn't necessarily bad. Sometimes it takes an obligation viewpoint to get there. It can be unhealthy, but it can also give us crazy levels of neat software, as we see proof of daily.
keybored 3y ago
I agree that the maintainer doesn’t really owe anything to anyone. Including the so-called consumers on the issue tracker. Furthermore the consumers have no real power over the maintainer; since they aren’t doing anything for them they in turn have nothing to withhold from the maintainer. And they can’t bother the maintainer unless they decide to turn truly vile and evil by doxing and harassing the maintainer. Thus the complaints of these consumers can be completely tuned out. It follows then that the maintainer can just log off. Because either you believe that the maintainer has some minimal obligation to the community (like a quarterly update about maintainership status or making a notice in the readme if you decide to abandon the project) or you don’t. But sometimes in these discussions things get muddied because people claim that 1. The maintainer doesn’t owe anything to anyone whatever 2. But she’s a nice gal which means that she will triage some bugs and fix some bugs and explain to a few of the issue reporters that they haven’t encountered a bug they just haven’t installed the program so that PowerShell can find it and What’s the groans moral angle, here? We’ve already established (1). But what ought the maintainer do for herself? That’s also a moral problem. Are you willing to say that the maintainer ought to log off if they are experiencing burnout? (Again: this is about the moral obligations that the maintainer has to herself only.) Or does that trample on her rights? And if you are unwilling to say that the maintainer ought to log off, what does that make the maintainer? Are you going to claim that they have (a) started an altruistic hobby by themselves (b) got too caught up in it and (c) are now a victim of circumstance/outside forces because they have no moral obligation to log off? Considering how much power the maintainer has (and how the consumers have none), how is the maintainer a victim of circumstance when the whole enterprise was created by them and can be terminated at will by them?
lifeisstillgood 3y ago
>>> Software developers are not fungible cogs that you can swap in and out at will. I am thinking a lot about this. One of the issues is scale and proof. I suspect that I am interested in introducing gated ability to comment / participate in a community Say for example github introduces “gates”. The first might be add a test to the test suite that generates a haha of the version number and test output. Then adding that number to your profile means github trusts you have at least downloaded and run make test. It shows some level of commitment. (I suppose the zero level is logging in to github and commenting on some maintainers mental health)
- Kwpolska 3y ago
  What would that magic number achieve? Having basic coding ability can be checked by looking at the user's repos and contributions. But the primary attacker had pretty good skills, and the other commenters (the attacker's sockpuppets) could also demonstrate some skills.
- klabb3 3y ago
  Are you suggesting to ask a state actor to demonstrate commitment before participating in the discussion? :) I mean, the idea of commitment -> privilege isn’t bad. But it’s got nothing to offer in the story of xz.
dwb 3y ago
I'm under no illusions that this is a totally new thought, but for me first with cryptocurrencies, then "AI", and now this, the fundamental issue that the biggest problems come back to is one of trust. Cryptocurrencies try to code around it, LLM boosters try to dazzle you into it, and the attacker here half-succeeded in laundering it. The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust. In this case, the failure was 100% understandable – I have all the sympathy for burnt-out and (almost always) unpaid open source developers. In each case, capital either lures people away from thinking about trust, or neglects and exploits people to the point that they're broken down.
- zvmaz 3y ago
  > The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust. It is implicit in your statement that you know some "proper" way to think about trust. If it is the case, please do share with us your thoughts on trust.
  - cassianoleal 3y ago
    I didn't read it that way. I read it as in "actually giving it thought" as opposed to just barely acknowledging it or even at all. So, not so much a judgement of the quality of the thought but rather of its existence in the first place.
    - dwb 3y ago
      Yes, exactly this. I certainly don't claim to have any big answers, but at the same time I think these problems with trust were very clear very early on.
  - o11c 3y ago
    Trust is a lot easier if real-life identities and real-life consequences are involved. Unfortunately the Open Source community has a long history of ties to anonymity culture, which is fundamentally untrustworthy.
- 3y ago
lifeisstillgood 3y ago
Some random thoughts: - every Fortune 500 company tracks exactly which FOSS code it includes in its ecosystem (usually code scanning and fingerprinting - can’t remember the usual Provider of such) - this is essentially the software BOM that Biden signed a while back. - this (made public) would give a real time map of the dependancies of all organisations - and linking that to things like the above thread (“cry for help”) would be an interesting place for “intervention” - anything from plain old cash to “here are three interns doing two years in gov.uk. They will help for the next 2 years (It’s not a great idea that last one but we need to start somewhere- no way can government “pick winners” but also no way can society just sit back and hope. And commercial incentives break this horribly. Essentially we are going to find at some point we treat some developers like lawyers - this stuff is our societies laws, rules, processes
- kzrdude 3y ago
  > “here are three interns doing two years in gov.uk. They will help for the next 2 years Having temp workers come and go into all kinds of various open source projects.. does that help? :)
  - lifeisstillgood 3y ago
    Does two years count as temp? What if one in five stays on for “life” I am not claiming I have perfect solutions, just that we need to be more mindful of our supply chains. And when we are we start demanding things of those supply chains - and if we are sensible we support those suppliers
- klabb3 3y ago
  Agreed it’s a decent way to create a “map” of vulnerable projects. But it’s over-fitting the MO of adversaries - plugging an arbitrary hole. Why not play into the strengths of OSS instead of trying to replicate frictionous trust models from elsewhere? Like, listen to ourselves. We are admitting here that open source software is not feasible to audit? Like did we just accept that an adversary can basically smuggle exploit payloads in plain sight? Can we at least try to simplify and reduce the messiness of checked in generated code, irreproducible builds, and magical binary blobs, conditional compilation, “ifuncs”? Wtf? Granted I’m not a domain expert in this area, but I’m horrified by all the complexity for what I understand to be a compression library. I would much rather play whack-a-mole with obscurity, which as a side effect is great for software quality overall.
xenocratus 3y ago
As a maintainer of a security-oriented open source library, the paranoia of "is this person trying to help or to exploit?" weighs down on me every time I have to read a PR (EDIT: even though the libraries are by no means as widely used), regardless of whether it's from a long-time contributor or someone new. I think accepting a slower pace of development is the only viable solution (as I'm not interested in making these libraries my full-time job), but that comes with the same feelings reflected in the article listed. If there was a simple way of advertising the need for help to a community of experts that I/we could trust, I'd take that any day.
- deleted 3y ago
  [deleted]
- barcist 3y ago
  > As a maintainer of a security-oriented open source library, the paranoia of "is this person trying to help or to exploit?" That's an excellent mind set when reviewing code, no matter security or not. But especially for security. How could this be wrong? What are the corner cases? How could anyboy break this? What do we need to test? That kind of scrutiny is crucial for keeping the quality of your code base high, no matter who posts the PR.
  - saulpw 3y ago
    Yes but that's a huge amount of work, and is not the 'fun' part of coding. It's one thing to have to spend those cycles looking for errors or incompetence, and hopefully helping the contributor improve their skills so they can in turn help the project more in the future. It's quite another to spend even more energy being suspicious and looking for subterfuge from someone that has been one of your most helpful contributors.
draw_down 3y ago
[dead]
thrnkkm 3y ago
[flagged]
- pityJuke 3y ago
  What does working for free have anything to do with diversity? I don’t get how these two points are linked.
- polio 3y ago
  This is a non-sequitur.
zb3 3y ago
Could we somehow search for such takeovers done in the past?
tkiolp4 3y ago
If I were a chinese hacker trying to do something evil, why on earth would I use a chinese handler/username? Wouldn’t it be better to use an English/European name to gain (even more) trust from open source maintainers? On the other hand, if I were a non-chinese hacker trying to do evil, then using a chinese handler does make more sense (China is evil, blah blah blah)
- yunohn 3y ago
  100% this. I don’t know why this obvious angle is being ignored. The other puppet accounts had Indian and German names - it’s very clear they were trying to obscure their real origins.
  - michaellee88 3y ago
    The hacker's name "Jia Cheong Tan" actually includes phonics from both variants of Chinese from China, Singpore and Hong Kong region, hence no way it is a name of a real person, so the name probably don't indicate anything. However, since the Hong Kong and Singapore variants is rarely used, the group behind the hacker probably has good knowledge of Chinese linguistics. Either way, the fact that the hacker's account shows GMT+8 and that the hacker's activity is mostly in Southeast Asia working hours (read GMT+8) would only make sense if the hacker is originated from Asia countries. I doubt that group of people behind the hack can consistently stay late night for 2+ years just to fake the timezone.
    - mik1998 3y ago
      > that the hacker's activity is mostly in Southeast Asia working hours (read GMT+8) Opposite is true. His activity is focused around midnight in UTC+0800, with significant activity at 2AM in that timezone.
- damagednoob 3y ago
  Something I've observed as an English speaking immigrant is that in general, the US/UK is very forgiving when it comes to foreigners trying to speak English. Where certain types of phrasing and harsh words would not be tolerated with first language English speakers, the benefit of the doubt is given to 2nd/3rd language speakers because they 1. Might not have the vocabulary to express themselves correctly 2. Not understand the nuance and connotations that their word choices imply I've seen my colleagues get frustrated in meetings while simulatenously having to assume good intent. Choosing names like the bad actor(s) did gives them an advantage in this scenario.
deleted 3y ago
[deleted]
nuc1e0n 3y ago
My take away from this is that people are still far too blasé about introducing hard dependencies and complexity, even after the left-pad incident as a warning. OpenSSH is a massive wall of code. Such complex systems are inherently untrustworthy to me, no matter what language they might be written in. Even with ernest devs there are still more opportunities for mistakes.
- lrvick 3y ago
  Agreed, which is why IMO ssh in production is a terrible idea that reveals even worse problems. A server with ssh generally implies there is a shell, and cli tools, and an administration model that involves humans connecting to servers to manually update them in place like pets. Having a full workstation-optimized distro like ubuntu or debian with hundreds of packages constantly shifting and updating as a critical production server is wildly high risk in terms of both reliability and security. I understand this is how most sysadmins were taught but it is a 70s unix mainframe mindset from a time before security was a thing and everyone on the internet was a good actor. Only a workstation or dev server should have tools for humans like ssh installed. Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. No production systems I launch today even have xz installed, or a package manager, or a shell.
  - ris 3y ago
    > Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. And then you need to debug something. What do?
    - nuc1e0n 3y ago
      Have good logging and virtual machines that can replay those logs offline?
rafpin 3y ago
Would ssh servers with port knocking set up be safe from this backdoor? I'm not sure I got it correctly, but seems the RCE can only be performed after connecting to the ssh server, but if the port is hidden behind a reasonable sequence of tcp/udp knocks, then it won't happen? I've been using port knocking on ssh servers, and it definitely does not replace proper ssh configuration, but so far seems like a cheap extra layer of defense that might either prevent or give extra time to respond when these ssh vulnerabilities appear.
- lrvick 3y ago
  I do not use or recommend security through obscurity. Such things can always be automatically discovered. https://github.com/eliemoutran/KnockIt https://github.com/eliemoutran/KnockIt Any effort spent on setups like this is likely better spent removing the need for having ssh at all by moving to immutable appliance distros be they specialized like homeassistant or general purpose like TalosOS.
  - est 3y ago
    > security through obscurity This is not obscurity, as someone said on HN earlier, good security is good armor, good port knocking is camouflage. They are both important.
  - rafpin 3y ago
    > Such things can always be automatically discovered. Wouldn't fail2ban + reasonably sized (10+ ports using a mix of udp and tcp) knock sequence prevent brute force attacks like these?
    - lrvick 3y ago
      You can potentially go down those rabbit holes, then an attacker gets fixated on you for being so unusual and they come back doing only 2 attempts per ip per day with thousands of IPs. I have been on both sides of this game many times. Better time investment in security is just do not have ssh or a shell in production in the first place, because ssh is probably not required for your production application to function.
jmclnx 3y ago
This is another interesting link about the issue: https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... It mentions something not being reported, the issue is with a Linux specific patch to OpenSSH. If the patch did not exist there would be no issue. This is not an OpenSSH issue but a Linux issue.
- iforgotpassword 3y ago
  > It mentions something not being reported This has been reported and discussed from the start, even in the initial report. Also it's not a Linux specific patch or issue, rather a patch used by some Linux distros for tighter systemd integration.
nukeyoular 3y ago
[dead]
jpz 3y ago
I think the idea this was HUMINT operation by a state sponsored intelligence service is more likely. The twitter thread here was interesting. https://x.com/thegrugq/status/1774392858101039419 https://x.com/thegrugq/status/1774392858101039419 Raging about these being inconsiderate people, when they were likely fictional personalities that were part of a long con seems to be a bit foolish to me.
- mustache_kimono 3y ago
  > I think the idea this was HUMINT operation by a state sponsored intelligence service is more likely. It's not an either/or proposition. I definitely think it was state sponsored, AND one method used was social engineering a burned out maintainer.
  - bloak 3y ago
    It seems to me that people are very much exaggerating how "professional" this attack was. Yes, it doesn't look like the actions of a single bored teenager but I don't think the government of a country like the USA or China would deliberately permit their employees to get involved with crap like this. Any backdoor they try to insert would look exactly like an innocent bug. So my (uninformed) guess would be that this is done by criminals, something like a ransomware gang branching out a bit. Though North Korea sometimes sponsors activities that are indistinguishable from those of a criminal gang so it could come from there. I'm just speculating, of course. I don't know anything really.
    - mustache_kimono 3y ago
      > I don't think the government of a country like the USA or China would deliberately permit their employees to get involved with crap like this. Not the USA, but I can easily imagine this is China. Because, as of right now, this seems like the way China does business. The Chinese/the PLA fund a hacking complex of contractors, and when one gets caught, they simply deny involvement. [0],[1] [0]: https://www.npr.org/2024/02/22/1233178131/leaked-document-trove-shows-a-chinese-hacking-scheme-focused-on-harassing-dissid https://www.npr.org/2024/02/22/1233178131/leaked-document-tr... [1]: https://www.washingtonpost.com/world/2024/02/21/china-hacking-leak-documents-isoon/ https://www.washingtonpost.com/world/2024/02/21/china-hackin...
EasyMark 3y ago
Saying that it's a microcosm for "open source" is a bit hyperbolic. What would be more accurate is saying that it's a microcosm of "there is a certain set of users to be ignored" when writing open source software. One has to learn to develop a thick skin and guilt free ignoring of those who don't contribute anything to a project such as constructive suggestions, bug reports, or actual code.
higeorge13 3y ago
Serious question, but what is your expectations for linux and all these libraries/dependencies after 20-40+ years when most of old school or current devs are retired? How can anyone guarantee reliable continuity by good actors in all these dependencies?
- apantel 3y ago
  Steve Jobs said that someone needs to be the keeper and maintainer of the vision. I think that’s the key to the making of anything great over the long run: there needs to be a Benevolent Dictator For Life who has the vision, competence, energy, passion, and caring to consistently iterate the thing well. The best case scenario for a project that loses its BDFL(s) is to hold on maybe as decently as Apple has under Tim Cook. They haven’t done anything truly innovative since Jobs died in my opinion — they just figured out more ways to apply the combination of a multitouch display, compact computer, and sensors; but they’ve iterated well on everything they produce and have held on to their lead in terms of hardware, user experience, and ecosystem cohesion. In short: someone very competent has to care a lot about the project for the right reasons, and continue driving it forward. If the old guard is rotating out, someone has to step up. If no one steps up, then leeches of various types will attach themselves to the project and simultaneously milk it for value and kill it.
  - mnau 3y ago
    44% of active open source projects have only one maintainer. Out of all maintainers, 60% would describe themselves as unpaid hobbyist. They already are BDFL for themselves or one other person. The problem is a chronic deficiency of maintainers, which is at the root of this attack. Your comment implies there is a wide pool of peoplecwho want to do that, along with mythical "someone". Here is a little story: There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it. Somebody got angry about that because it was Everybody’s job. Everybody thought that Anybody could do it, but Nobody realized that Everybody wouldn’t do it. It ended up that Everybody blamed Somebody when Nobody did what Anybody could have done. - Charles R. Swindoll
rrr_oh_man 3y ago
> This is the way it works. It needs to change. Well, it’s on you to set your boundaries. "Feel free to submit a PR" works wonders.
- matheusmoreira 3y ago
  Be careful what you wish for. Someone will eventually take you up on that challenge. The responsibility will lie with you when they do make that pull request. Is that something you actually want? I gotta ask because it seems there's lots of people out there who are saying that but don't actually want it to happen. Look I get it, it's a great line, it instantly weeds out anyone who can't code, it even filters out those who can code but don't actually care enough about your project. If you choose that route though, you better be prepared for the day when the fabled PR finally arrives in your inbox. You told them to do it themselves if they wanted it so bad, they went and did it. Even though you know your code base and could probably have implemented it 10 times faster than they did, they labored over it to spare you the trouble instead of begging for the feature in your issue tracker. You asked to see the code, you got the code. Now the buck stops with you. Seriously, if you say this, be ready to engage with whomever does show up with a patch. Unless they send you literal irredeemable garbage, the least they are entitled to is your sincere peer review. Nothing's worse than putting in effort into learning somebody else's project and writing the patch, only to not be given the time of day, or worse. I'd rather have a Linus Torvalds type take a big shit on the patch in public than be ignored like that. Someone who does that is at least treating you like a peer.
  - rrr_oh_man 3y ago
    I like (and upvoted) how you replied, but disagree with you. What's the problem, honestly? Telling someone that you don't like / want their code / direction / feature? It just takes a modicum of emotional maturity to do that.
gavinhoward 3y ago
And this is why, for my build system coming out in two days, you need to pay to have me respond to bug reports.
- sethammons 3y ago
  That would stop a nation state or other nefarious actor because? This isn't spam that works in volume
  - gavinhoward 3y ago
    It means I can ignore loud but non-paying users. I also don't take outside contributions.
- fl0ki 3y ago
  You just reinvented the "it's open source but we charge for support" model which has been so well known for so long that it was a line in the movie Antitrust (2001).
krupan 3y ago
This should make us all reconsider any criticisms we've levied towards Linus Torvalds, Richard Stallman, or any other gruff open source project lead
- asylteltine 3y ago
  [dead]
- account42 3y ago
  It should also make us reconsider the motivations of those who push such criticisms.
jrochkind1 3y ago
The multi-entity (sock puppet or not hard to say) social engineering attack that was apparently two years in the making (?), is to me a much more salient story than "users are often mean." Although it's true that the expectation of mean users let the attack fit right in, I guess.
- mwcampbell 3y ago
  The "users are mean" story is something that we can all do something about, and, honestly, I prefer stories with morals that most of us can actually put into practice.
  - jrochkind1 3y ago
    Ah, well, I guess my thought is that we need to figure out what to do about the "take over from a burned out maintainer and then inject malware" attack. It's not obvious what to do about it to me either. Which is why I'm concerned to talk about it. That this attack was run by someone who had been participating in the project for possibly years before making the attack -- is not what i would have expected, and makes it even much harder to defend against. Before I was thinking it was about awareness of "don't just turn over the thing to someone new who just showed up, they might be an attacker." But not that easy in this case. I suppose "try to get users to be less mean, by doing our part by being less mean individually" is arguably one piece of strenghtening defenses to this kind of attack, I guess, ok.
    - transpute 3y ago
      > I suppose "try to get users to be less mean, by doing our part by being less mean individually" is arguably one piece of strenghtening defenses to this kind of attack, I guess, ok. Yes, by denying cover traffic.
lamontcg 3y ago
> ”Progress will not happen until there is new maintainer. … The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this.” While this might be a sockpuppet deliberately manipulating the maintainer, the tone of this post is all too common, and the real thing that pisses me off is actually that last sentence. The use of the words "sad", "shame" and "pity" have been weaponized by a generation of people to dump negative emotions onto to overworked/understaffed people in issue trackers. If you're using those words to express an opinion in an issue tracker, you're probably being a fucking asshole.
anthk 3y ago
Not an issue of OpenSSH, but some GNU/Linux distros patching and linking OpenSSH against xz.
bshanks 3y ago
Even if other users (or possibly, sock puppets of the attacker) had not been complaining, the original maintainer probably would have began to trust the attacker after enough real contributions.
shp0ngle 3y ago
Stop making software open source, start making it closed source and charge for licenses. That works! Unfortunately it's bad for open source.
cryptonector 3y ago
> This thread is a microcosm of the interactions in Open Source projects. Consumers make demands (some polite, some not-so-polite) of one maintainer (rarely two) that does everything. > Make no mistake. This is the way it works. > It needs to change. How?
- pizzafeelsright 3y ago
  Deny by default anything from a State that's hostile to American interests.
  - fluoridation 3y ago
    How would that help with the problem of people being rude to unpaid OSS maintainers?
    - pizzafeelsright 3y ago
      Rude is subjective and words are not violence. I think the OSS worked as intended.
      - fluoridation 3y ago
        >Rude is subjective The person you replied to accepts the supposition that some users of open source software are rude when they interact with the developers, and asks how this could change. If you don't accept the premise that some users are rude then you can't pretend to answer the question. >words are not violence And? If your coworker reaches into your desk drawer while you're not looking and takes money out of your wallet that's also not violence, yet I imagine you would not think it's acceptable behavior just for that reason.
  - shp0ngle 3y ago
    let's add passport (and visa) checks to github
semiquaver 3y ago
At first I thought it was paranoid to suggest that people on the linked thread might be complicit in the attack but then I read this message: > You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs? It genuinely looks like we’re seeing a demonstration of supply chain psyops in retrospect. Amazing how sophisticated and patient this attack was.
- jijijijij 3y ago
  > It genuinely looks like we’re seeing a demonstration of supply chain psyops in retrospect. Worthwhile noting: It happened in the open, archived for the world to see at any time! The attacker needed to be extra careful not to raise suspicion, both acutely and for accumulated evidence in history. Of course, easy to say "Hindsight is 20/20", but we can probably agree in actual hindsight, we easily see a lot of suspicious acts around the issue. An individual incident may be chance, but as a whole things become clear more easily. So, I think there is a teachable moment here: When something seems just a tiny bit fishy, do a background check, investigate. You may not catch all something, but you probably won't miss everything. Now, looking at the level of sophistication and risk sneaking people and code into open infrastructure, imagine chances of getting caught infiltrating a large company writing closed source binary blobs for drivers, firmware, ...
  - shp0ngle 3y ago
    There are not that many suspicious acts tbh. Randos complaining about unmaintained repos and unclosed issues is a constant in FOSS world. Sometimes it's even true, some projects really die and stop actually addressing real issues. But that's just inherent downside of the "bazaar" model. I don't think how we can "treat maintainers better" without going full corporate/without going full "cathedral".
    - transpute 3y ago
      > But that's just inherent downside of the "bazaar" model. I don't think how we can "treat maintainers better" without going full corporate/without going full "cathedral". We now have two decades of arms-race data in OSS projects influenced by ESR's paper [1]. At least one bazaar [2] has operated for centuries. Bazaars can develop decentralized responses to dynamic local threats. [1] http://www.catb.org/~esr/writings/cathedral-bazaar/ http://www.catb.org/~esr/writings/cathedral-bazaar/ [2] https://en.wikipedia.org/wiki/Grand_Bazaar,_Istanbul https://en.wikipedia.org/wiki/Grand_Bazaar,_Istanbul
archi42 3y ago
User Hakkin made an interesting observation: The "consumers" are likely sock puppets controlled by the attacker. I did not independently verify the claim, but it seems reasonable to me. Post with Hakkin's reply: https://news.ycombinator.com/item?id=39869703 https://news.ycombinator.com/item?id=39869703 (But yeah, I've seen some entitled idiots trying to push their demands on FOSS maintainers and being irritated by the harsh reaction...)

526 points by transpute 3y ago

Funny. I was just saying the same thing to one of my partners just 4 hours ago. Culpability also must be laid at RedHat's feet for sanctioning the practice of side loading libraries into such a critical service's address space. Their drive to cellularize systems management has overtaken their common sense. The idea that they could not be bothered to answer the call of the sole maintainer of a library used in such a critical path in his time of need is, well, astonishing.

dralley 3y ago
1) Debian includes this downstream patch, also. 2) A potential explanation for "why now" is that systemd DID prevent these dependencies from loading automatically in a patch one month ago [0], and the patches to lzma enabling the backdoor merged a few days later, followed by (as we know) an immediate and somewhat heavy push to get distros to upgrade driven by sockpuppets. It could be a total coincidence, or it could be that the attacker jumped to pull the trigger before the window of vulnerability started closing on them [0] https://github.com/systemd/systemd/pull/31550#issuecomment-1972737923 https://github.com/systemd/systemd/pull/31550#issuecomment-1... I think it's a bit cheap to blame systemd here, and systemd does not equate directly to Red Hat either.
- 1oooqooq 3y ago
  I believe they were talking about sideloading a binary tarball instead of building and repackaging. Choosing a distro is nothing but choosing where you place your trust. I can understand debian cutting corners here and there. But RH have little excuses with the money they make. Yet, even a superficial analysis, show them to be less trustworthy than the anime-avatars maintaining gentoo or arch.
  - dralley 3y ago
    Arch was doing the same..

fredamaral 3y ago

awkward, but heard very recently that open source is not "vc backable, go away". maybe it will change now, after the infrastructure pillars of the modern world ruins in front of those many saas/ai/web3/cloud/whatever investors

creshal 3y ago
Why should VCs back oss infrastructure directly? It doesn't help the VCs and only creates perverse incentives for the oss projects. The SaaS/cloud/etc. companies themselves should fund the projects they depend on. They actually know what those are and don't have to force their own monetization/growth models onto the projects.
- 331c8c71 3y ago
  > The SaaS/cloud/etc. companies themselves should fund the projects they depend on. There is a bit of a free rider problem it seems.
  - FridgeSeal 3y ago
    Noooooooo you can’t ask them to pay! You’re supposed to do this for the good of the community only! They ought to be able to take whatever you do and resell at their leisure, after all you made it open source!
    - cherryteastain 3y ago
      And don't forget that GPL is bad, only BSD and MIT are the real free licenses because they let us take the code and sell it on without forcing us to contribute back.
- fredamaral 3y ago
  definitely. not talking about the vc specifically, but the investment chain. a typical free rider, directly or indirectly.
- 3y ago

forgotpwd16 3y ago

>Our no-longer-reasonable requestor also offers a suggestions. Notice there is no offer to actually help. Help in maintainship how? Patches had already been made and were awaiting to be reviewed and merged. This was up to the maintainer to do and requestor couldn't help with it.

ribasushi 3y ago
> Help in maintainship how? Pay.
- amszmidt 3y ago
  That assumes the maintainer wants to be paid. There are plenty of us who maintain FLOSS projects that do it for other reasons and any monetary exchange would burden us, since it might pressure one that this is now a job and you have to execute on tasks - there are enough headaches handling other things as it is.
  - kamma4434 3y ago
    Agreed - I already have a job where people come asking. If developing OSS came with obligations, I’d do something else instead.
    - creato 3y ago
      That’s of course a fine approach for you. I think a big part of the problem in this case is not with the maintainer, but with the critical software that took a dependency on this hobby project (in the maintainer’s own words).
      - kamma4434 3y ago
        Agreed.
- yunohn 3y ago
  Huh? So anyone who asks/says something in OSS, should just throw money into the ring to have an opinion?

SuperHeavy256 3y ago

So the first step of this huge mess was: a social engineering attack. Attacking a tired, burnt-out open source project developer and peer pressuring him into giving more control of the repo to the attacker.

aborsy 3y ago
Enabled by customers who don’t pay or donate.
- grumpyprole 3y ago
  Are these customers or additional attackers who have never posted before and will never post again?
  - kijin 3y ago
    Tired maintainers have no way to distinguish one from the other, that's the problem. Even if we say "no payment, no customer," it won't prevent determined attackers from paying significant amounts of laundered money in order to be treated as customers.
    - Ekaros 3y ago
      Also one thing that is easy to come across at this type of work is money. I mean in the cases where someone is injecting backdoors or vulnerabilities. Might not be for individuals or criminal groups. But once agencies and corporations get involved, the sums are trivial...
- rini17 3y ago
  On the other hand, accepting significant amounts of money causes overhead (accounting, taxation). Plus it reinforces the psychological obligation and it's not fun anymore. Thus many maintainers avoid it.
makapuf 3y ago
Well before that: letting a central lib and project to be maintained by a single tires burned out project dev (see xkcd2347)

feverzsj 3y ago

Considering how kind(naive) our open source projects maintainers are, backdoors are probably already everywhere.

lionkor 3y ago
theyre much harder to add if people use a buildsystem from the last, idk, 20 years
- feverzsj 3y ago
  Didn't Jia Tan add a single '.' in cmake script to disable the landlock?
  - astrange 3y ago
    That was some sort of theoretical security loosening, it didn't contribute to the backdoor.
redox99 3y ago
I don't think this kind of backdoor is that widespread. But intentional vulnerabilities for sure are.

WesolyKubeczek 3y ago

“Community desires more” At which point if “the community” consists of one guy doing everything and a couple whiners making demands, fuck it.

resource_waste 3y ago
As someone who has a community who follows my stuff and demands more.... I somewhat get it. I became the central spot for this particular product. I get occasional emails from people saying the did this same thing, but on a smaller scale. I have scale, so I can do things much bigger than anyone else. I have a million people using this, vs a startup who has ~100. (Btw, I'm barely profitable, I was selling to frugal people...)
kijin 3y ago
Well, in this case it's even worse. The "community" here in all likelihood was overrun with sock puppets of a malicious state-level actor.
- WesolyKubeczek 3y ago
  I also don’t really understand the notion of a community around a fucking data compression program. “Bro, do you even compress?” Is there a community of ½" plumbing pipe enthusiasts?
keybored 3y ago
I by default distrust any statement that claims to speak on behalf of some community.
noobermin 3y ago
This line in light of what we know now makes me think that while OP has a point about open source, this thread and others like it definitely were sockpuppets trying to goad Collin into transferring maintainership quickly.

sumanthvepa 3y ago

The Jigar Kumar account should be treated with extreme suspicion. It seems like it was part of a social engineering attack.

armada651 3y ago
Given how all participants in the discussion were very focused on transferring ownership to the attacker I'd treat everyone in the entire e-mail thread with suspicion. I would not be surprised if Dennis Ens, the one who started the thread in the first place, was also a sock puppet of Jia Tan.
- WesolyKubeczek 3y ago
  The word that springs to mind is “campaigning”. If persons campaigning are nobodies as their contribution is concerned, whatever they are pushing for should be considered noise and put straight into the bin.
WhyNotHugo 3y ago
I don’t agree with speculating like this, but I also can help wonder how many participants in this thread are the same person.

kimixa 3y ago

I do sometimes wonder if by trying to be "nice" to users and try to see the best intentions of commenters, many developers waste huge amounts of mental energy. For context, I've really only worked on "fun" side projects, namely emulators and game remakes, where I've explicitly avoided any mention of donations or similar. Both as it's intended to be a distraction from my job, not become part of it. And generally avoid as many issues as possible around said money distribution within the project, and possible copyright problems that such things often skirt. And those sort of projects tend to be extremely limited in progress not by total interest, but skilled interest with the ability to contribute. I wish as many as 1% of the regular discussion participants in the community could contribute to that level. There's a natural belief that unless someone is egregiously out of line, all discussion around a project from users is allowed, even encouraged. But many people just can't seem to help themselves making "suggestions" (or, another reading, "demands"), and that can have a significant impact on motivation of volunteers. I believe the vast majority are from based in good intentions, wanting a "better" result, but arguably are mostly self-defeating in that I find the discussion around much of these things to be draining and demotivating. I want to have "fun" coding game remakes, not defusing drama in discord threads. But it's expected to have a "community" for all such projects, and not exclude non-direct contributors. Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men. But I sometimes wonder if allowing this sort of open community is better in the long run.

TeMPOraL 3y ago
> Even writing it down here I'm worried I sound like I'm trying to curate a closed community of ego-boosting yes men. There's this weird trend that's been happening for some time now that tries to make non-fully-open groups look wrong, but honestly, has anything ever actually been done by such open groups? As far as I can tell, "closed community" is a necessary (but not sufficient) condition for any kind of quality creative output (this includes individual projects as single-person closed group). Having a core of creators surrounded by distinct group of fans and secondary contributors is a natural organization that forms spontaneously.
- lukan 3y ago

slavik81 3y ago

I'm starting to feel that one of the lessons here is that individuals invited into trusted positions should be identifiable. Jia Tan is not a real person. We don't know who they are, so there is no way to hold them accountable.

armada651 3y ago
As much as I support the privacy of maintainers I do feel like this is necessary to raise the bar for these types of social engineering attacks. However that would've likely done little to impede this attack if it is backed by organized crime or by a state as is being speculated. It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air.
- sedatk 3y ago
  Such identity checks would also hurt honest contributors who want to hide their identity for other, legitimate reasons.
- avsteele 3y ago
  It is trivially easy for those types of actors to simply use a stolen identity or create an entirely plausible one out of thin air. Can you provide a historical example of when something like that has happened? It sounds like hyperbole to say that it is 'easy' (even for a state) to impersonate a real-life identity or create one for a professional developer with a multi-year work history.
fulafel 3y ago
Reputation works with pseudonymous identities too, and those have security upsides (eg can't be as easily pressed into service of others by extortion or rubber hose). And of course privacy is a value in itself.
- dist-epoch 3y ago
  You can build multiple pseudonymous identities in parallel. If one is burned, it doesn't matter, you still have 9 other.

MASNeo 3y ago

Sadly, this is how it works in many large social systems, left unchecked. The “reasonable consumers” drain the kind contributors until they cannot do this anymore. The curse of kindness strikes in software and other industries alike.

goku12 3y ago
This is all due to the ongoing attempt to convert FOSS into an obligation. The original idea behind FOSS was to code to scratch your itch and then leave it out there for anyone to find, modify and use. Instead we now treat FOSS code like commercial projects with performance goals and expectations. Why? My guess is that this new culture is promoted to extract free labor from hobby developers. Take for example, Github. It has a bot to close stale issues after a given interval of inactivity. But what if the issue was valid and unresolved? What's the problem in leaving it open for someone interested to take it up? But what GH wants to promote is a culture where open issues are considered as bad performance on the maintainer's side. It indirectly prompts the maintainers to take open issues too seriously.
- kqr 3y ago
  > Why? Well, also a lot of people get into FOSS for altruistic reasons. They want to pay it forward and make the world a better place. For these people, applying good operations principles allows them to more efficiently do good -- just like it allows a commercial company to deliver more end-user value for lower cost. Running an efficient operation in FOSS isn't necessarily bad. Sometimes it takes an obligation viewpoint to get there. It can be unhealthy, but it can also give us crazy levels of neat software, as we see proof of daily.

keybored 3y ago

I agree that the maintainer doesn’t really owe anything to anyone. Including the so-called consumers on the issue tracker. Furthermore the consumers have no real power over the maintainer; since they aren’t doing anything for them they in turn have nothing to withhold from the maintainer. And they can’t bother the maintainer unless they decide to turn truly vile and evil by doxing and harassing the maintainer. Thus the complaints of these consumers can be completely tuned out. It follows then that the maintainer can just log off. Because either you believe that the maintainer has some minimal obligation to the community (like a quarterly update about maintainership status or making a notice in the readme if you decide to abandon the project) or you don’t. But sometimes in these discussions things get muddied because people claim that 1. The maintainer doesn’t owe anything to anyone whatever 2. But she’s a nice gal which means that she will triage some bugs and fix some bugs and explain to a few of the issue reporters that they haven’t encountered a bug they just haven’t installed the program so that PowerShell can find it and What’s the groans moral angle, here? We’ve already established (1). But what ought the maintainer do for herself? That’s also a moral problem. Are you willing to say that the maintainer ought to log off if they are experiencing burnout? (Again: this is about the moral obligations that the maintainer has to herself only.) Or does that trample on her rights? And if you are unwilling to say that the maintainer ought to log off, what does that make the maintainer? Are you going to claim that they have (a) started an altruistic hobby by themselves (b) got too caught up in it and (c) are now a victim of circumstance/outside forces because they have no moral obligation to log off? Considering how much power the maintainer has (and how the consumers have none), how is the maintainer a victim of circumstance when the whole enterprise was created by them and can be terminated at will by them?

lifeisstillgood 3y ago

>>> Software developers are not fungible cogs that you can swap in and out at will. I am thinking a lot about this. One of the issues is scale and proof. I suspect that I am interested in introducing gated ability to comment / participate in a community Say for example github introduces “gates”. The first might be add a test to the test suite that generates a haha of the version number and test output. Then adding that number to your profile means github trusts you have at least downloaded and run make test. It shows some level of commitment. (I suppose the zero level is logging in to github and commenting on some maintainers mental health)

Kwpolska 3y ago
What would that magic number achieve? Having basic coding ability can be checked by looking at the user's repos and contributions. But the primary attacker had pretty good skills, and the other commenters (the attacker's sockpuppets) could also demonstrate some skills.
klabb3 3y ago
Are you suggesting to ask a state actor to demonstrate commitment before participating in the discussion? :) I mean, the idea of commitment -> privilege isn’t bad. But it’s got nothing to offer in the story of xz.

dwb 3y ago

I'm under no illusions that this is a totally new thought, but for me first with cryptocurrencies, then "AI", and now this, the fundamental issue that the biggest problems come back to is one of trust. Cryptocurrencies try to code around it, LLM boosters try to dazzle you into it, and the attacker here half-succeeded in laundering it. The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust. In this case, the failure was 100% understandable – I have all the sympathy for burnt-out and (almost always) unpaid open source developers. In each case, capital either lures people away from thinking about trust, or neglects and exploits people to the point that they're broken down.

zvmaz 3y ago
> The most consequential (rightly or wrongly) technologists of our time are failing to properly think about trust. It is implicit in your statement that you know some "proper" way to think about trust. If it is the case, please do share with us your thoughts on trust.
- cassianoleal 3y ago
  I didn't read it that way. I read it as in "actually giving it thought" as opposed to just barely acknowledging it or even at all. So, not so much a judgement of the quality of the thought but rather of its existence in the first place.
  - dwb 3y ago
    Yes, exactly this. I certainly don't claim to have any big answers, but at the same time I think these problems with trust were very clear very early on.
- o11c 3y ago
  Trust is a lot easier if real-life identities and real-life consequences are involved. Unfortunately the Open Source community has a long history of ties to anonymity culture, which is fundamentally untrustworthy.
3y ago

lifeisstillgood 3y ago

Some random thoughts: - every Fortune 500 company tracks exactly which FOSS code it includes in its ecosystem (usually code scanning and fingerprinting - can’t remember the usual Provider of such) - this is essentially the software BOM that Biden signed a while back. - this (made public) would give a real time map of the dependancies of all organisations - and linking that to things like the above thread (“cry for help”) would be an interesting place for “intervention” - anything from plain old cash to “here are three interns doing two years in gov.uk. They will help for the next 2 years (It’s not a great idea that last one but we need to start somewhere- no way can government “pick winners” but also no way can society just sit back and hope. And commercial incentives break this horribly. Essentially we are going to find at some point we treat some developers like lawyers - this stuff is our societies laws, rules, processes

kzrdude 3y ago
> “here are three interns doing two years in gov.uk. They will help for the next 2 years Having temp workers come and go into all kinds of various open source projects.. does that help? :)
- lifeisstillgood 3y ago
  Does two years count as temp? What if one in five stays on for “life” I am not claiming I have perfect solutions, just that we need to be more mindful of our supply chains. And when we are we start demanding things of those supply chains - and if we are sensible we support those suppliers
klabb3 3y ago
Agreed it’s a decent way to create a “map” of vulnerable projects. But it’s over-fitting the MO of adversaries - plugging an arbitrary hole. Why not play into the strengths of OSS instead of trying to replicate frictionous trust models from elsewhere? Like, listen to ourselves. We are admitting here that open source software is not feasible to audit? Like did we just accept that an adversary can basically smuggle exploit payloads in plain sight? Can we at least try to simplify and reduce the messiness of checked in generated code, irreproducible builds, and magical binary blobs, conditional compilation, “ifuncs”? Wtf? Granted I’m not a domain expert in this area, but I’m horrified by all the complexity for what I understand to be a compression library. I would much rather play whack-a-mole with obscurity, which as a side effect is great for software quality overall.

xenocratus 3y ago

As a maintainer of a security-oriented open source library, the paranoia of "is this person trying to help or to exploit?" weighs down on me every time I have to read a PR (EDIT: even though the libraries are by no means as widely used), regardless of whether it's from a long-time contributor or someone new. I think accepting a slower pace of development is the only viable solution (as I'm not interested in making these libraries my full-time job), but that comes with the same feelings reflected in the article listed. If there was a simple way of advertising the need for help to a community of experts that I/we could trust, I'd take that any day.

deleted 3y ago
[deleted]
barcist 3y ago
> As a maintainer of a security-oriented open source library, the paranoia of "is this person trying to help or to exploit?" That's an excellent mind set when reviewing code, no matter security or not. But especially for security. How could this be wrong? What are the corner cases? How could anyboy break this? What do we need to test? That kind of scrutiny is crucial for keeping the quality of your code base high, no matter who posts the PR.
- saulpw 3y ago
  Yes but that's a huge amount of work, and is not the 'fun' part of coding. It's one thing to have to spend those cycles looking for errors or incompetence, and hopefully helping the contributor improve their skills so they can in turn help the project more in the future. It's quite another to spend even more energy being suspicious and looking for subterfuge from someone that has been one of your most helpful contributors.

draw_down 3y ago

[dead]

thrnkkm 3y ago

[flagged]

pityJuke 3y ago
What does working for free have anything to do with diversity? I don’t get how these two points are linked.
polio 3y ago
This is a non-sequitur.

zb3 3y ago

Could we somehow search for such takeovers done in the past?

tkiolp4 3y ago

If I were a chinese hacker trying to do something evil, why on earth would I use a chinese handler/username? Wouldn’t it be better to use an English/European name to gain (even more) trust from open source maintainers? On the other hand, if I were a non-chinese hacker trying to do evil, then using a chinese handler does make more sense (China is evil, blah blah blah)

yunohn 3y ago
100% this. I don’t know why this obvious angle is being ignored. The other puppet accounts had Indian and German names - it’s very clear they were trying to obscure their real origins.
- michaellee88 3y ago
  The hacker's name "Jia Cheong Tan" actually includes phonics from both variants of Chinese from China, Singpore and Hong Kong region, hence no way it is a name of a real person, so the name probably don't indicate anything. However, since the Hong Kong and Singapore variants is rarely used, the group behind the hacker probably has good knowledge of Chinese linguistics. Either way, the fact that the hacker's account shows GMT+8 and that the hacker's activity is mostly in Southeast Asia working hours (read GMT+8) would only make sense if the hacker is originated from Asia countries. I doubt that group of people behind the hack can consistently stay late night for 2+ years just to fake the timezone.
  - mik1998 3y ago
    > that the hacker's activity is mostly in Southeast Asia working hours (read GMT+8) Opposite is true. His activity is focused around midnight in UTC+0800, with significant activity at 2AM in that timezone.
damagednoob 3y ago
Something I've observed as an English speaking immigrant is that in general, the US/UK is very forgiving when it comes to foreigners trying to speak English. Where certain types of phrasing and harsh words would not be tolerated with first language English speakers, the benefit of the doubt is given to 2nd/3rd language speakers because they 1. Might not have the vocabulary to express themselves correctly 2. Not understand the nuance and connotations that their word choices imply I've seen my colleagues get frustrated in meetings while simulatenously having to assume good intent. Choosing names like the bad actor(s) did gives them an advantage in this scenario.

deleted 3y ago

[deleted]

nuc1e0n 3y ago

My take away from this is that people are still far too blasé about introducing hard dependencies and complexity, even after the left-pad incident as a warning. OpenSSH is a massive wall of code. Such complex systems are inherently untrustworthy to me, no matter what language they might be written in. Even with ernest devs there are still more opportunities for mistakes.

lrvick 3y ago
Agreed, which is why IMO ssh in production is a terrible idea that reveals even worse problems. A server with ssh generally implies there is a shell, and cli tools, and an administration model that involves humans connecting to servers to manually update them in place like pets. Having a full workstation-optimized distro like ubuntu or debian with hundreds of packages constantly shifting and updating as a critical production server is wildly high risk in terms of both reliability and security. I understand this is how most sysadmins were taught but it is a 70s unix mainframe mindset from a time before security was a thing and everyone on the internet was a good actor. Only a workstation or dev server should have tools for humans like ssh installed. Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. No production systems I launch today even have xz installed, or a package manager, or a shell.
- ris 3y ago
  > Production servers should be hardened immutable appliance kernels with read only root filesystems that verify and run signed containers, or run a tiny shim init system in a couple hundred lines that spawns a single application specific binary you trust. And then you need to debug something. What do?
  - nuc1e0n 3y ago
    Have good logging and virtual machines that can replay those logs offline?

rafpin 3y ago

Would ssh servers with port knocking set up be safe from this backdoor? I'm not sure I got it correctly, but seems the RCE can only be performed after connecting to the ssh server, but if the port is hidden behind a reasonable sequence of tcp/udp knocks, then it won't happen? I've been using port knocking on ssh servers, and it definitely does not replace proper ssh configuration, but so far seems like a cheap extra layer of defense that might either prevent or give extra time to respond when these ssh vulnerabilities appear.

lrvick 3y ago
I do not use or recommend security through obscurity. Such things can always be automatically discovered. https://github.com/eliemoutran/KnockIt https://github.com/eliemoutran/KnockIt Any effort spent on setups like this is likely better spent removing the need for having ssh at all by moving to immutable appliance distros be they specialized like homeassistant or general purpose like TalosOS.
- est 3y ago
  > security through obscurity This is not obscurity, as someone said on HN earlier, good security is good armor, good port knocking is camouflage. They are both important.
- rafpin 3y ago
  > Such things can always be automatically discovered. Wouldn't fail2ban + reasonably sized (10+ ports using a mix of udp and tcp) knock sequence prevent brute force attacks like these?
  - lrvick 3y ago
    You can potentially go down those rabbit holes, then an attacker gets fixated on you for being so unusual and they come back doing only 2 attempts per ip per day with thousands of IPs. I have been on both sides of this game many times. Better time investment in security is just do not have ssh or a shell in production in the first place, because ssh is probably not required for your production application to function.

jmclnx 3y ago

This is another interesting link about the issue: https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... It mentions something not being reported, the issue is with a Linux specific patch to OpenSSH. If the patch did not exist there would be no issue. This is not an OpenSSH issue but a Linux issue.

iforgotpassword 3y ago
> It mentions something not being reported This has been reported and discussed from the start, even in the initial report. Also it's not a Linux specific patch or issue, rather a patch used by some Linux distros for tighter systemd integration.

nukeyoular 3y ago

[dead]

jpz 3y ago

I think the idea this was HUMINT operation by a state sponsored intelligence service is more likely. The twitter thread here was interesting. https://x.com/thegrugq/status/1774392858101039419 https://x.com/thegrugq/status/1774392858101039419 Raging about these being inconsiderate people, when they were likely fictional personalities that were part of a long con seems to be a bit foolish to me.

mustache_kimono 3y ago
> I think the idea this was HUMINT operation by a state sponsored intelligence service is more likely. It's not an either/or proposition. I definitely think it was state sponsored, AND one method used was social engineering a burned out maintainer.
- bloak 3y ago
  It seems to me that people are very much exaggerating how "professional" this attack was. Yes, it doesn't look like the actions of a single bored teenager but I don't think the government of a country like the USA or China would deliberately permit their employees to get involved with crap like this. Any backdoor they try to insert would look exactly like an innocent bug. So my (uninformed) guess would be that this is done by criminals, something like a ransomware gang branching out a bit. Though North Korea sometimes sponsors activities that are indistinguishable from those of a criminal gang so it could come from there. I'm just speculating, of course. I don't know anything really.
  - mustache_kimono 3y ago
    > I don't think the government of a country like the USA or China would deliberately permit their employees to get involved with crap like this. Not the USA, but I can easily imagine this is China. Because, as of right now, this seems like the way China does business. The Chinese/the PLA fund a hacking complex of contractors, and when one gets caught, they simply deny involvement. [0],[1] [0]: https://www.npr.org/2024/02/22/1233178131/leaked-document-trove-shows-a-chinese-hacking-scheme-focused-on-harassing-dissid https://www.npr.org/2024/02/22/1233178131/leaked-document-tr... [1]: https://www.washingtonpost.com/world/2024/02/21/china-hacking-leak-documents-isoon/ https://www.washingtonpost.com/world/2024/02/21/china-hackin...

EasyMark 3y ago

Saying that it's a microcosm for "open source" is a bit hyperbolic. What would be more accurate is saying that it's a microcosm of "there is a certain set of users to be ignored" when writing open source software. One has to learn to develop a thick skin and guilt free ignoring of those who don't contribute anything to a project such as constructive suggestions, bug reports, or actual code.

higeorge13 3y ago

Serious question, but what is your expectations for linux and all these libraries/dependencies after 20-40+ years when most of old school or current devs are retired? How can anyone guarantee reliable continuity by good actors in all these dependencies?

apantel 3y ago
Steve Jobs said that someone needs to be the keeper and maintainer of the vision. I think that’s the key to the making of anything great over the long run: there needs to be a Benevolent Dictator For Life who has the vision, competence, energy, passion, and caring to consistently iterate the thing well. The best case scenario for a project that loses its BDFL(s) is to hold on maybe as decently as Apple has under Tim Cook. They haven’t done anything truly innovative since Jobs died in my opinion — they just figured out more ways to apply the combination of a multitouch display, compact computer, and sensors; but they’ve iterated well on everything they produce and have held on to their lead in terms of hardware, user experience, and ecosystem cohesion. In short: someone very competent has to care a lot about the project for the right reasons, and continue driving it forward. If the old guard is rotating out, someone has to step up. If no one steps up, then leeches of various types will attach themselves to the project and simultaneously milk it for value and kill it.
- mnau 3y ago
  44% of active open source projects have only one maintainer. Out of all maintainers, 60% would describe themselves as unpaid hobbyist. They already are BDFL for themselves or one other person. The problem is a chronic deficiency of maintainers, which is at the root of this attack. Your comment implies there is a wide pool of peoplecwho want to do that, along with mythical "someone". Here is a little story: There was an important job to be done and Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it. Somebody got angry about that because it was Everybody’s job. Everybody thought that Anybody could do it, but Nobody realized that Everybody wouldn’t do it. It ended up that Everybody blamed Somebody when Nobody did what Anybody could have done. - Charles R. Swindoll

rrr_oh_man 3y ago

> This is the way it works. It needs to change. Well, it’s on you to set your boundaries. "Feel free to submit a PR" works wonders.

matheusmoreira 3y ago
Be careful what you wish for. Someone will eventually take you up on that challenge. The responsibility will lie with you when they do make that pull request. Is that something you actually want? I gotta ask because it seems there's lots of people out there who are saying that but don't actually want it to happen. Look I get it, it's a great line, it instantly weeds out anyone who can't code, it even filters out those who can code but don't actually care enough about your project. If you choose that route though, you better be prepared for the day when the fabled PR finally arrives in your inbox. You told them to do it themselves if they wanted it so bad, they went and did it. Even though you know your code base and could probably have implemented it 10 times faster than they did, they labored over it to spare you the trouble instead of begging for the feature in your issue tracker. You asked to see the code, you got the code. Now the buck stops with you. Seriously, if you say this, be ready to engage with whomever does show up with a patch. Unless they send you literal irredeemable garbage, the least they are entitled to is your sincere peer review. Nothing's worse than putting in effort into learning somebody else's project and writing the patch, only to not be given the time of day, or worse. I'd rather have a Linus Torvalds type take a big shit on the patch in public than be ignored like that. Someone who does that is at least treating you like a peer.
- rrr_oh_man 3y ago
  I like (and upvoted) how you replied, but disagree with you. What's the problem, honestly? Telling someone that you don't like / want their code / direction / feature? It just takes a modicum of emotional maturity to do that.

gavinhoward 3y ago

And this is why, for my build system coming out in two days, you need to pay to have me respond to bug reports.

sethammons 3y ago
That would stop a nation state or other nefarious actor because? This isn't spam that works in volume
- gavinhoward 3y ago
  It means I can ignore loud but non-paying users. I also don't take outside contributions.
fl0ki 3y ago
You just reinvented the "it's open source but we charge for support" model which has been so well known for so long that it was a line in the movie Antitrust (2001).

krupan 3y ago

This should make us all reconsider any criticisms we've levied towards Linus Torvalds, Richard Stallman, or any other gruff open source project lead

asylteltine 3y ago
[dead]
account42 3y ago
It should also make us reconsider the motivations of those who push such criticisms.

jrochkind1 3y ago

The multi-entity (sock puppet or not hard to say) social engineering attack that was apparently two years in the making (?), is to me a much more salient story than "users are often mean." Although it's true that the expectation of mean users let the attack fit right in, I guess.

mwcampbell 3y ago
The "users are mean" story is something that we can all do something about, and, honestly, I prefer stories with morals that most of us can actually put into practice.
- jrochkind1 3y ago
  Ah, well, I guess my thought is that we need to figure out what to do about the "take over from a burned out maintainer and then inject malware" attack. It's not obvious what to do about it to me either. Which is why I'm concerned to talk about it. That this attack was run by someone who had been participating in the project for possibly years before making the attack -- is not what i would have expected, and makes it even much harder to defend against. Before I was thinking it was about awareness of "don't just turn over the thing to someone new who just showed up, they might be an attacker." But not that easy in this case. I suppose "try to get users to be less mean, by doing our part by being less mean individually" is arguably one piece of strenghtening defenses to this kind of attack, I guess, ok.
  - transpute 3y ago
    > I suppose "try to get users to be less mean, by doing our part by being less mean individually" is arguably one piece of strenghtening defenses to this kind of attack, I guess, ok. Yes, by denying cover traffic.

lamontcg 3y ago

> ”Progress will not happen until there is new maintainer. … The current maintainer lost interest or doesn’t care to maintain anymore. It is sad to see for a repo like this.” While this might be a sockpuppet deliberately manipulating the maintainer, the tone of this post is all too common, and the real thing that pisses me off is actually that last sentence. The use of the words "sad", "shame" and "pity" have been weaponized by a generation of people to dump negative emotions onto to overworked/understaffed people in issue trackers. If you're using those words to express an opinion in an issue tracker, you're probably being a fucking asshole.

anthk 3y ago

Not an issue of OpenSSH, but some GNU/Linux distros patching and linking OpenSSH against xz.

bshanks 3y ago

Even if other users (or possibly, sock puppets of the attacker) had not been complaining, the original maintainer probably would have began to trust the attacker after enough real contributions.

shp0ngle 3y ago

Stop making software open source, start making it closed source and charge for licenses. That works! Unfortunately it's bad for open source.

cryptonector 3y ago

> This thread is a microcosm of the interactions in Open Source projects. Consumers make demands (some polite, some not-so-polite) of one maintainer (rarely two) that does everything. > Make no mistake. This is the way it works. > It needs to change. How?

pizzafeelsright 3y ago
Deny by default anything from a State that's hostile to American interests.
- fluoridation 3y ago
  How would that help with the problem of people being rude to unpaid OSS maintainers?
  - pizzafeelsright 3y ago
    Rude is subjective and words are not violence. I think the OSS worked as intended.
    - fluoridation 3y ago
      >Rude is subjective The person you replied to accepts the supposition that some users of open source software are rude when they interact with the developers, and asks how this could change. If you don't accept the premise that some users are rude then you can't pretend to answer the question. >words are not violence And? If your coworker reaches into your desk drawer while you're not looking and takes money out of your wallet that's also not violence, yet I imagine you would not think it's acceptable behavior just for that reason.
- shp0ngle 3y ago
  let's add passport (and visa) checks to github

semiquaver 3y ago

At first I thought it was paranoid to suggest that people on the linked thread might be complicit in the attack but then I read this message: > You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs? It genuinely looks like we’re seeing a demonstration of supply chain psyops in retrospect. Amazing how sophisticated and patient this attack was.

jijijijij 3y ago
> It genuinely looks like we’re seeing a demonstration of supply chain psyops in retrospect. Worthwhile noting: It happened in the open, archived for the world to see at any time! The attacker needed to be extra careful not to raise suspicion, both acutely and for accumulated evidence in history. Of course, easy to say "Hindsight is 20/20", but we can probably agree in actual hindsight, we easily see a lot of suspicious acts around the issue. An individual incident may be chance, but as a whole things become clear more easily. So, I think there is a teachable moment here: When something seems just a tiny bit fishy, do a background check, investigate. You may not catch all something, but you probably won't miss everything. Now, looking at the level of sophistication and risk sneaking people and code into open infrastructure, imagine chances of getting caught infiltrating a large company writing closed source binary blobs for drivers, firmware, ...
- shp0ngle 3y ago
  There are not that many suspicious acts tbh. Randos complaining about unmaintained repos and unclosed issues is a constant in FOSS world. Sometimes it's even true, some projects really die and stop actually addressing real issues. But that's just inherent downside of the "bazaar" model. I don't think how we can "treat maintainers better" without going full corporate/without going full "cathedral".
  - transpute 3y ago
    > But that's just inherent downside of the "bazaar" model. I don't think how we can "treat maintainers better" without going full corporate/without going full "cathedral". We now have two decades of arms-race data in OSS projects influenced by ESR's paper [1]. At least one bazaar [2] has operated for centuries. Bazaars can develop decentralized responses to dynamic local threats. [1] http://www.catb.org/~esr/writings/cathedral-bazaar/ http://www.catb.org/~esr/writings/cathedral-bazaar/ [2] https://en.wikipedia.org/wiki/Grand_Bazaar,_Istanbul https://en.wikipedia.org/wiki/Grand_Bazaar,_Istanbul

archi42 3y ago

User Hakkin made an interesting observation: The "consumers" are likely sock puppets controlled by the attacker. I did not independently verify the claim, but it seems reasonable to me. Post with Hakkin's reply: https://news.ycombinator.com/item?id=39869703 https://news.ycombinator.com/item?id=39869703 (But yeah, I've seen some entitled idiots trying to push their demands on FOSS maintainers and being irritated by the harsh reaction...)