13 ms·

So the first step of this huge mess was: a social engineering attack. Attacking a tired, burnt-out open source project developer and peer pressuring him into gi

So the first step of this huge mess was: a social engineering attack. Attacking a tired, burnt-out open source project developer and peer pressuring him into giving more control of the repo to the attacker.

aborsy 3y ago
Enabled by customers who don’t pay or donate.
- grumpyprole 3y ago
  Are these customers or additional attackers who have never posted before and will never post again?
  - kijin 3y ago
    Tired maintainers have no way to distinguish one from the other, that's the problem. Even if we say "no payment, no customer," it won't prevent determined attackers from paying significant amounts of laundered money in order to be treated as customers.
    - Ekaros 3y ago
      Also one thing that is easy to come across at this type of work is money. I mean in the cases where someone is injecting backdoors or vulnerabilities. Might not be for individuals or criminal groups. But once agencies and corporations get involved, the sums are trivial...
- rini17 3y ago
  On the other hand, accepting significant amounts of money causes overhead (accounting, taxation). Plus it reinforces the psychological obligation and it's not fun anymore. Thus many maintainers avoid it.
makapuf 3y ago
Well before that: letting a central lib and project to be maintained by a single tires burned out project dev (see xkcd2347)
fulafel 3y ago
The tiredness or other problems of the developer seems an easy narrative, but did anything actually happen that wouldn't in any understaffed open source project? A contributor shows up and does work for 2 years, I feel most projects would have given the person full project developer status by then.
- orbital-decay 3y ago
  It's not like organizations writing proprietary software are magically immune to sleeper agents either. Social engineering is not a software or tech problem in general. Trust is required to get anything done, and can also be abused to hell and back by a sufficiently motivated actor. But important software needs to be identified and proportionally more scrutinized by multiple independent parties, that's the lesson. Identification is the hard part. You can't easily determine that half of the world relies on this particular piece of software, or that it enables access to desirable targets.
  - b112 3y ago
    This is why OSS can be more secure. How much software has the build scripts, the code, all of it, locked away and hidden behind propriety software? Instead of lots of eyes, just 2 DEVs? Yes, this almost succeeded... but can you imagine how many scenarios where someone such as Andres Freund would have found irregularities, but then.. what? Just had to report it to some webpage's contact page? Without being able to even dig further? Would he have known what was happening, or would it have just ended up as an oddity, with no source code, and with the binary purposefully obscured, and so on? Or... even worse, it's reported directly to the 2 guy team, and the guy who put the back door in... takes the bug report?! From where I sit the sleeper/mole problem exists in companies, and can be far harder to detect.
    - creato 3y ago
      I think there are huge factors that push things both for and against open source here. Yes, you get more eyes and people like Andres Freund. However, if this had been a mole in a company, he wouldn't be able to hide behind a possibly anonymous fake persona and (likely) be immune from any consequences/fallout from this attack. It would be harder to gain entry in the first place, he would have needed a real identity. Background checks may not be a big hurdle, but they're at least something more than signing up for a GitHub account. He also wouldn't have had his posse of anonymous sock puppet accounts to add pressure to the original maintainer. I also think that just being able to ramp up on liblzma and its dependencies to undertake this effort in the first place is a huge head start vs. trying to execute the same attack on a closed source corporate product. At the same time, there are probably lower hanging fruits to attack if you really do get a mole inside a company, in addition to there being fewer eyes/opportunities for the attacks to be discovered as you pointed out. I honestly don't know how all of these factors add up. I expect the argument opposite to yours to be raised (again) in the wake of this incident. I'd personally be hesitant to raise this argument (not that it matters here on HN).
resource_waste 3y ago
I often debate if I should go into the hacking world, best case I get bug bounties, worst case I get rich and I contribute immoral actions. I think its far easier to make $3,000,000 as a hacker than a worker/entrepreneur. Its way easier to find flaws/bugs than to do the entire Capitalism thing correctly. Then I see that half of these major attacks required social engineering.... Maybe being a hacker is significantly easier. I only need to fool 1 person, there are a lot of people, and merit isnt exactly how everyone got to their position. Anyway point being: People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. Most of us pick relatively moral work, so the number of attacks are small. It is also why we really need to treat security on computers like its no stronger than your home's front door lock. There are too many attack vectors to be perfectly safe.
- AndyMcConachie 3y ago
  > People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. +1 Between roughly 1999 and 2013 I was primarily a test engineer for networking switches/routers/telephony products. I found bugs for a living and wrote them up, and in the process I found plenty of security vulnerabilities and wrote them up. Security bugs aren't really that different from other bugs. Yet for some reason we lionize people who find security bugs. Most security issues are simply quality issues. But by calling them security issues we shift the focus away from the software producer creating shit code to an attacker doing something bad. The case with xz is a little different, because we have someone who intentionally added bad code and tried to hide it. But for unintentional bugs that rise to security vulnerabilities it's 99% a QA problem.
  - _kb 3y ago
    QA is focused on testing a product or system against how it should work. Security analysis tests against how it shouldn't. The second of those is a much larger search space. Both are important. The argument around blame shifting is apt. The same case has been made for the usage of the term 'bug' (aka an externality). It's 2024, we don't have moths crawling into relays on our computers. We have implementation faults, invalid designs, unsound architecture, inaccurate documentation, ambiguous requirements, and a myriad of other ways to express how software may be defective. Using those terms hurts, and may even invoke some level of concern from those outside of the engineering org - this is a great reason to embrace them.
arp242 3y ago
In the end you're only pressured as much as you allow yourself to be pressured. "I don't feel like it, if it's important to you then feel free to fork". That's really all that's needed. "I don't feel like it" is all the justification you need. Some guy just made a compression tool, because some people like doing that kind of thing, or because it was useful for him. He didn't ask to be made "critical infrastructure" or to be responsible for the security of sshd or to have some business depend on it, or anything like that. No one even asked him.
- gmerc 3y ago
  When it was created, it was a different time. There was a sense of community around open source, much more tightly nit. And the more socially minded you are, the more vulnerable you are to these kind of attacks.
  - arp242 3y ago
    2007 wasn't that long ago, and these type of maintainership issues aren't new – they were a thing when I was starting out in the early 2000s as well. What changed are the stakes, and also the amount of effort bad actors are willing to spend to mine their cryptoblahblah or whatever. And sure, I understand why people feel a responsibility. And it's fine to take this responsibility too. I'm just saying: there is no need to. The entire point of this Free Software/Open Source is to give people the freedom to do whatever you want with some piece of software, without having to be beholden to the original author. That's pretty much the entire point. Anyone in the world can be a maintainer for xz. By forking it and applying useful patches.
    - Klonoar 3y ago
      > 2007 wasn't that long ago It was almost 20 years ago, and no, I generally agree with the person you're responding to that OSS has changed significantly in these regards since the early-to-middle 2000s. I'm not even disagreeing with the rest of your take, just poking at this idea that time hasn't passed and changed things. Some days I look around our industry and feel like it's nowhere near the one I was working in before.
WesolyKubeczek 3y ago
Here’s the rub: peer pressure counts if done by peers. Users who subscribed to a mailing list only to campaign for a governance change are not peers. Longtime contributors who have earned street cred, can be. (I guess in the case of xz, Jia Tan has earned the cred before going rogue; the one-off “maintainer needs to be replaced” campaigners, however, haven’t.)
wood_spirit 3y ago
Occams razor says that those commentators were not in on it; that pressure is common on all projects, there is no need to think that they were part of an attack. In fact, Occam’s razor says that the malicious code was injected by a compromised account and not a malicious actor who spent years steadily getting into position to attack?
- acdha 3y ago
  I don’t think that’s a good application of Occam. Does it really seem parsimonious to think that someone who shows up with no prior history or subsequent activity is really just a random open source user who cares deeply about new maintainers for a low-level library they’re otherwise silent about?
  - wood_spirit 3y ago
    But the point of the article, which matches my own observations, is that comments pouring guilt onto maintainers is commonplace. The depth of feeling it invokes in the maintainer is likely orders of magnitude more than the depth of caring on the part of the commentator. The normal commentator who complains is probably not malicious, probably not aware of the pain they might cause, is probably just not even thinking of that angle.
    - acdha 3y ago
      Yes, I think that’s what made this attack so effective: that kind of abuse is normalized in much of the tech world so it’s very easy to miss that in this targeted case it was coming from accounts with no prior involvement in the community. I like the open feel we’ve had for the last 3 decades but I do think this will likely mean a lot of projects becoming less open, which is warranted but going to suck for people trying to start a career.
AtlasBarfed 3y ago
Here's the deliciousness: Let's take on face value that it was the Chinese, and that China is communist. I mean, "Kumar" and "Tan"? Maybe it wasn't, but it doesn't matter for my purposes: They took an overworked peon of the capitalist enemy that provides a ... ... do I even need to expound? well it's fun ... ... collectively and idealistically produced common operating system "for the people" ... that is exploited and neglected by the rich and powerful, to a degree that society, not just computers, overall society operates on this operating system, and the profits from that are hoovered up by the powerful. The communists attacked the exploited proletariat to get to the enemy. And they are forcing the enemy capitalists to either pay the proletariat properly (they won't) or continue to be vulnerable to the growing communist power in the far east. That's what this distills so well, within a political/ideological conflict that has now spanned 100 years: capitalism vs communism. To wit, a proper functioning capitalist system to reward market value for produced value would properly pass compensation to this poor soul, and incentivize others to help him. Barring that, a proper functioning government would recognize the public good of this and provide support to the core infrastructure software to enable other private enterprise to produce tax revenue. THOSE AREN'T HAPPENING, so the Communists can attack this with impunity and in perpetuity. Here's the thing folks, we've been living in, as they used to say "uninteresting times". Post-WWII Pax Americana, even the Cold War was basically peace, has been cranking for 80 years now. People, that is coming to an end: - Russia / China are destabilizing demographically and simultaneously becoming militant and totalitarian - Global Warming will ramp up the pressure on populations, food shortages, production - The US will likely retreat to a more regional focus as production is onshored, regionalized (or at least centralized to our hemisphere) There's going to be more state conflict, the Ukraine war is just the beginning. The world stakes are rising.
- saagarjha 3y ago
  > Let's take on face value that it was the Chinese, and that China is communist. Let’s not.

aborsy 3y ago

Enabled by customers who don’t pay or donate.

grumpyprole 3y ago
Are these customers or additional attackers who have never posted before and will never post again?
- kijin 3y ago
  Tired maintainers have no way to distinguish one from the other, that's the problem. Even if we say "no payment, no customer," it won't prevent determined attackers from paying significant amounts of laundered money in order to be treated as customers.
  - Ekaros 3y ago
    Also one thing that is easy to come across at this type of work is money. I mean in the cases where someone is injecting backdoors or vulnerabilities. Might not be for individuals or criminal groups. But once agencies and corporations get involved, the sums are trivial...
rini17 3y ago
On the other hand, accepting significant amounts of money causes overhead (accounting, taxation). Plus it reinforces the psychological obligation and it's not fun anymore. Thus many maintainers avoid it.

makapuf 3y ago

Well before that: letting a central lib and project to be maintained by a single tires burned out project dev (see xkcd2347)

fulafel 3y ago

The tiredness or other problems of the developer seems an easy narrative, but did anything actually happen that wouldn't in any understaffed open source project? A contributor shows up and does work for 2 years, I feel most projects would have given the person full project developer status by then.

orbital-decay 3y ago
It's not like organizations writing proprietary software are magically immune to sleeper agents either. Social engineering is not a software or tech problem in general. Trust is required to get anything done, and can also be abused to hell and back by a sufficiently motivated actor. But important software needs to be identified and proportionally more scrutinized by multiple independent parties, that's the lesson. Identification is the hard part. You can't easily determine that half of the world relies on this particular piece of software, or that it enables access to desirable targets.
- b112 3y ago
  This is why OSS can be more secure. How much software has the build scripts, the code, all of it, locked away and hidden behind propriety software? Instead of lots of eyes, just 2 DEVs? Yes, this almost succeeded... but can you imagine how many scenarios where someone such as Andres Freund would have found irregularities, but then.. what? Just had to report it to some webpage's contact page? Without being able to even dig further? Would he have known what was happening, or would it have just ended up as an oddity, with no source code, and with the binary purposefully obscured, and so on? Or... even worse, it's reported directly to the 2 guy team, and the guy who put the back door in... takes the bug report?! From where I sit the sleeper/mole problem exists in companies, and can be far harder to detect.
  - creato 3y ago
    I think there are huge factors that push things both for and against open source here. Yes, you get more eyes and people like Andres Freund. However, if this had been a mole in a company, he wouldn't be able to hide behind a possibly anonymous fake persona and (likely) be immune from any consequences/fallout from this attack. It would be harder to gain entry in the first place, he would have needed a real identity. Background checks may not be a big hurdle, but they're at least something more than signing up for a GitHub account. He also wouldn't have had his posse of anonymous sock puppet accounts to add pressure to the original maintainer. I also think that just being able to ramp up on liblzma and its dependencies to undertake this effort in the first place is a huge head start vs. trying to execute the same attack on a closed source corporate product. At the same time, there are probably lower hanging fruits to attack if you really do get a mole inside a company, in addition to there being fewer eyes/opportunities for the attacks to be discovered as you pointed out. I honestly don't know how all of these factors add up. I expect the argument opposite to yours to be raised (again) in the wake of this incident. I'd personally be hesitant to raise this argument (not that it matters here on HN).

resource_waste 3y ago

I often debate if I should go into the hacking world, best case I get bug bounties, worst case I get rich and I contribute immoral actions. I think its far easier to make $3,000,000 as a hacker than a worker/entrepreneur. Its way easier to find flaws/bugs than to do the entire Capitalism thing correctly. Then I see that half of these major attacks required social engineering.... Maybe being a hacker is significantly easier. I only need to fool 1 person, there are a lot of people, and merit isnt exactly how everyone got to their position. Anyway point being: People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. Most of us pick relatively moral work, so the number of attacks are small. It is also why we really need to treat security on computers like its no stronger than your home's front door lock. There are too many attack vectors to be perfectly safe.

AndyMcConachie 3y ago
> People are amazed by hacking, they shouldn't be, its relatively easy if you are a mere 10 year programmer. +1 Between roughly 1999 and 2013 I was primarily a test engineer for networking switches/routers/telephony products. I found bugs for a living and wrote them up, and in the process I found plenty of security vulnerabilities and wrote them up. Security bugs aren't really that different from other bugs. Yet for some reason we lionize people who find security bugs. Most security issues are simply quality issues. But by calling them security issues we shift the focus away from the software producer creating shit code to an attacker doing something bad. The case with xz is a little different, because we have someone who intentionally added bad code and tried to hide it. But for unintentional bugs that rise to security vulnerabilities it's 99% a QA problem.
- _kb 3y ago
  QA is focused on testing a product or system against how it should work. Security analysis tests against how it shouldn't. The second of those is a much larger search space. Both are important. The argument around blame shifting is apt. The same case has been made for the usage of the term 'bug' (aka an externality). It's 2024, we don't have moths crawling into relays on our computers. We have implementation faults, invalid designs, unsound architecture, inaccurate documentation, ambiguous requirements, and a myriad of other ways to express how software may be defective. Using those terms hurts, and may even invoke some level of concern from those outside of the engineering org - this is a great reason to embrace them.

arp242 3y ago

In the end you're only pressured as much as you allow yourself to be pressured. "I don't feel like it, if it's important to you then feel free to fork". That's really all that's needed. "I don't feel like it" is all the justification you need. Some guy just made a compression tool, because some people like doing that kind of thing, or because it was useful for him. He didn't ask to be made "critical infrastructure" or to be responsible for the security of sshd or to have some business depend on it, or anything like that. No one even asked him.

gmerc 3y ago
When it was created, it was a different time. There was a sense of community around open source, much more tightly nit. And the more socially minded you are, the more vulnerable you are to these kind of attacks.
- arp242 3y ago
  2007 wasn't that long ago, and these type of maintainership issues aren't new – they were a thing when I was starting out in the early 2000s as well. What changed are the stakes, and also the amount of effort bad actors are willing to spend to mine their cryptoblahblah or whatever. And sure, I understand why people feel a responsibility. And it's fine to take this responsibility too. I'm just saying: there is no need to. The entire point of this Free Software/Open Source is to give people the freedom to do whatever you want with some piece of software, without having to be beholden to the original author. That's pretty much the entire point. Anyone in the world can be a maintainer for xz. By forking it and applying useful patches.
  - Klonoar 3y ago
    > 2007 wasn't that long ago It was almost 20 years ago, and no, I generally agree with the person you're responding to that OSS has changed significantly in these regards since the early-to-middle 2000s. I'm not even disagreeing with the rest of your take, just poking at this idea that time hasn't passed and changed things. Some days I look around our industry and feel like it's nowhere near the one I was working in before.

WesolyKubeczek 3y ago

Here’s the rub: peer pressure counts if done by peers. Users who subscribed to a mailing list only to campaign for a governance change are not peers. Longtime contributors who have earned street cred, can be. (I guess in the case of xz, Jia Tan has earned the cred before going rogue; the one-off “maintainer needs to be replaced” campaigners, however, haven’t.)

wood_spirit 3y ago

Occams razor says that those commentators were not in on it; that pressure is common on all projects, there is no need to think that they were part of an attack. In fact, Occam’s razor says that the malicious code was injected by a compromised account and not a malicious actor who spent years steadily getting into position to attack?

acdha 3y ago
I don’t think that’s a good application of Occam. Does it really seem parsimonious to think that someone who shows up with no prior history or subsequent activity is really just a random open source user who cares deeply about new maintainers for a low-level library they’re otherwise silent about?
- wood_spirit 3y ago
  But the point of the article, which matches my own observations, is that comments pouring guilt onto maintainers is commonplace. The depth of feeling it invokes in the maintainer is likely orders of magnitude more than the depth of caring on the part of the commentator. The normal commentator who complains is probably not malicious, probably not aware of the pain they might cause, is probably just not even thinking of that angle.
  - acdha 3y ago
    Yes, I think that’s what made this attack so effective: that kind of abuse is normalized in much of the tech world so it’s very easy to miss that in this targeted case it was coming from accounts with no prior involvement in the community. I like the open feel we’ve had for the last 3 decades but I do think this will likely mean a lot of projects becoming less open, which is warranted but going to suck for people trying to start a career.

AtlasBarfed 3y ago

Here's the deliciousness: Let's take on face value that it was the Chinese, and that China is communist. I mean, "Kumar" and "Tan"? Maybe it wasn't, but it doesn't matter for my purposes: They took an overworked peon of the capitalist enemy that provides a ... ... do I even need to expound? well it's fun ... ... collectively and idealistically produced common operating system "for the people" ... that is exploited and neglected by the rich and powerful, to a degree that society, not just computers, overall society operates on this operating system, and the profits from that are hoovered up by the powerful. The communists attacked the exploited proletariat to get to the enemy. And they are forcing the enemy capitalists to either pay the proletariat properly (they won't) or continue to be vulnerable to the growing communist power in the far east. That's what this distills so well, within a political/ideological conflict that has now spanned 100 years: capitalism vs communism. To wit, a proper functioning capitalist system to reward market value for produced value would properly pass compensation to this poor soul, and incentivize others to help him. Barring that, a proper functioning government would recognize the public good of this and provide support to the core infrastructure software to enable other private enterprise to produce tax revenue. THOSE AREN'T HAPPENING, so the Communists can attack this with impunity and in perpetuity. Here's the thing folks, we've been living in, as they used to say "uninteresting times". Post-WWII Pax Americana, even the Cold War was basically peace, has been cranking for 80 years now. People, that is coming to an end: - Russia / China are destabilizing demographically and simultaneously becoming militant and totalitarian - Global Warming will ramp up the pressure on populations, food shortages, production - The US will likely retreat to a more regional focus as production is onshored, regionalized (or at least centralized to our hemisphere) There's going to be more state conflict, the Ukraine war is just the beginning. The world stakes are rising.

saagarjha 3y ago
> Let's take on face value that it was the Chinese, and that China is communist. Let’s not.