25 ms·

My personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away

by EdiX 3y ago

My personal takeaways from this: 1. Source distribution tarballs that contain code different from what's in the source repository are bad, we should move away from them. The other big supply chan attack (event-stream) also took advantage of something similar. 1a. As a consequence of (1) autogenerated artifacts should always be committed. 2. Autogenerated artifacts that everyone pagedowns over during code reviews is a problem. If you have this type of stuff in your repository also have an automatic test that checks that nobody tampered with it (it will also keep you from having stale autogenerated files in your repository). 3. A corollary of (1) and (2) is that autotools is bad and the autotools culture is bad. 4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. 5. In general there's a culture that code reuse is always good, that depending on large libraries for small amounts of functionality is good. This is not true, dependencies are maintenance burden and a security risk, this needs to be weighted against the functionality they bring in. 6. Distro maintainers applying substantial patches to packages is a problem, it creates widely used de facto forks for libraries and applications that do not have real maintainers looking at them. 7. We need to make OSS work from the financial point of view for developers. Liblzma and xz-utils probably have tens of millions of install but a single maintainer with mental health problems. 8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations.

delfinom 3y ago
>4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. They never did. In fact the systemd maintainers are confused on that point and adding documentation on how to implement the simple datagram without libsystemd. >7. We need to make OSS work from the financial point of view for developers. Liblzma and xz-utils probably have tens of millions of install but a single maintainer with mental health problems. Way more than tens of millions. Python, php, ruby and many other languages depend on libxml2, libxml2 uses liblzma. And there's many other dependencies. >8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations. Not any maintainer's job. OSS is provided without warranty. Also indication is "Jia Tan" may have been completely fake as their commit timestamps show even on the same day that their timezone switches from Eastern Europe to Asia. So at the very least, they were playing identity games.
- supposemaybe 3y ago
  I said this days ago, but re timezones - they are meaningless as even GCHQ and NSA etc will place false flags in code which has any kind of risk of exposure. I first learned about those techniques from all the high profile intelligence agency leaks from the USA who were performing those themselves.
  - kuschku 3y ago
    It's not that simple. You can falsify a lot of things, but you can't easily falsify the working hours at which you reply to issues or push commits without taking a lot of care. Especially when DST has to be considered. Sure, the +0800 timestamps are definitely fake. A handful of timestamps that were later scrubbed show +0200 and +0300, though. And all the commits match 9am to 6pm working hours if you interpret them as +0200/+0300. The working hours even shift around correctly with the DST change. The issue is that russia doesn't observe DST anymore. That leaves Bulgaria, Cyprus, Estonia, Finland, Greece, Israel, Latvia, Lebanon, Lithuania, Moldova, Romania and Ukraine. Very few of those have the infosec capabilities needed for something like this. Jia Tan was registered in 2021, but first sprung into action during the buildup to the russian invasion of Ukraine 2022. Jia Tan also used a VPN provider that's headquartered in the US. That only makes sense if they're in a US-aligned country, as using a US VPN would give the US more insight into what you're doing, and only protect you from other countries. Personally, I'd guess that it was Israeli intelligence. But Finland, where the original XZ author lives, is another interesting possibility.
    - supposemaybe 3y ago
      Re the falsifying working hours, wouldn’t these boffins be able to automate Git commits at certain times or even pass instructions to another team who is working the late night shift to post these changes etc. I went to MacDonald’s last night, it is open 24/7, these spy agencies surely aren’t more lazy than minimum wage MacD employees - I am sure they work around the clock. Plus, you have night hawks like me who get more stuck in to a project at 4am and sleep through the day. Israeli intelligence, ah probably. Wouldn’t be surprised. I imagine if it was GCHQ, it wouldn’t have been so noisy and got uncovered like this.
disconnect3d 3y ago
I would personally add: test your build scripts. There are so many bugs like the "added dot to disable landlock" added as part of this action (which can also be typos [0]), not to mention that relying on some tools in autoconf to set feature flags will just disable them if those tools are not present [1]. [0] https://twitter.com/disconnect3d_pl/status/1774496509259645392 https://twitter.com/disconnect3d_pl/status/17744965092596453... [1] https://twitter.com/disconnect3d_pl/status/1774747022362325263 https://twitter.com/disconnect3d_pl/status/17747470223623252...
- josephg 3y ago
  I don’t understand how this is still the best way to test if features are available in C. Can’t the OS / environment provide a “features_available” JSON blob listing all the features on the host system? Is AVX2 available on the cpu? OpenSSL? (And if so, where?) and what about kernel features like io_uring? Doing haphazard feature detection by test compiling random hand written C programs in a giant sometimes autogenerated configure script is an icon of everything wrong with Unix. This hack shows that the haphazard mess of configure isn’t just ugly. It’s also a pathway for malicious people to sneak backdoors into our projects and our computers. It’s time to move on.
  - turol 3y ago
    https://xkcd.com/927/ https://xkcd.com/927/
    - okr 3y ago
      "How to demotivate people and prevent innovation!"
    - josephg 3y ago
      I think that would land harder if configure / automake / autoconf were actually a standard. And not, you know, a bunch of cobbled together shell scripts that generate other shell scripts.
      - 2y ago
Nullabillity 3y ago
> 1a. As a consequence of (1) autogenerated artifacts should always be committed. Or… just have downstream users run autotools as part of the build?
- hk__2 3y ago
  > Or… just have downstream users run autotools as part of the build? See point 3: > 3. A corollary of (1) and (2) is that autotools is bad and the autotools culture is bad.
  - jorams 3y ago
    Whether it's autotools or not is not very relevant to point 1a. I'm also confused why point 1 leads to 1a, and not to the opposite of 1a. Source distribution tarballs should not contain code different from what's in the source repository. They should not contain automatically generated artifacts, since those should not be in the repository, since they are by definition not the source, but output of some kind of build process. Having the automatically generated configure script in the repository would have made it slightly easier to spot the backdoor if anyone took the time to read the committed configure script, but if it's already in the repository most people will just take that for granted, not run whatever process generates it, and not notice that it's not actually the output of said process.
    - sokoloff 3y ago
      That very last point seems like something that's fairly amenable to automation, though. (Then, that automation can be attacked, but that seems like one more [fairly independent] layer that must be bypassed to execute one of these attacks.)
    - simiones 3y ago
      I think the point is that all of the code which will get compiled to produce the final binary should be in the repo, and so any generated code that affects the final binary should be in the repo. The use of autotools or other similar tools, ones that are supposed to generate code on the fly on the final user's machine, make this requirement essentially impossible.
tomohawk 3y ago
libsystemd is too juicy of a target, especially with the code reuse that does not appear to take into account these attack vectors. Perhaps any reuse of libraries in sensitive areas like libsystemd should require a separate copy and more rigorous review? This would allow things like libxv to be 'reused', but the 'safe' versions would require a separate codebase that gets audited updates from the mainline.
- kuschku 3y ago
  libsystemd had already removed the lzma dependencies last week, before the backdoor became public, to reduce external dependencies.
  - xorcist 3y ago
    No they did not. The dependency is still there, it's just being lazy loaded. This would be prevented this particular exploit, which would have needed to take another approach, but at the price of making dependencies invisible and hard to debug. You could no longer have found vulnerable systems by way of ldd. The only solution for the attack surface of systemd is to make the individual components more loosely coupled. There is no reason the same library is responsible for readiness reporting and reading logs. One could even argue that none of those functions have anything to do with the job of init. Readiness can break in a number of ways, robustness is built on health checks.
    - kuschku 3y ago
      Even if it hadn't been loaded by libsystemd, liblzma is also loaded by SELinux, which would have allowed the same vulnerability via a different vector. Personally I think projects like fedora silverblue/kinoite and other container-based OSes are going in the right direction. We need a base OS that's as small as possible so it can be audited, and everything else then needs to live in a container so it doesn't have to be audited but is still secured properly.
      - xorcist 3y ago
heresie-dabord 3y ago
More personal observations: 8. Consumers are naive, yes. But the software industry itself is naive about the security threat. 9. The social exploit is part of the code exploit. 10. The FOSS axiom "More Eyes On The Code" works, but only if the "eyes" are educated. FOSS needs material support from industry. A MSFT engineer caught this exploit, but it still was released to G.A. in Fedora 41, openSUSE, and Kali. 11. The dev toolchain and testing process were never conceived to test for security. (edit: Also see Solarwinds [1] ) = = = [1] _ https://www.wired.com/story/the-untold-story-of-solarwinds-the-boldest-supply-chain-hack-ever/ https://www.wired.com/story/the-untold-story-of-solarwinds-t...
- jbergknoff 3y ago
  > 10. The FOSS axiom "More Eyes On The Code" works, but only if the "eyes" are educated. One thing that could help with this is if somebody points an LLM at all these foundational repositories, prompted with "does this code change introduce any security issues?".
  - Osiris 3y ago
    Not sure why an LLM would be better than existing static analysis tools. Many projects I have worked on run static vulnerability analysis on PRs.
  - kqr 3y ago
    I found the black hat!
- andwoe 3y ago
  > "More Eyes On The Code" works That's the whole problem right there: lack of eyes on the code. If this code was actually maintained by more than one person, there's a high chance one of them would have caught on to it.
binkHN 3y ago
> 4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. I couldn't agree more. Coming from the BSD world, systemd is a shock to the system; it's monstrous and has tendrils everywhere.
- growse 3y ago
  If you actually come from BSD, you'd hopefully recognize a set of different utilities combined to form a holistic system released under a single name. It's not a new idea. Besides, the gpp is incorrect: systemd dependencies are not needed for initialisation notifications.
  - deleted 3y ago
    [deleted]
  - 01HNNWZ0MV43FF 3y ago
    I think there's a low-effort solution to GP: Just split off the notification function for now. There's a dilemma here: Make a huge number of tiny libraries and people complain about left-pad. Make a monolith and this type of attack can happen. If left-pad is more preventable, let's go that way. The fact that C and C++ have tons of overhead in producing a package is their problem to deal with through better tooling.
    - growse 3y ago
      > I think there's a low-effort solution to GP: Just split off the notification function for now. 100% agree that some of the functionality could be decoupled, and either the project should provide independent helper libs or at least do a better job of documenting the interfaces. In this specific case, the notification interface is documented (and there's client implementations in a bunch of languages).
    - to11mtm 3y ago
rcxdude 3y ago
I would more just say autogenerated artifacts should just be autogenerated by the build. Committing them doesn't really solve the problem. This is pretty much just a historical hangover in autotools where it targeted building on platforms where autotools wasn't installed, but it's not really a particularly relevant use-case anymore. (I do agree in general that autotools is bad. Especially on projects where a simple makefile is almost always sufficient and much more debuggable if autotools fails). I don't think libsystemd is a particular problem. Or at least it being linked in only made the job of writing the exploit slightly easier: there's enough services running as root that will pull in a dependency like this that the compromise still exists, it just requires a few more hoops to jump through. And systemd has in fact deliberately made the notification process simple specifically so people can avoid the dependency (if not for security, then simply for ease of building in a way which supports systemd notification but doesn't need anything else). Dependencies are a liability, for sure, but I think a lot of the reaction there is not entirely helpful. At least, the size of the dependency tree in a package manager is only about as good a proxy for the risk as number of lines of code is for software project progress. Dependencies need to be considered, but not just minimised out of hand. There are plenty of risks on the reimplement-it-yourself side. The main thing to consider is how many people and who you are depending on, and who's keeping an eye on them. The latter part is something which is really lacking: the most obvious thing about these OSS vulnerabilites is that basically no-one is really auditing code at all, and if people are, they are not sharing the results. It should in principle be possible to apply the advantages of open-source to that as well, but it's real hard to set up the incentives to do it (anyone starting needs to do a lot to make it worthwhile).
- EdiX 3y ago
  > I would more just say autogenerated artifacts should just be autogenerated by the build. There are practical and philosophical problems with this. From the practical point of view you generally want to make contributing (or even just building) your stuff as low friction as possible and having extra manual build steps (install tools X at version X1.X2.X3, Y at version Y1.Y2 and Z at version Z1.Z2rc2) isn't low friction. Philosophically, you are just shifting the attack vector around, you now need to compromise one of tools X, Y and Z, which are probably less under your control than the artifacts they produce. > And systemd has in fact deliberately made the notification process simple specifically so people can avoid the dependency People say this but I'm skeptical, this is the actual documentation of the protocol: "These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable. If the first character of $NOTIFY_SOCKET is "/" or "@", the string is understood as an AF_UNIX or Linux abstract namespace socket (respectively), and in both cases the datagram is accompanied by the process credentials of the sending service, using SCM_CREDENTIALS. If the string starts with "vsock:" then the string is understood as an AF_VSOCK address, which is useful for hypervisors/VMMs or other processes on the host to receive a notification when a virtual machine has finished booting. Note that in case the hypervisor does not support SOCK_DGRAM over AF_VSOCK, SOCK_SEQPACKET will be used instead. The address should be in the form: "vsock:CID:PORT". Note that unlike other uses of vsock, the CID is mandatory and cannot be "VMADDR_CID_ANY". Note that PID1 will send the VSOCK packets from a privileged port (i.e.: lower than 1024), as an attempt to address concerns that unprivileged processes in the guest might try to send malicious notifications to the host, driving it to make destructive decisions based on them." So technically you have to support unix domain sockets, abstract namespace sockets, whatever SCM_CREDENTIALS is, whatever AF_VSOCK is and the SOCK_SEQPACKET note is completely obscure to me.
kijin 3y ago
To clarify (1), we should not be exchanging tarballs, period. Regardless of whether it's different from what's in the source repository. It's 2024, not 1994. If something masquerading as open-source software is not committed to, and built from, a publicly verifiable version-controlled repository, it might as well not exist.
- professoretc 3y ago
  So all of Fabrice Bellard's projects (https://bellard.org/ https://bellard.org/) should not exist?
  - kijin 3y ago
    Most of his well-known projects have public git repositories that you can clone, inspect, and contribute to.
  - ycombinatrix 3y ago
    if these projects have any value, someone will clone them to a public repository.
    - AshamedCaptain 3y ago
      While introducing minor changes to the build system scripts, you mean?
      - ycombinatrix 3y ago
        i'll take any publicly visible build system over a hidden one
        AshamedCaptain 3y ago
        Nothing here was ever hidden.
        3y ago
gumby 3y ago
> 1a. As a consequence of (1) autogenerated artifacts should always be committed. I philosophically and fundamentally hate this suggestion, but have to agree with it. It's going to make porting harder, but is sadly a cost worth paying. > dependencies are maintenance burden and a security risk, this needs to be weighted against the functionality they bring in Tough call. A major library is more likely to be bug fixed and tuned than something you write (which is a good reason to use them which is what makes them attractive as an attack vector). Getting this right requires taste and experience. The comment says "depending on large libraries for small amounts of functionality [is bad but thought to be good]". What constitutes "small amount" vs large requires experience. Certainly cases of this tip my bias towards re-implement vs re-use.
- Kuinox 3y ago
  Why would it make porting harder ?
  - gumby 3y ago
    Often the autogenerated things are dependent on the machine they are running on.
    - Kuinox 3y ago
      So a big reproducible build issue.
      - gumby 3y ago
        Well it really sucks if you build a system but the autogenerated file is for a different CPU.
Denvercoder9 3y ago
> 8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations. That won't help. There's no evidence that Jia Tan is a real name, or even a real person for that matter. If projects stop accepting contributions from asian-sounding names, the next attack will just use Richard Jones as a name.
- lenerdenator 3y ago
  I think you could interpret this as "you need to know, personally, the party you're handing this off to, and make reasonable judgments as to whether or not they could be easily compromised by bad actors". Like, meeting someone at several dev conferences should be a requirement at the very least.
  - redserk 3y ago
    1) Not everyone (current and potential future maintainers) has the time to go to dev conferences. 2) Simply meeting IRL is a terrible proxy for credibility.
    - MiscIdeaMaker99 3y ago
      What would be better?
      - redserk 3y ago
        It's naive to believe that any form of physical presence means someone isn't going to do something nefarious in the eyes of the project. This problem can only be solved by more skilled eyes on the projects that we rely on. How do we get there? shrug.gif. Anything less is trying to find a cheap and ineffective shortcut in this trust model.
        gunapologist99 3y ago
        You have a good point, but there's also a reason why companies like people to come into work and don't hire remotely as much as they should (or could). There's a reason why interviews often include a meal together. Meeting people IRL is good for building trust, on both sides.
AndyMcConachie 3y ago
[flagged]
- haswell 3y ago
  Let’s step away from the extremist position and there’s an interesting problem underneath. If a company or government organization is hiring engineers who will be responsible for critical code, they will undergo background checks at a minimum to ensure they are not an obvious threat to the organization. This isn’t about “country of origin”, but about the criminal history and ties of the individual regardless of origin. In a government setting, security clearances will be required on top of basic background checks. The FOSS community doesn’t have the tools to do this kind of screening, and arguably the openness of the community is what makes it successful. But the uncomfortable reality is that there are indeed malicious actors, and the community doesn’t currently have a mechanism to proactively identify them and instead relies on discovering the results of their malicious behavior. I don’t claim to know the solution, or if there even is one, but the existence of this comment thread is exhibit A for why we need to be thinking hard about how to proceed as a community. FOSS projects are in the big leagues now, and will continue to come under threat from increasingly sophisticated actors.
  - pepa65 3y ago
    Anyone that maintains a critical piece of code could get compromised by threats and blackmail, regardless of how clean they were.
- retrac 3y ago
  I read the comment as -- we shouldn't hand control of valuable open source projects over to probable operatives of government security services. (Indeed, as many have mentioned, US intelligence might be behind this.) Do you disagree with that? Or do you think OP meant something different? Because if they did it went right over my head.
- swsieber 3y ago
  I read it more as "random people on the internet may be part of some country trying backdoor software because countries are sngaged in cyber warfare, so actually meet the people you want to hand over access to"
deleted 3y ago
[deleted]
dcow 3y ago
Not agreeing or disagreeing, just curious how you draw conclusion (4)? This attack has nothing to do with systemd or its ecosystem etc.
- SAI_Peregrinus 3y ago
  This attack used the fact that several distros patch OpenSSH to link to libsystemd for notifications. Libsystemd links liblzma, and the backdoor checks if it's been linked into OpenSSH's sshd process to run. Without distro maintainers linking libsystemd, xz wouldn't have been a useful target for attacking OpenSSH.
xorcist 3y ago
Re: 1. People keep saying this. We should stop distributing tarballs. It's an argument that completely ignores why we have release artifacts in the first place. A release artifact contains more than just autoconf scripts. There can be many reasons to include binary blobs in a release archive. Game resources, firmware images, test cases. There was today a comment that mpv includes parts of media files generated with proprietary encoders as test cases. That's good, not bad. The well maintained library sqlite is everywhere, and has an excellent test suite. They release not one but two tarballs with every release, for different stages of compilation. It would be trivial to stop doing this, but it would make maintaining packages more work, which does nothing to improve security. The reason Debian builds from curated tarballs are because they are curated by a human, and signed with a well known key. They could certainly build from git instead. But would that improve the situation? Not all projects sign their release tags. And for those that do, it is more likely to be automated. We the collective want changes to be vetted first by the upstream maintainer, then by the package maintainer, and would prefer these entities to be unrelated. This time the process was successfully attacked by a corrupt upstream maintainer, but that does not mean we should do away with upstream maintainers. Several backdoor attempts have been stopped over the years by this arrangement and that process is not something we should throw away without careful consideration. The same improvements we have been talking about for years must continue: We should strive for more reproducible builds. We should strive for lower attack surface and decrease build complexity when possible. We should trust our maintainers, but verify their work.
Karellen 3y ago
> 1a. As a consequence of (1) autogenerated artifacts should always be committed. Why don't object files and binaries count as autogenerated artifacts? Should we commit those to the repo too? Where is the line between an artifact that should be committed, and one that shouldn't be? > 4. Libsystemd is a problem for the ecosystem. libc will dynamically load libnss-* on a lot of platforms, some of which can link to a bunch of other helper libraries. What if the attack had come via one of those 2-or-3-dependencies-removed libraries? libc is big and complicated and most programs only use a tiny fraction of it. Is libc a problem for the ecosystem?
- marcosdumay 3y ago
  > Is libc a problem for the ecosystem? Absolutely yes. And also the size of the kernel. Those two currently have a much better guaranteed quality than systemd, thus systemd is a much more pressing issue. But they don't stop being a problem just because they are not the largest one.
  - Karellen 3y ago
    > [libc and the kernel] currently have a much better guaranteed quality than systemd, What are you basing that on?
    - marcosdumay 3y ago
      Historical behavior. There doesn't exist anything else this could be based on.
      - Karellen 3y ago
        Does systemd have a historical record of its defect rate being significantly greater than that of the kernel/glibc? I just checked Coverity scans, and the most recent defect densities appear to be: kernel : 0.59 glibc : 0.54 systemd: 0.04 which actually looks pretty good for systemd. Is there some other analysis you're basing this off? Or are the current rates atypical, and systemd used to be a lot worse? https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/gnu-c-library-glibc https://scan.coverity.com/projects/gnu-c-library-glibc https://scan.coverity.com/projects/systemd https://scan.coverity.com/projects/systemd
SAI_Peregrinus 3y ago
I'd add a 9: performance differences can indicate code differences. Without the 0.5s startup delay being noticed the backdoor wouldn't have been found. It would be much easier to backdoor low-performance software that takes several seconds to start than something that starts nearly instantly.
benlivengood 3y ago
9. We should move toward formal verification for the trusted core of systems (compilers, kernel, drivers, networking, systemd/rc, and access control). With regard to 1, there are some other practical steps to take. Use deterministic builds and isolate the compilation and linking steps from testing. Every build should emit the hashes of the artifacts it produces and the build system should durably sign them along with the checksum of the git commit it was built from. If there need to be more transformations of the artifacts (packaging, etc.) it should happen as a separate deterministic build. Tests should run on a different machine than the one producing the signed build artifacts. Dropping privileges with SECCOMP for tests might be enough but it's also unlikely to be practical for existing tests that expect a normal environment.
mannyv 3y ago
No system is safe from bad actors. The only way to armor yourself is to have consistent policies. Would this have happened if there were code reviews and testing? Consistency is key. At my workplace we routinely bypass branch protections, but we're only responsible for a few customers.
marcosdumay 3y ago
Well, so no news? But seriously, yes, I think I've seen people dismissing each one of those points. And now we have concrete proof they are real. The fact that somehow an scandal like this didn't happen before due to #1, 2, or 3 is almost incredible... on the meaning that a viable explanation is that somebody is suppressing knowledge somewhere. Point 8 simply isn't going to happen. And that means that if you want secure OSS, you must pay somebody to look around and verify those things. And the problem with that is this means you are now into the software vendor political dump - anybody that gets big doing that is instantaneously untrustworthy. Overall, my point is that we need some actual democratic governance on software. Because it's political by nature, and pushing for anarchy works just as well as with any other political body.
acdha 3y ago
I think your eight point is regrettable but mostly true - I’d soften it to professional relationships, which kind of sucks for anyone trying to get started in the field who doesn’t get a job with someone established, and adds an interesting wrinkle to the RTO discussion since you might “work” with someone for years without necessarily knowing anything about them. It also seems like we need some careful cultural management around trust: enshrine trust-but-verify pervasively to avoid focusing only on, say, Chinese H1-Bs or recent immigrants (whoops, spent all of your time on them and it turns out you missed the Mossad and Bulgarian hackers) and really doubling down on tamper-evidence, which also has the pleasant property of reducing the degree to which targeting OSS developers makes sense. Combining your 7th point with that one, I’ve been wondering whether you could expand what happened with OpenSSL to have some kind of general OSS infrastructure program where everyone would pay to support a team which prioritizes supporting non-marquee projects and especially stuff like modernizing tool chains, auditing, sandboxing, etc. so basically any maintainer of something in the top n dependencies would have a trusted group to ask for help and be able to know that everyone on that team has gone through background checks, etc.
paulmd 3y ago
The other problem is that C’s engineering culture is termites all the way down. A test resource getting linked into a final build is, itself, a problem - the tooling should absolutely make this difficult, and transparent/obvious when it happens. But that’s difficult because C never shed the “pile of bash scripts” approach to build engineering… and fundamentally it’s an uphill battle to engineering a reliable system out of a pile of bash scripts. The oft-discussed problems with undefined behavior, obscure memory/aliasing rules, etc are just the obvious smoke. C is termites all the way down and really shouldn’t be used anymore, it’s just also Too Big To Fail. Like if the world’s most critical infrastructure had been built in PHP.
dralley 3y ago
>4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. This is ridiculous, nobody "encourages" every service to depend on it for initialization notifications, you can implement the logic in 10 lines of code or less.
xign 3y ago
There are a lot of bad to terrible takes here, ranging from hindsight 20/20 to borderline discriminatory: 3. The issue here has more to do with the generated tarball doesn't match source. You (i.e. distro owners) should be able to generate the tarball locally and compare with the generated artifact and compare. Autotools is just a scapegoat. 4. xz is used in a lot of places. Reducing dependencies is good, but trying to somehow say this is all systemd's fault, for depending on liblzma is not understanding the core issue here. The attacker could have found another dependency to social engineer into, or find a way to add dependencies and whatnot. It's very easy to say all these stuff in hindsight. 5. Again, I agree with you on principle that dependencies and complexity is a big issue and I always roll my eyes when people bring in 100's of dependencies, but xz is a pretty reputable project. I really really doubt someone would have raised an issue with adding liblzma or think that the build script would introduce a vulnerability like that. Again, a lot of hindsight talking here, instead of actually looking forward to how something like this could realistically be prevented. Too many dependencies are but it's not suddenly everyone will write their own compression libs. 6. Again, I mean, I don't disagree with you on principle but that is not the lesson from this particular incident. This may be your pet peeve but it wasn't like the integration with libsystemd would have raised anyone's alarm. 8. This is just a thinly veiled way of saying "don't work with anyone of Chinese descent". I don't want to use the R word but you know exactly what I mean. There's no evidence Jia Tan is Chinese anyway, or that this is done by China. We simply don't know right now, and as far as we know they could have used any western sounding name. The core issue here is that the trust was misplaced, and the overworked maintainer didn't try to make sure the other person is a real one (e.g. basic Googling). So what, if you don't work with any Chinese, if someone is called "Ryan Gosling" you automatically trust them? --- I do agree with point 7.

by EdiX 3y ago

delfinom 3y ago

>4. Libsystemd is a problem for the ecosystem. People get dismissed as systemd haters for pointing this out but it's big, complicated, has a lot of dependencies and most programs use a tiny fraction of it. Encouraging every service to depend on it for initialization notifications is insane. They never did. In fact the systemd maintainers are confused on that point and adding documentation on how to implement the simple datagram without libsystemd. >7. We need to make OSS work from the financial point of view for developers. Liblzma and xz-utils probably have tens of millions of install but a single maintainer with mental health problems. Way more than tens of millions. Python, php, ruby and many other languages depend on libxml2, libxml2 uses liblzma. And there's many other dependencies. >8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations. Not any maintainer's job. OSS is provided without warranty. Also indication is "Jia Tan" may have been completely fake as their commit timestamps show even on the same day that their timezone switches from Eastern Europe to Asia. So at the very least, they were playing identity games.

supposemaybe 3y ago
I said this days ago, but re timezones - they are meaningless as even GCHQ and NSA etc will place false flags in code which has any kind of risk of exposure. I first learned about those techniques from all the high profile intelligence agency leaks from the USA who were performing those themselves.
- kuschku 3y ago
  It's not that simple. You can falsify a lot of things, but you can't easily falsify the working hours at which you reply to issues or push commits without taking a lot of care. Especially when DST has to be considered. Sure, the +0800 timestamps are definitely fake. A handful of timestamps that were later scrubbed show +0200 and +0300, though. And all the commits match 9am to 6pm working hours if you interpret them as +0200/+0300. The working hours even shift around correctly with the DST change. The issue is that russia doesn't observe DST anymore. That leaves Bulgaria, Cyprus, Estonia, Finland, Greece, Israel, Latvia, Lebanon, Lithuania, Moldova, Romania and Ukraine. Very few of those have the infosec capabilities needed for something like this. Jia Tan was registered in 2021, but first sprung into action during the buildup to the russian invasion of Ukraine 2022. Jia Tan also used a VPN provider that's headquartered in the US. That only makes sense if they're in a US-aligned country, as using a US VPN would give the US more insight into what you're doing, and only protect you from other countries. Personally, I'd guess that it was Israeli intelligence. But Finland, where the original XZ author lives, is another interesting possibility.
  - supposemaybe 3y ago
    Re the falsifying working hours, wouldn’t these boffins be able to automate Git commits at certain times or even pass instructions to another team who is working the late night shift to post these changes etc. I went to MacDonald’s last night, it is open 24/7, these spy agencies surely aren’t more lazy than minimum wage MacD employees - I am sure they work around the clock. Plus, you have night hawks like me who get more stuck in to a project at 4am and sleep through the day. Israeli intelligence, ah probably. Wouldn’t be surprised. I imagine if it was GCHQ, it wouldn’t have been so noisy and got uncovered like this.

disconnect3d 3y ago

I would personally add: test your build scripts. There are so many bugs like the "added dot to disable landlock" added as part of this action (which can also be typos [0]), not to mention that relying on some tools in autoconf to set feature flags will just disable them if those tools are not present [1]. [0] https://twitter.com/disconnect3d_pl/status/1774496509259645392 https://twitter.com/disconnect3d_pl/status/17744965092596453... [1] https://twitter.com/disconnect3d_pl/status/1774747022362325263 https://twitter.com/disconnect3d_pl/status/17747470223623252...

josephg 3y ago
I don’t understand how this is still the best way to test if features are available in C. Can’t the OS / environment provide a “features_available” JSON blob listing all the features on the host system? Is AVX2 available on the cpu? OpenSSL? (And if so, where?) and what about kernel features like io_uring? Doing haphazard feature detection by test compiling random hand written C programs in a giant sometimes autogenerated configure script is an icon of everything wrong with Unix. This hack shows that the haphazard mess of configure isn’t just ugly. It’s also a pathway for malicious people to sneak backdoors into our projects and our computers. It’s time to move on.
- turol 3y ago
  https://xkcd.com/927/ https://xkcd.com/927/
  - okr 3y ago
    "How to demotivate people and prevent innovation!"
  - josephg 3y ago
    I think that would land harder if configure / automake / autoconf were actually a standard. And not, you know, a bunch of cobbled together shell scripts that generate other shell scripts.
    - 2y ago

Nullabillity 3y ago

> 1a. As a consequence of (1) autogenerated artifacts should always be committed. Or… just have downstream users run autotools as part of the build?

hk__2 3y ago
> Or… just have downstream users run autotools as part of the build? See point 3: > 3. A corollary of (1) and (2) is that autotools is bad and the autotools culture is bad.
- jorams 3y ago
  Whether it's autotools or not is not very relevant to point 1a. I'm also confused why point 1 leads to 1a, and not to the opposite of 1a. Source distribution tarballs should not contain code different from what's in the source repository. They should not contain automatically generated artifacts, since those should not be in the repository, since they are by definition not the source, but output of some kind of build process. Having the automatically generated configure script in the repository would have made it slightly easier to spot the backdoor if anyone took the time to read the committed configure script, but if it's already in the repository most people will just take that for granted, not run whatever process generates it, and not notice that it's not actually the output of said process.
  - sokoloff 3y ago
    That very last point seems like something that's fairly amenable to automation, though. (Then, that automation can be attacked, but that seems like one more [fairly independent] layer that must be bypassed to execute one of these attacks.)
  - simiones 3y ago
    I think the point is that all of the code which will get compiled to produce the final binary should be in the repo, and so any generated code that affects the final binary should be in the repo. The use of autotools or other similar tools, ones that are supposed to generate code on the fly on the final user's machine, make this requirement essentially impossible.

tomohawk 3y ago

libsystemd is too juicy of a target, especially with the code reuse that does not appear to take into account these attack vectors. Perhaps any reuse of libraries in sensitive areas like libsystemd should require a separate copy and more rigorous review? This would allow things like libxv to be 'reused', but the 'safe' versions would require a separate codebase that gets audited updates from the mainline.

kuschku 3y ago
libsystemd had already removed the lzma dependencies last week, before the backdoor became public, to reduce external dependencies.
- xorcist 3y ago
  No they did not. The dependency is still there, it's just being lazy loaded. This would be prevented this particular exploit, which would have needed to take another approach, but at the price of making dependencies invisible and hard to debug. You could no longer have found vulnerable systems by way of ldd. The only solution for the attack surface of systemd is to make the individual components more loosely coupled. There is no reason the same library is responsible for readiness reporting and reading logs. One could even argue that none of those functions have anything to do with the job of init. Readiness can break in a number of ways, robustness is built on health checks.
  - kuschku 3y ago
    Even if it hadn't been loaded by libsystemd, liblzma is also loaded by SELinux, which would have allowed the same vulnerability via a different vector. Personally I think projects like fedora silverblue/kinoite and other container-based OSes are going in the right direction. We need a base OS that's as small as possible so it can be audited, and everything else then needs to live in a container so it doesn't have to be audited but is still secured properly.
    - xorcist 3y ago

heresie-dabord 3y ago

More personal observations: 8. Consumers are naive, yes. But the software industry itself is naive about the security threat. 9. The social exploit is part of the code exploit. 10. The FOSS axiom "More Eyes On The Code" works, but only if the "eyes" are educated. FOSS needs material support from industry. A MSFT engineer caught this exploit, but it still was released to G.A. in Fedora 41, openSUSE, and Kali. 11. The dev toolchain and testing process were never conceived to test for security. (edit: Also see Solarwinds [1] ) = = = [1] _ https://www.wired.com/story/the-untold-story-of-solarwinds-the-boldest-supply-chain-hack-ever/ https://www.wired.com/story/the-untold-story-of-solarwinds-t...

jbergknoff 3y ago
> 10. The FOSS axiom "More Eyes On The Code" works, but only if the "eyes" are educated. One thing that could help with this is if somebody points an LLM at all these foundational repositories, prompted with "does this code change introduce any security issues?".
- Osiris 3y ago
  Not sure why an LLM would be better than existing static analysis tools. Many projects I have worked on run static vulnerability analysis on PRs.
- kqr 3y ago
  I found the black hat!
andwoe 3y ago
> "More Eyes On The Code" works That's the whole problem right there: lack of eyes on the code. If this code was actually maintained by more than one person, there's a high chance one of them would have caught on to it.

binkHN 3y ago

growse 3y ago
If you actually come from BSD, you'd hopefully recognize a set of different utilities combined to form a holistic system released under a single name. It's not a new idea. Besides, the gpp is incorrect: systemd dependencies are not needed for initialisation notifications.
- deleted 3y ago
  [deleted]
- 01HNNWZ0MV43FF 3y ago
  I think there's a low-effort solution to GP: Just split off the notification function for now. There's a dilemma here: Make a huge number of tiny libraries and people complain about left-pad. Make a monolith and this type of attack can happen. If left-pad is more preventable, let's go that way. The fact that C and C++ have tons of overhead in producing a package is their problem to deal with through better tooling.
  - growse 3y ago
    > I think there's a low-effort solution to GP: Just split off the notification function for now. 100% agree that some of the functionality could be decoupled, and either the project should provide independent helper libs or at least do a better job of documenting the interfaces. In this specific case, the notification interface is documented (and there's client implementations in a bunch of languages).
  - to11mtm 3y ago

rcxdude 3y ago

I would more just say autogenerated artifacts should just be autogenerated by the build. Committing them doesn't really solve the problem. This is pretty much just a historical hangover in autotools where it targeted building on platforms where autotools wasn't installed, but it's not really a particularly relevant use-case anymore. (I do agree in general that autotools is bad. Especially on projects where a simple makefile is almost always sufficient and much more debuggable if autotools fails). I don't think libsystemd is a particular problem. Or at least it being linked in only made the job of writing the exploit slightly easier: there's enough services running as root that will pull in a dependency like this that the compromise still exists, it just requires a few more hoops to jump through. And systemd has in fact deliberately made the notification process simple specifically so people can avoid the dependency (if not for security, then simply for ease of building in a way which supports systemd notification but doesn't need anything else). Dependencies are a liability, for sure, but I think a lot of the reaction there is not entirely helpful. At least, the size of the dependency tree in a package manager is only about as good a proxy for the risk as number of lines of code is for software project progress. Dependencies need to be considered, but not just minimised out of hand. There are plenty of risks on the reimplement-it-yourself side. The main thing to consider is how many people and who you are depending on, and who's keeping an eye on them. The latter part is something which is really lacking: the most obvious thing about these OSS vulnerabilites is that basically no-one is really auditing code at all, and if people are, they are not sharing the results. It should in principle be possible to apply the advantages of open-source to that as well, but it's real hard to set up the incentives to do it (anyone starting needs to do a lot to make it worthwhile).

EdiX 3y ago
> I would more just say autogenerated artifacts should just be autogenerated by the build. There are practical and philosophical problems with this. From the practical point of view you generally want to make contributing (or even just building) your stuff as low friction as possible and having extra manual build steps (install tools X at version X1.X2.X3, Y at version Y1.Y2 and Z at version Z1.Z2rc2) isn't low friction. Philosophically, you are just shifting the attack vector around, you now need to compromise one of tools X, Y and Z, which are probably less under your control than the artifacts they produce. > And systemd has in fact deliberately made the notification process simple specifically so people can avoid the dependency People say this but I'm skeptical, this is the actual documentation of the protocol: "These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable. If the first character of $NOTIFY_SOCKET is "/" or "@", the string is understood as an AF_UNIX or Linux abstract namespace socket (respectively), and in both cases the datagram is accompanied by the process credentials of the sending service, using SCM_CREDENTIALS. If the string starts with "vsock:" then the string is understood as an AF_VSOCK address, which is useful for hypervisors/VMMs or other processes on the host to receive a notification when a virtual machine has finished booting. Note that in case the hypervisor does not support SOCK_DGRAM over AF_VSOCK, SOCK_SEQPACKET will be used instead. The address should be in the form: "vsock:CID:PORT". Note that unlike other uses of vsock, the CID is mandatory and cannot be "VMADDR_CID_ANY". Note that PID1 will send the VSOCK packets from a privileged port (i.e.: lower than 1024), as an attempt to address concerns that unprivileged processes in the guest might try to send malicious notifications to the host, driving it to make destructive decisions based on them." So technically you have to support unix domain sockets, abstract namespace sockets, whatever SCM_CREDENTIALS is, whatever AF_VSOCK is and the SOCK_SEQPACKET note is completely obscure to me.

kijin 3y ago

To clarify (1), we should not be exchanging tarballs, period. Regardless of whether it's different from what's in the source repository. It's 2024, not 1994. If something masquerading as open-source software is not committed to, and built from, a publicly verifiable version-controlled repository, it might as well not exist.

professoretc 3y ago
So all of Fabrice Bellard's projects (https://bellard.org/ https://bellard.org/) should not exist?
- kijin 3y ago
  Most of his well-known projects have public git repositories that you can clone, inspect, and contribute to.
- ycombinatrix 3y ago
  if these projects have any value, someone will clone them to a public repository.
  - AshamedCaptain 3y ago
    While introducing minor changes to the build system scripts, you mean?
    - ycombinatrix 3y ago
      i'll take any publicly visible build system over a hidden one
      - AshamedCaptain 3y ago
        Nothing here was ever hidden.
        3y ago

gumby 3y ago

> 1a. As a consequence of (1) autogenerated artifacts should always be committed. I philosophically and fundamentally hate this suggestion, but have to agree with it. It's going to make porting harder, but is sadly a cost worth paying. > dependencies are maintenance burden and a security risk, this needs to be weighted against the functionality they bring in Tough call. A major library is more likely to be bug fixed and tuned than something you write (which is a good reason to use them which is what makes them attractive as an attack vector). Getting this right requires taste and experience. The comment says "depending on large libraries for small amounts of functionality [is bad but thought to be good]". What constitutes "small amount" vs large requires experience. Certainly cases of this tip my bias towards re-implement vs re-use.

Kuinox 3y ago
Why would it make porting harder ?
- gumby 3y ago
  Often the autogenerated things are dependent on the machine they are running on.
  - Kuinox 3y ago
    So a big reproducible build issue.
    - gumby 3y ago
      Well it really sucks if you build a system but the autogenerated file is for a different CPU.

Denvercoder9 3y ago

> 8. This sucks to say, but code reviews and handing off maintainership, at the moment, need to take into account geopolitical considerations. That won't help. There's no evidence that Jia Tan is a real name, or even a real person for that matter. If projects stop accepting contributions from asian-sounding names, the next attack will just use Richard Jones as a name.

lenerdenator 3y ago
I think you could interpret this as "you need to know, personally, the party you're handing this off to, and make reasonable judgments as to whether or not they could be easily compromised by bad actors". Like, meeting someone at several dev conferences should be a requirement at the very least.
- redserk 3y ago
  1) Not everyone (current and potential future maintainers) has the time to go to dev conferences. 2) Simply meeting IRL is a terrible proxy for credibility.
  - MiscIdeaMaker99 3y ago
    What would be better?
    - redserk 3y ago
      It's naive to believe that any form of physical presence means someone isn't going to do something nefarious in the eyes of the project. This problem can only be solved by more skilled eyes on the projects that we rely on. How do we get there? shrug.gif. Anything less is trying to find a cheap and ineffective shortcut in this trust model.
      - gunapologist99 3y ago
        You have a good point, but there's also a reason why companies like people to come into work and don't hire remotely as much as they should (or could). There's a reason why interviews often include a meal together. Meeting people IRL is good for building trust, on both sides.

AndyMcConachie 3y ago

[flagged]

haswell 3y ago
Let’s step away from the extremist position and there’s an interesting problem underneath. If a company or government organization is hiring engineers who will be responsible for critical code, they will undergo background checks at a minimum to ensure they are not an obvious threat to the organization. This isn’t about “country of origin”, but about the criminal history and ties of the individual regardless of origin. In a government setting, security clearances will be required on top of basic background checks. The FOSS community doesn’t have the tools to do this kind of screening, and arguably the openness of the community is what makes it successful. But the uncomfortable reality is that there are indeed malicious actors, and the community doesn’t currently have a mechanism to proactively identify them and instead relies on discovering the results of their malicious behavior. I don’t claim to know the solution, or if there even is one, but the existence of this comment thread is exhibit A for why we need to be thinking hard about how to proceed as a community. FOSS projects are in the big leagues now, and will continue to come under threat from increasingly sophisticated actors.
- pepa65 3y ago
  Anyone that maintains a critical piece of code could get compromised by threats and blackmail, regardless of how clean they were.
retrac 3y ago
I read the comment as -- we shouldn't hand control of valuable open source projects over to probable operatives of government security services. (Indeed, as many have mentioned, US intelligence might be behind this.) Do you disagree with that? Or do you think OP meant something different? Because if they did it went right over my head.
swsieber 3y ago
I read it more as "random people on the internet may be part of some country trying backdoor software because countries are sngaged in cyber warfare, so actually meet the people you want to hand over access to"

deleted 3y ago

[deleted]

dcow 3y ago

Not agreeing or disagreeing, just curious how you draw conclusion (4)? This attack has nothing to do with systemd or its ecosystem etc.

SAI_Peregrinus 3y ago
This attack used the fact that several distros patch OpenSSH to link to libsystemd for notifications. Libsystemd links liblzma, and the backdoor checks if it's been linked into OpenSSH's sshd process to run. Without distro maintainers linking libsystemd, xz wouldn't have been a useful target for attacking OpenSSH.

xorcist 3y ago

Re: 1. People keep saying this. We should stop distributing tarballs. It's an argument that completely ignores why we have release artifacts in the first place. A release artifact contains more than just autoconf scripts. There can be many reasons to include binary blobs in a release archive. Game resources, firmware images, test cases. There was today a comment that mpv includes parts of media files generated with proprietary encoders as test cases. That's good, not bad. The well maintained library sqlite is everywhere, and has an excellent test suite. They release not one but two tarballs with every release, for different stages of compilation. It would be trivial to stop doing this, but it would make maintaining packages more work, which does nothing to improve security. The reason Debian builds from curated tarballs are because they are curated by a human, and signed with a well known key. They could certainly build from git instead. But would that improve the situation? Not all projects sign their release tags. And for those that do, it is more likely to be automated. We the collective want changes to be vetted first by the upstream maintainer, then by the package maintainer, and would prefer these entities to be unrelated. This time the process was successfully attacked by a corrupt upstream maintainer, but that does not mean we should do away with upstream maintainers. Several backdoor attempts have been stopped over the years by this arrangement and that process is not something we should throw away without careful consideration. The same improvements we have been talking about for years must continue: We should strive for more reproducible builds. We should strive for lower attack surface and decrease build complexity when possible. We should trust our maintainers, but verify their work.

Karellen 3y ago

> 1a. As a consequence of (1) autogenerated artifacts should always be committed. Why don't object files and binaries count as autogenerated artifacts? Should we commit those to the repo too? Where is the line between an artifact that should be committed, and one that shouldn't be? > 4. Libsystemd is a problem for the ecosystem. libc will dynamically load libnss-* on a lot of platforms, some of which can link to a bunch of other helper libraries. What if the attack had come via one of those 2-or-3-dependencies-removed libraries? libc is big and complicated and most programs only use a tiny fraction of it. Is libc a problem for the ecosystem?

marcosdumay 3y ago
> Is libc a problem for the ecosystem? Absolutely yes. And also the size of the kernel. Those two currently have a much better guaranteed quality than systemd, thus systemd is a much more pressing issue. But they don't stop being a problem just because they are not the largest one.
- Karellen 3y ago
  > [libc and the kernel] currently have a much better guaranteed quality than systemd, What are you basing that on?
  - marcosdumay 3y ago
    Historical behavior. There doesn't exist anything else this could be based on.
    - Karellen 3y ago
      Does systemd have a historical record of its defect rate being significantly greater than that of the kernel/glibc? I just checked Coverity scans, and the most recent defect densities appear to be: kernel : 0.59 glibc : 0.54 systemd: 0.04 which actually looks pretty good for systemd. Is there some other analysis you're basing this off? Or are the current rates atypical, and systemd used to be a lot worse? https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/linux https://scan.coverity.com/projects/gnu-c-library-glibc https://scan.coverity.com/projects/gnu-c-library-glibc https://scan.coverity.com/projects/systemd https://scan.coverity.com/projects/systemd

SAI_Peregrinus 3y ago

I'd add a 9: performance differences can indicate code differences. Without the 0.5s startup delay being noticed the backdoor wouldn't have been found. It would be much easier to backdoor low-performance software that takes several seconds to start than something that starts nearly instantly.

benlivengood 3y ago

9. We should move toward formal verification for the trusted core of systems (compilers, kernel, drivers, networking, systemd/rc, and access control). With regard to 1, there are some other practical steps to take. Use deterministic builds and isolate the compilation and linking steps from testing. Every build should emit the hashes of the artifacts it produces and the build system should durably sign them along with the checksum of the git commit it was built from. If there need to be more transformations of the artifacts (packaging, etc.) it should happen as a separate deterministic build. Tests should run on a different machine than the one producing the signed build artifacts. Dropping privileges with SECCOMP for tests might be enough but it's also unlikely to be practical for existing tests that expect a normal environment.

mannyv 3y ago

No system is safe from bad actors. The only way to armor yourself is to have consistent policies. Would this have happened if there were code reviews and testing? Consistency is key. At my workplace we routinely bypass branch protections, but we're only responsible for a few customers.

marcosdumay 3y ago

Well, so no news? But seriously, yes, I think I've seen people dismissing each one of those points. And now we have concrete proof they are real. The fact that somehow an scandal like this didn't happen before due to #1, 2, or 3 is almost incredible... on the meaning that a viable explanation is that somebody is suppressing knowledge somewhere. Point 8 simply isn't going to happen. And that means that if you want secure OSS, you must pay somebody to look around and verify those things. And the problem with that is this means you are now into the software vendor political dump - anybody that gets big doing that is instantaneously untrustworthy. Overall, my point is that we need some actual democratic governance on software. Because it's political by nature, and pushing for anarchy works just as well as with any other political body.

acdha 3y ago

I think your eight point is regrettable but mostly true - I’d soften it to professional relationships, which kind of sucks for anyone trying to get started in the field who doesn’t get a job with someone established, and adds an interesting wrinkle to the RTO discussion since you might “work” with someone for years without necessarily knowing anything about them. It also seems like we need some careful cultural management around trust: enshrine trust-but-verify pervasively to avoid focusing only on, say, Chinese H1-Bs or recent immigrants (whoops, spent all of your time on them and it turns out you missed the Mossad and Bulgarian hackers) and really doubling down on tamper-evidence, which also has the pleasant property of reducing the degree to which targeting OSS developers makes sense. Combining your 7th point with that one, I’ve been wondering whether you could expand what happened with OpenSSL to have some kind of general OSS infrastructure program where everyone would pay to support a team which prioritizes supporting non-marquee projects and especially stuff like modernizing tool chains, auditing, sandboxing, etc. so basically any maintainer of something in the top n dependencies would have a trusted group to ask for help and be able to know that everyone on that team has gone through background checks, etc.

paulmd 3y ago

The other problem is that C’s engineering culture is termites all the way down. A test resource getting linked into a final build is, itself, a problem - the tooling should absolutely make this difficult, and transparent/obvious when it happens. But that’s difficult because C never shed the “pile of bash scripts” approach to build engineering… and fundamentally it’s an uphill battle to engineering a reliable system out of a pile of bash scripts. The oft-discussed problems with undefined behavior, obscure memory/aliasing rules, etc are just the obvious smoke. C is termites all the way down and really shouldn’t be used anymore, it’s just also Too Big To Fail. Like if the world’s most critical infrastructure had been built in PHP.

dralley 3y ago

xign 3y ago

There are a lot of bad to terrible takes here, ranging from hindsight 20/20 to borderline discriminatory: 3. The issue here has more to do with the generated tarball doesn't match source. You (i.e. distro owners) should be able to generate the tarball locally and compare with the generated artifact and compare. Autotools is just a scapegoat. 4. xz is used in a lot of places. Reducing dependencies is good, but trying to somehow say this is all systemd's fault, for depending on liblzma is not understanding the core issue here. The attacker could have found another dependency to social engineer into, or find a way to add dependencies and whatnot. It's very easy to say all these stuff in hindsight. 5. Again, I agree with you on principle that dependencies and complexity is a big issue and I always roll my eyes when people bring in 100's of dependencies, but xz is a pretty reputable project. I really really doubt someone would have raised an issue with adding liblzma or think that the build script would introduce a vulnerability like that. Again, a lot of hindsight talking here, instead of actually looking forward to how something like this could realistically be prevented. Too many dependencies are but it's not suddenly everyone will write their own compression libs. 6. Again, I mean, I don't disagree with you on principle but that is not the lesson from this particular incident. This may be your pet peeve but it wasn't like the integration with libsystemd would have raised anyone's alarm. 8. This is just a thinly veiled way of saying "don't work with anyone of Chinese descent". I don't want to use the R word but you know exactly what I mean. There's no evidence Jia Tan is Chinese anyway, or that this is done by China. We simply don't know right now, and as far as we know they could have used any western sounding name. The core issue here is that the trust was misplaced, and the overworked maintainer didn't try to make sure the other person is a real one (e.g. basic Googling). So what, if you don't work with any Chinese, if someone is called "Ryan Gosling" you automatically trust them? --- I do agree with point 7.