6 ms·
Fifty Years of Open Source Software Supply Chain Security
- aadhavans 2y agoA very well-written piece. The section on funding open source is as relevant as it's ever been, and I don't think we've learnt much since last year. As the proportion of younger engineers contributing to open-source decreases (a reasonable choice, given the state of the economy), I see only two future possibilities: 1. Big corporations take ownership of key open-source libraries in an effort to continue their development. 2. Said key open-source libraries die, and corporations develop proprietary replacements for their own use. The open source scene remains alive, but with a much smaller influence.
- bluGill 2y agoUnfortunately I have no clue how to get a company to put money into the open source we use. Not just my current company, but any company. I've sometimes been able to get permission to contribute something I build on company time, but often what I really want is someone on the project to spend a year or two maintaining it. Do the boring effort of creating a release. Write that complex feature everyone (including me) wants. In decades past companies you to pay for my license for Visual Studio (I think of a MSDN subscription), clear case, a dozen different issue/work trackers. However as soon as an open source alternative is used I don't know how to get the money that would have been spent to them. Come to think of it I'm maintainer of a couple open source projects that I don't use anymore and I don't normally bother even looking at the project either. Either someone needs to pay me to continue maintaining it (remember I don't find them useful myself so I'm not doing it to scratch an itch), or someone needs to take them over from me - but given xz attacks I'm no longer sure how to hand maintenance over.
- johngossman 2y agoIn my prior career I talked to many companies about open source usage. If you tell them they are running an unsupported database or operating system in production, they will often see the value of buying support. But it is much harder to get them to pay for non-production stuff, especially development tools. And even if you find an enlightened manager, getting budget to pay a maintainer for a project is very difficult to even explain. “We’re paying for contract development? But it’s not one of our products and we’ll have no rights to the software? They’ll fix all the bugs we find, right? Right?” This a hard conversation at most companies, even tech companies.
- ghaff 2y agoDevelopment tools was almost always a tough standalone business even before open source became so prevalent.
- akoboldfrying 2y ago"They’ll fix all the bugs we find, right?" -- that sounds to me like a reasonable requirement on the maintainer, if they are going to be paid a non negligible amount.
- ndiddy 2y agoAt companies where I've worked, all of the money we've put into open source has been in contracting the developer(s) to add a feature we needed to the upstream version. Of course, this means that we didn't really fund ongoing maintenance on anything we used that had all the features we needed.
- RossBencina 2y agoAs an independent maintainer I don't really know where to start trying to organise an ongoing income stream from users to support maintenance. I thought that the idea of a funding manifest to advertise funding requests was a good idea: https://floss.fund/funding-manifest/ https://floss.fund/funding-manifest/ No idea if it works.
- bluGill 2y agoMost open source projects don't really need an income stream. It only becomes an issue when the project is large enough that there is desire for someone to work on it half time or more. Smaller projects can still be done as a hobbyist thing. (the project I "maintain" only needs a few hours of my time per year, but since I no longer use it I can't be bothered - there is a problem for those who still use it). Of course it is hard to say - curl seems like it should be a small project but in fact it is large enough to support someone full time.
- lrvick 2y agoMy company, Distrust, exists to produce, support, and fund our open source security tools. So far our core full time team of 3 gets to spend about half our time consulting/auditing and half our time contributing to our open projects that most of our clients use and depend on. The key is for companies to have visibility into the current funding status of the software they depend on, and relationships with maintainers, so they can offer to fund features or fixes they need instead of being blocked. https://distrust.co https://distrust.co
- throw83848484 2y agoSadly, as OS dev, I see third way: development behind closed doors. With AI and CV reference hunting, number of contributions is higher than ever. Open-source projects are basically spammed, with low quality contributions. Public page is just a liability. I am considering to close public bugzilla, git repo and discussions. I would just take bug reports and patches from very small circle of customers and power users. Everything except release source tarball, and short changelog would be private! Open-source means you get a source code, not free customer and dev support!
- ozim 2y agoI think big corporations will take ownership - well not directly but via paying to foundations and it already is the case. Second thing is there are bunch of things corporations need to use but don't want to develop on their own like SSH. There is already too much internal tooling inside of big corporations that is rotting there and a lot of times it would be much better if they give it out to a foundation - like Apache foundation where projects go to die or limp through.
- pabs3 2y agoThe FOSSjobs wiki has a bunch of resources on this topic: https://github.com/fossjobs/fossjobs/wiki/resources https://github.com/fossjobs/fossjobs/wiki/resources
- mlinksva 2y agoGood article for what it covers, but sadly does not cover isolation/sandboxing/least privilege.
- Alive-in-2025 2y agoYes. The crucial issue to me is the increasing frequency of attacks where some piece of open source gets an update - leading to endless hidden supply chain attacks. I don't see anything that is going to block this from getting worse and worse. It became a pretty common issue that I first heard about with npm or node.js and their variants, maybe because people update software so much there and have lots of dependencies. I don't see a solution. A single program can have huge numbers of dependencies, even c++ or java programs now. It's not new, here's one from 6 years ago on c++ - https://www.trendmicro.com/en_us/research/19/d/analyzing-c-c-runtime-library-code-tampering-in-software-supply-chain-attacks.html https://www.trendmicro.com/en_us/research/19/d/analyzing-c-c... Don't forget log4j - https://www.infoworld.com/article/3850718/developers-apply-these-10-mitigations-first-to-prevent-supply-chain-attacks.html https://www.infoworld.com/article/3850718/developers-apply-t..., points to this recent paper https://arxiv.org/pdf/2503.12192 https://arxiv.org/pdf/2503.12192
- bitwize 2y agoIndeed. In 2020s, if you're not sandboxing each thing, and then sandboxing each library the thing depends on, you're running with way too many opportunities for vulnerability.
- LtWorf 2y agoWell said! How?
- bitwize 2y agoI have no freaking idea. Needless to say I don't think our current operating systems are up to the task of actually being secure. You have to be able to somehow dynamic-link in a library whilst only giving calls into that library certain permissions/capabilities... which I don't think even Windows can do.
- ndiddy 2y ago> The OpenSSH project is careful about not taking on unnecessary dependencies, but Debian was not as careful. That distribution patched sshd to link against libsystemd, which in turn linked against a variety of compression packages, including xz's liblzma. Debian's relaxing of sshd's dependency posture was a key enabler for the attack, as well as the reason its impact was limited to Debian-based systems such as Debian, Ubuntu, and Fedora, avoiding other distributions such as Arch, Gentoo, and NixOS. Does Fedora use Debian's patch set for sshd, or a similar patch set that adds libsystemd? Edit: It looks like Fedora wasn't affected because the backdoor triggered a valgrind test failure, so they shipped it with a flag that disabled the functionality that was backdoored. Seems like they lucked out. https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/YTOGJVBNOSW7FSEE7B35GETS25KFPKBO/ https://lists.fedoraproject.org/archives/list/devel@lists.fe...
- deleted 2y ago[deleted]
- aragilar 2y agoI'm not sure show Fedora is derived from Debian... If I recall correctly, the backdoor was set up to only activate on rpm and deb based systems, so it wouldn't have been trigged on Arch, Gentoo or NixOS, even if they linked systemd to ssh.
- lrvick 2y agoGreat coverage, however it failed to mention code review and artifact signing as well as full source bootstrapping which are fundamental defenses most distros skip. In our distro, Stagex, our threat model assumes at least one maintainer, sysadmin, or computer is compromised at all times. This has resulted in some specific design choices and practices: - 100% deterministic, hermetic, reproducible - full source bootstrapped from 180 bytes of human-auditable machine code - all commits signed by authors - all reviews signed by reviewers - all released artifacts are multi-party reproduced and signed - fully OCI (container) native all the way down "FROM scratch" - All packages easily hash-locked to give downstream software easy determinism as well This all goes well beyond the tactics used in Nix and Guix. As far as we know, Stagex is the only distro designed to strictly distrust maintainers. https://stagex.tools https://stagex.tools
- AstralStorm 2y agoGood step. It doesn't distrust the developers of the software though, so does not fix the biggest hole. Multiparty reproduction does not fix it either, that only distrusts the build system. The bigger the project, the higher the chance something slips through, if even an exploitable bug. Maybe it's the developer themselves being compromised, or their maintainer. Reviews are done on what, you have someone reviewing clang code? Binutils?
- TacticalCoder 2y ago> Reviews are done on what, you have someone reviewing clang code? Binutils? There aren't random developers pushing commits to these codebases: these are used by virtually every Linux distro out there (OK, maybe not the Kubernetes one that ships only 12 binaries, forgot its name). It seems obvious to me that GP is talking about protection against rogue distro maintainers, not fundamental packages being backdoored. You're basically saying: "GP's work is pointless because Linus could insert a backdoor in the Linux kernel". In addition to that determinism and 100% reproducibility brings another gigantic benefit: should a backdoor ever be found in clang or one of the binutils tool, it's going to be 100% reproducible. And that is a big thing: being able to reproduce a backdoor is a godsend for security.
- amiga386 2y agoA lovely article, but one section definitely needs a [citation needed] > (OpenSSL is written in C, so this mistake was incredibly easy to make and miss; in a memory-safe language with proper bounds checking, it would have been nearly impossible.) package main import "fmt" type CmdType int const ( WriteMsg CmdType = iota ReadMsg ) type Cmd struct { t CmdType d []byte l int } var buffer [256]byte var cmds = []Cmd{ Cmd{WriteMsg, []byte("Rain. And a little ice. It's a damn good thing he doesn't know how much I hate his guts."), 88}, Cmd{WriteMsg, []byte("Rain. And a little ice."), 23}, Cmd{ReadMsg, nil, 23}, Cmd{ReadMsg, nil, 88}, // oops! } func main() { for c := range cmds { if cmds[c].t == WriteMsg { copy(buffer[:], cmds[c].d[:cmds[c].l]) } else if cmds[c].t == ReadMsg { fmt.Println(string(buffer[:cmds[c].l])) } } } The heartbleed problem was that user-controlled input could say how long it was, separate from how long it actually was. OpenSSL then copied the (short) thing into a buffer, but returned the (long) thing, thus revealing all sorts of other data it was keeping in the same buffer. It wasn't caught because OpenSSL had built its own buffer/memory management routines on top of the actual ones provided by the language (malloc, memcpy, realloc, free), and all sorts of unsafe manipulations were happening inside one big buffer. That buffer could be in a language with perfect memory safety, the same flaw would still be there.
- deleted 2y ago[deleted]
- neuroelectron 2y agoVery suspicious article. Sounds like the "nothing to see here folks, move along" school of security. Reproducibility is more like a security smell; a symptom you’re doing things right. Determinism is the correct target and subtly different. The focus on supply chain is a distraction, a variant of The “trusting trust” attack Ken Thompson described in 1984 is still among the most elegant and devastating. Infected development toolchains can spread horizontally to “secure” builds. Just because it’s open doesn’t mean anyone’s been watching closely. "50 years of security"? Important pillars of OSS have been touched by thousands of contributors with varying levels of oversight. Many commits predate strong code-signing or provenance tracking. If a compiler was compromised at any point, everything it compiled—including future versions of itself—could carry that compromise forward invisibly. This includes even "cleanroom" rebuilds.
- lrvick 2y agoThe best defense we have against the Trusting Trust attack is full source bootstrapping, now done by two distros: Guix and Stagex.
- AstralStorm 2y agoNo you do not. If you have not actually validated each and every source package your trust is only related to the generated binaries corresponding to the sources you had. The trusting trust attack was deployed against the source code of the compiler, poisoning specific binaries. Do you know if GCC 6.99 or 7.0 doesn't put a backdoor in some specific condition? There's no static or dynamic analysis deployed to enhance this level of trust. The initial attempts are simulated execution like in valgrind, all the sanitizer work, perhaps difference on the functional level beyond the text of the source code where it's too easy to smuggle things through... (Like on an abstracted conditional graph.) We cannot even compare binaries or executables right given differing compiler revisions.
- neuroelectron 2y agoBesides full source boostrapping which could adopt progressive verification of hardware features and assumption of untrusted hardware, integration of Formal Verification into the lowest levels of boostrapping is a must. Bootstap security with the compiler. This won't protect against more complex attacks like RoP or unverified state. For that we need to implement simple artifacts that are verifiable and mapped. Return to more simple return states (pass/error). Do error handling external to the compiled binaries. Automate state mapping and combine with targeted fuzzing. Systemd is a perfect example of this kind of thing, what not to do: internal logs and error states being handled by a web of interdependent systems.
- edoceo 2y agoOne of my struggles is to get docker to lockdown which images it loads. I'd like to only pull from my own blessed registry and it seems Docker wants to always go back to theirs. For other "package" managers (eg: CPAN, Debian) I can point to my own archive and be sure everything I manage down stream gets the blessed bits. I basically have a huge archive/mirror for the supply chain for my perl, PHP, JavaScript, etc. If anyone has pro tips on how to "lock" docker to one registry that would be cool.
- dgl 2y agoDon't use Docker, use podman (which has a registries.conf for this, with many settings). You can then use podman-docker to have command line Docker compatibility. Podman is more secure than Docker too, by default it runs as a user, rather than as root.
- edoceo 2y agoThanks, podman has moved up on my "to eval" list.
- EVa5I7bHFq9mnYK 2y agoRelated; https://news.ycombinator.com/item?id=43617352 https://news.ycombinator.com/item?id=43617352 North Korean IT workers have infiltrated the Fortune 500
- transpute 2y agoFrom Linux Security Summit 2019, a retrospective on mandatory access control and bounding "damage that can be caused by flawed or malicious applications" in Android, iOS, macOS, Linux, FreeBSD and Zephyr, https://static.sched.com/hosted_files/lssna19/e5/LSS2019-Retrospective-16-9.pdf https://static.sched.com/hosted_files/lssna19/e5/LSS2019-Ret... For the past 26 years, the speaker has been engaged in the design, implementation, technology transfer, and application of flexible Mandatory Access Control (MAC). In this talk, he describes the history and lessons learned from this body of work. The background and motivation for MAC is first presented, followed by a discussion of how a flexible MAC architecture was created and matured through a series of research systems. The work to bring this architecture to mainstream systems is then described, along with how the architecture and implementation evolved. The experience with applying this architecture to mobile platforms is examined. The role of MAC in a larger system architecture is reviewed in the context of a secure virtualization system. The state of MAC in mainstream systems is compared before and after our work. Work to bring MAC to emerging operating systems is discussed. video: https://www.youtube.com/watch?v=AKWFbxbsU3o https://www.youtube.com/watch?v=AKWFbxbsU3o
- successful23 2y ago[flagged]
- LeoPanthera 2y agoPlease don't use an LLM to write comments.