14 ms·
Why has software supply chain security exploded?
- gbrindisi 4y agolog4j introduced supply chain security to the wider corporate world where most production run on Java and a lot more people finally realized how complicated it is to do incident response for these kind of situations.
- deleted 4y ago[deleted]
- rcarmo 4y agoi have a personal take on this, which is that people have yet to realize that pulling new shiny dependencies off GitHub (or entire unaudited container images off Docker Hub) and putting them directly into production is not necessarily a good idea…
- spyremeown 4y agoI mean, it's a speed v. security thing. Every engineer (and I don't mean software engineers here) will tell you this is insane. You can't trust your bricks and yet, you're building a cathedral. I know that automating stuff to check for CVEs and vulnerabilities of sorts is getting a lot of traction recently, but I don't think this is really sufficient. Who do you trust? Are we going to see some companies creating a "Security Seal of Approval" for packages? I'm not sure what's the answer here, but certainly it isn't what we have been doing for the last couple years.
- qikInNdOutReply 4y agoYou can trust those who depend upon a lifelihood from you. You can trust who you pay properly. Who entrust the survival of theire family to your continued existence. Whos bread i take, whos song i sing. Thats the supply chain. Turns out there is no free lunch to be had even in open source.
- freeqaz 4y agoThere are a few companies in this space that are trying to do the "Security Seal of Approval" thing to various degrees. Tidelift is one company that has a bunch of "catalogs"[0] of packages. I'm not sure how their package metadata is generated though -- maybe semi-manually? There is also Bytesafe[1] which is supposed to help give you a way to "firewall" yourself from unapproved dependencies. I don't think they sell data though. Just tools. A company based on the Open Source project, packj[2], is Ossilate[3] which is trying to use automated analysis for analyzing 3rd party packages. And another that's in this space is Socket.dev[4] which tries to warn you in your PRs about bad packages. I know about this space because I work on a project[5] that's also related to supply chain security. It's a bit different from all of the above since we're focused on patching known vulns, but the idea of "vetted packages" has crossed my mind before. Are there any other services in this space that I missed? 0: https://tidelift.com/solutions/catalogs https://tidelift.com/solutions/catalogs 1: https://bytesafe.dev/ https://bytesafe.dev/ 2: https://github.com/ossillate-inc/packj https://github.com/ossillate-inc/packj 3: https://ossillate.com/ https://ossillate.com/ 4: https://socket.dev/ https://socket.dev/ 5: https://github.com/lunasec-io/lunasec/ https://github.com/lunasec-io/lunasec/
- selfmodruntime 4y agoThis is only insane in the engineering world because actual human lives are depending on the structural integrity of your cathedral. Even if you face a supply chain attack, the only value lost may be money for most businesses, and not enough that worrying about supply chain attacks is worth it.
- Cymrukicks 4y agoActual human lives depend on software development. Hospitals, Airports, Airplanes, missiles, satellites. If any of these are attacked via supply chain lives will be lost and countries could even collapse if the attack was severe enough.
- AstralStorm 4y agoAlmost neither of which use a lot of dependencies. It's still C and C++ world out there, and everything gets shrink wrapped. It's more likely that you would find a logic bug or a well known security issue in a single library than somehow infest the supply chain of these particular applications. Every piece of off the shelf software is labelled and audited in that world.
- ciupicri 4y agoIsn't this what Linux distributions like Red Hat Enterprise Linux are doing? Providing a secure (among other things) platform on which you can build on?
- staticassertion 4y agoAt least github supports 2FA, signing of commits, SSO, etc. It can be a lot more trustworthy than a package distro.
- have_faith 4y agoOr people do realise the risk and are balancing the risk of speed of dev with the likelihood of something going wrong (and them being blamed for it). Imperfect systems often produce a lot of value, even if it means playing a game of spinning plates to keep it going. It's very possible that the value produced by this system overshadows the lost value from an ocassional black swan event as a result of the systems failure modes.
- mnd999 4y agoSecurity snake oil merchants figured out a new way to make a quick buck.
- qikInNdOutReply 4y agoI run around the office with scissors air gapping your cables, pay me or i shall destroy productivity to the point security is the least of your concerns. Once i get your employees to work around me, violating contract, your problems are "over".
- rswail 4y agoOf course they will, but the actual problem is there. I've been delivering systems where we had to deliver the source environment (ie what is used to build production) into escrow with a lawyer as part of the contract. It effectively required us to prove that what we delivered (both hardware and software) into escrow, would be able to reproduce what was in production, reliably and provably. Reproducible builds are really hard, especially with people encoding CVS/SVN/git tags into the build, compilers and linkers not producing equivalent results between runs (eg caches warm/cold can make a difference) etc etc. It's a real problem and a real issue.
- deleted 4y ago[deleted]
- Rygian 4y agoThere was never a supply "chain" to begin with. If you're looking for a more precise metaphor, think of it as a large pot of slightly overcooked spaghetti and meatballs. Where anyone can throw stuff into the pot.
- ben_w 4y agoAt the top level, that sounds apt. But people have automated the throwing in of stuff, and even when the original stuff was excellent, stuff sources are sometimes bought out by bad actors. And sometimes the stuff has ingredients, where the ingredients are also so managed, so I would say that there is still a chain.
- yosito 4y agoI think you're oversimplifying spaghetti and meatballs. There are supply chains that go into that as well. Nothing in the modern world is built from raw materials. In theory, if it weren't for supply chain security, someone could poison you in your kitchen by going to a chicken farm, feeding toxins to chickens, who lay contaminated eggs which are sold to a factory that makes pasta, which is sold to a grocery store where your coworker bought it to make spaghetti and meatballs that he brought to your potluck. Even I'm over simplifying it. The point is, without supply chain security, anyone already can throw stuff into any pot.
- diamondo25 4y agoHow about all the throubles with hijacked NPM packages? These attacks are still "small", but have had big effects, multiple times.
- _int3_ 4y agoBusiness depends on trust, there is risk involved in doing business. There is always chance somebody will either not pay or not deliver, or deliver lower quality products than ordered (like this is the case with supply chain attack). And there is not much you can do about it. Later you can sue them or not do business with them anymore, but you don't know that in advance.
- perlgeek 4y agoI really wonder why supply chain security wasn't such a big issue before, and why there weren't really many supply-chain attacks on software before. Too few attackers understood it well enough? Or were other attack vectors easier?
- _int3_ 4y agoProgramming style has changed over time to include more dependencies from 3rd party. Look at style of programming from 70s, they would do everything from OS , filesystem to application. Today there are many dependencies. EDIT: It's much like building a tower. Upper floors depends on lower floors. Taller the construction more unstable it gets.
- whatever1 4y agoBecause C does not have package managers. We were copy pasting libraries as .h files in our projects and never updating them.
- taneq 4y agoI still find it mind boggling that businesses have build processes for their core properties that rely on pulling hugs swathes off third party code in from the internet. How can that ever possibly be secure and robust?
- gw99 4y agoIt’s even scarier when you actually sit down and review some of the code you’ve been importing for years… It has resulted in a major thinking shift for me. Mostly initiated by a malicious nodejs package being quietly pulled in.
- jrib 4y agoIs there a language ecosystem you prefer now?
- 4y ago
- bobsmooth 4y agoBecause it was bound to happen eventually.
- chalst 4y agoThe hardwiring of dependencies in a Nixpkg-based deployment seems to be good for generating SBOMs. Cf. https://discourse.nixos.org/t/generating-software-bill-of-materials-from-derivation/14089 https://discourse.nixos.org/t/generating-software-bill-of-ma...
- ciupicri 4y agoAssuming you're using only Nixpkg and not doing pip install, npm install, cargo install and so on like many people suggest and do.
- nautilus12 4y ago
- hannob 4y agoThe answer is "too many dependencies". Package manages made it easy to have dependencies. Every added dependency adds risk. That doesn't mean you should never add dependencies - there are good reasons to do so - but you should weight their pros and cons. Adding dozends, hundreds or thousands of dependencies (as it is common in the npm ecosystem) adds lots of risk. None of that is particularly surprising.
- DrBazza 4y agoI built zellij (rust tmux clone) recently. 193 dependencies that cargo downloaded. When did rust turn into javascript? Are there supply chain attacks in Rust?
- sylvain_kerkour 4y agoYes :) https://kerkour.com/rust-crate-backdoor https://kerkour.com/rust-crate-backdoor
- TAForObvReasons 4y agoFor-profit package managers like NPM touted download and package counts as if they were KPI. Deep dependency trees is the natural result.
- ThePadawan 4y agoVendors had to start checking the "ran supply chain audit" box since investors demand they do. Boom, new industry of snake oil. Not suggesting that the investors have stakes in any companies doing supply chain audits or building automated solutions for it, of course.
- jimrandomh 4y agoIt used to be that adding library dependencies to a project had a lot of fraction, and if you were writing a library yourself, then imposing indirect dependencies on users was something you'd try fairly hard to avoid. Then npm came, along with some ideology about code-reuse, and the friction went away. But the friction was serving an important function: if adding a library is annoying, you'll add a small number of large libraries rather than a large number of small ones, and you avoid getting an ecosystem where major libraries have hundreds of tiny dependencies. This is important because the friction of adding a dependency is only a small portion of its true cost: the main cost is that you're trusting too many different developers and developers' computers.
- ElevenLathe 4y agoThis is a good framing of it. Can we keep the benefits of automatic dependency resolution while adding some other, artificial deterrent? What if NPM hosted packages for free if they have less than five (or whatever number) of transitive dependencies, but above that charged some number per additional one? Even if it were a small amount, this might put significant downward pressure on the size of dependency trees.
- egberts1 4y agoIt all started with glib/compilers/LLVM/Windows toolkit not being able to reproduce a build, byte-for-byte.
- egberts1 4y agoAs a QA can see, the inability to procure a reproducible executable checksum values allowed all sorts of non-validated (and un-validatable) things to creep into production.
- egberts1 4y agoOf course, timestamp is the biggest offender of recreatable (deterministic) build. https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...
- pabs3 4y agoIndeed, tons of other issues prevent deterministic builds though: https://reproducible-builds.org/docs/ https://reproducible-builds.org/docs/
- ciupicri 4y agoWhat difference does it make if the build is compromised in an identical way byte-for-byte?
- dusted 4y agonpm install
- DrBazza 4y agoAt a previous job we were building something significant in the Java stack, and as part of the CI/CD we have a jar checking stage which flagged that an emoji library, that wasn't an emoji library, suddenly was coming in to our build transitively. Presumably somewhere down the line, someone wasn't using a fixed version in gradle or maven.
- ThePhysicist 4y agoThese days it's always hard to say how much of the hype in areas like supply chain security is driven by the problem actually becoming worse and how much is due to VC funded startups with enormous marketing budgets that try to convince everyone of the nearing apocalypse (but no worries, their tools can save us). Looking at integrated vulnerability reporting as found e.g. in npm (and soon Golang, I think) it seems that 99 % of alerts are just false positives, as they affect build dependencies (e.g. the tar library) somewhere down the chain that will never run in production or even touch anything sensitive, and that cannot be easily exploited even by sophisticated actors. In my opinion the "high severity" threat category is thrown around way too lightly by these tools, which leads to everyone panicking. If anything good could come out of this whole trend it might be the realization that open-source software needs better funding to address supply chain issue, my expectation though is that most of the money will go to tool vendors that simple tape over the underlying problems by creating an artificial "security layer" between the maintainers and users.
- UncleMeat 4y agoThe CVE landscape has been broken for ages. In a past life I did a lot of work on a bug bounty program and the amount of "I ran some simple tool that says CVE-XYZ is present in your codebase" reporting we received when the relevant code was either dead, ran in a trusted environment, or ran on trusted input was outrageous. Then people would whine about not getting paid. I do think that organizations should have a process for identifying and triaging bugs in dependencies. But "there are vulns in dependencies" is not itself a compelling narrative because there are vulns in your business' code too. And people sure as hell are getting updates more frequently when using npm or whatever then in the alternative world of "link in some open source code and never update ever for all time." To me, the more interesting problem is the "malicious actor takes over an npm account and distributes malware" that is more unique to the npm ecosystem. This one is hard. On the one hand developers are expensive and a robust library ecosystem is basically a superpower. On the other hand you really need a trust relationship with your software vendors if you want to be confident you are delivering safe code.
- pornel 4y agoI think "npm bad" is a lazy take. • There used to be a focus on attacking endpoints directly, but eventually OSes, browsers, and servers stopped being easy targets, so attackers looked for another way in. • OS (especially Linux) distributors acted as gatekeepers. Getting into a distro requires having some reputation, and the risk of getting caught with a backdoor is a deterrent (although I don't think distros are as much of security barrier as people assume, since maintainers already have too much work to do and their source code review is at best at "LGTM" level). • New tooling gave more visibility into dependencies. Transitive and build-time dependencies in precompiled libraries typically aren't visible, so it's easy to think you have one or two deps, but transitively there are still tons of them: https://wiki.alopex.li/LetsBeRealAboutDependencies https://wiki.alopex.li/LetsBeRealAboutDependencies • Exploits became easier to monetize. Thanks to crypto miners and crypto wallets any exploit is almost a direct payout. Way easier and safer than trying to sell a botnet, adware or even stolen CC numbers, so it lowered barriers to entry to this "business" as well as made it more attractive.
- wikfwikf 4y agoI agree with this. The example of 1000s of packages having a transitive dep on a package which left-pads strings looks bad, however the two obvious alternatives to this are: 1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues. 2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.
- galdosdi 4y ago> 1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues. You sure? You can write it, with no security issues, with acceptable performance for most cases, in the same time than it takes to find the dependency. In fact, you'll save time since you don't have to vet the dependency. Uh, you do like, vet your dependencies right, whenever you add them, right? As well as every time you update to a new version, right? If you're not doing any of that and just adding dependencies willy nilly then I could see how it could seem easier, but this is an illusion. You need a dependency to do more than just left padding to make it worth adding, unless you don't give a crap about stability or security. > 2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities. Yeah this is how languages have traditionally done it, and when done well it is very nice. As you say, it is hard to change the contract because it gets depended on, and you know what? This is a feature, not a bug. Things that are really simple and settled like leftpad don't need to constantly change. This is also why emerging languages try to stay "alpha" without guarantees about breaking changes as long as possible early on, so by the time they have to really freeze the standard library, a lot of experience has helped polish it. A similar model is something like Java, which has done a fantastic job of evolving its standard library over the years while staying very (but, mostly, not excessively... mostly...) backwards compatible. Mostly through the trick of letting users create their own popular libraries and then eventually integrating it with the JEP process once one gets really big. Kind of like a lot of the stuff from Guava, or Jodatime.
- verisimilitudes 4y agoI'll take the advice of this article lacking substance, and complain. Well, companies used to have employees write code, rather than stitch together random garbage written by random dipshits who could be tricked into using loose licenses. That's one cause for concern. > Open source won because everyone worked together. No, ``open source'' has ``won'' because it helped corporations defang Free Software and get gratis labor. > Supply chain security will happen because everyone works together. If you try to do this alone, you will fail. This isn't food, but perfectly uniform applied mathematics. There's no ``working together'' with corporations. Doing it alone is the only reasonable option, which means aggressively reducing the size of this foul shit is necessary. > Thank goodness others had the foresight to create our current SBOM formats. Yeah, MicroSoft totally *<3* ``open source'' guys. > Long ago all the software supply chain work was done by hand. Oh, right, all of this new security theatre is always about trust and reputation, and not trusting those disgusting lone programmers such as me or other silly things; it's always really about doing anything but truly auditing that yucky code.
- oxff 4y agoThere is more money to be earned in exploiting the supply chain.
- m3047 4y agoI quit a gig recently because the CI/CD pipeline had automatic scanning of dependencies for vulns (yay!) and no tangible process for dealing with them. It was up to the dev to "fix" this "defect" before code could be merged. There was no pinning policy. There was no actual security review (where do we use this? are we vulnerable? why do we use this? are there alternatives?). Meanwhile it's running in production. (have we discussed this with our MSP? can I discuss this with our MSP? has our MSP notified us of any issues?) Although we say ignorance is no excuse, having spent a decade in cybersecurity and 20 years before that as a quality-focused consultant, I'd say ignorance lets people get away with a lot. But I know better and I would never be allowed to claim "I vass only followeeeng zee orders!" if the shit hit the fan.
- nokya 4y agoI felt clickbaited after reading the article. It's interesting to read, but if you're actually looking for the answer to the question then it's not in the article.
- mmaney13 4y agoI think it has a lot to do with 2 things: 1. A conscious shift in focus from triaging risks when they occur to stopping threats before they arise. This is a natural next step from contemporary security and disaster response. Threat response and continuity planning which both incorporate plans that respond to threats were once the primary objective of organizations. They are still valid, but a more modern and proactive approach includes mitigating the risk at the source. 2. A forced increase in material spend toward securing dev and devops ecosystems in a time where they are one of the most targeted parts of the organization. One only has to watch the news to see this play out... unfortunately after decades of deployment and intranet security emphasis, hackers have recognized that IP and source code are the best way to get money out of a company, and that both are ironically some of the least protected assets.