7 ms·
One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go! Its time
by psanford 3y ago
One big take away for me is that we should stop tolerating inscrutable code in our systems. M4 has got to go! Inscrutable shell script have got to go!
Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.
- kibwen 3y agoThat's a great first step, but ready your pitchforks for this next take, because the next step is to completely eliminate Turing-complete languages and arbitrary I/O access from standard build systems. 99.9% of all projects have the capability to be built with trivial declarative rulesets.
- azemetre 3y agoCan you explain more in-depth what you mean? I'm also unaware of how you could have declarative rulesets in a non turing-complete language. Sounds like it would be impossible but maybe my thinking is just enclosed and not free.
- ngruhn 3y agoHaven’t used it much but Dhall is a non Turing complete configuration language: https://dhall-lang.org/ https://dhall-lang.org/
- koito17 3y agoIn this case, I think the GP is absolutely right. If you look at the infamous patch with a "hidden" dot, you may think "any C linter should catch that syntax error and immediately draw suspicion." But the thing is, no linter at the moment exists for analyzing strings in a CMakeLists.txt file or M4 macro. Moreover, this isn't something one can reliably do runtime detection for, because there are plenty of legitimate reasons that program could fail to compile, but our tooling does not have a way to clearly communicate the syntax error being the reason for a compilation failure.
- fl7305 3y ago> there are plenty of legitimate reasons that program could fail to compile It's worse than that, these small C files are required to fail unless the target build environment meets some specific requirements.
- jononor 3y agoIt could be required to declare which error should happen, and at what line it should trigger. And the C program should still be syntactically valid and pass linting. One could create a tool which would have a folder of .c files to check with such declarations. And output similar things to as the configure script. Configure scrips tend to be a very slow part of projects, so a new tool that would support parallel compile/run of these checks would also have a considerable speedup. In addition to being more reviewable, verifiable and less error prone.
- fl7305 3y agoAll good things if you want to use the autotools. And other tools too. I've been porting and compiling Unix programs since 5 years before Linux. There was a lot of work needed to port between Unix variants, or even worse, non-Unix variants. So I can see the problem that the autotools solve. But I never spent much time learning the details on how the autotools work. When './configure' has some problem, it is a very steep uphill battle to get any traction on the issue. So I'd be more in the camp of looking for a different solution than autotools.
- jononor 3y agoYeah a lot of the complexity that configure scripts are supposed to solve are massively reduced now. Both because there is better harmonization between platforms, and many checks in a N year old are probably no longer relevant. So a first pass should probably be to eliminate as much checks as possible. But there will almost always be a need for a couple of build time checks/switches - should be supported by some kind of tooling.
- smallmancontrov 3y agoWhat does modern C project management look like? I'm only familiar with Autotools and CMake.
- infamouscow 3y agoRedis. Simple flat directory structure and a Makefile. If you need more, treat it as a sign from God you're doing something wrong.
- hyperman1 3y agoJava's Maven is an interesting case study, as it tried to be this:. A standard project layout, a standard dependency mechanism, pom.xml as standard metadata file, and a standard workflow with standard target(clean/compile/test/deploy). Plugins for what's left. There might have been a time where it worked, but people started to use plugins for all kinds of reasons, quite good ones in most cases. Findbugs, code coverage, source code generation,... Today, a maven project without plugins is rare. Maven brought us 95%, but there is a long tail left to cover.
- kibwen 3y ago> Findbugs, code coverage, source code generation,... For the purpose of this conversation we mostly just care about the use case of someone grabbing the code and wanting to use it in their own project. For this use case, dev tools like findbugs and code coverage can be ignored, so it would suffice to have a version of the build system with plugins completely disabled. Code generation is the thornier one, and we can at least be more principled about it than "run some arbitrary code", and at least it should be trivial to say "this codegen process gets absolutely no I/O access whatsoever; you're a dumb text pipeline". But at the end of the day, we have to Just Say No to things like this. Even if it makes the codebase grodier to check in generated code, if I can't inspect and audit the source code, that's a problem, and arbitrary build-time codegen prevents that. Some trade-offs are worth making.
- hyperman1 3y agoThe xz debacle happened partiallybecause the generated autoconf code was provided. Checking in generated code is not that much better. It's a bit more visible, but not much people will spend their limited time to validate it, as it's not worth it for generated code. xz also had checked in inscrutable test files, and nobody could know it was encrypted malware. I'm not a fan of generated code. It tends to cause misery, being in a no mans land between code and not-code. But it is usefull sometimes, e.g rust generating an API from the opengl XML specs. Sandboxing seems the least worst option, but it will still be uninspected half code that one day ends up in production.
- klysm 3y agountil you have to integrate with the rest of the world sure
- kibwen 3y agoLooking at the state of software security in the rest of the world, this may not be much of a disincentive. At some point we need to knuckle down and admit that times have changed, the context of tools built for the tech of the 80s is no longer applicable, and that we can do better. If that means rewriting the world from scratch, then I guess we better get started sooner rather than later.
- eadmund 3y agoI think that what we need is good sandboxing. All a sandboxed Turing-complete language can do is perform I/O on some restricted area, burn CPU and attempt to escape the sandbox. I would like to see this on the language level, not just on the OS level.
- semi-extrinsic 3y agoI have been thinking the exact same thing, and specifically I would like to try implementing something that works with the Rye python manager. Say I have a directory with a virtualenv and some code that needs some packages from PyPI. I would very much like to sandbox anything that runs in this virtualenv to just disk access inside that directory, and network access only to specifically whitelisted URLs. As a user I should only need to add "sandbox = True" to the pyproject.toml file, and optionally "network_whitelist = [...]". From my cursory looking around, I believe Cloudflare sandbox utils, which are convenience wrappers around systemd Seccomp, might be the best starting point. Edit: or just use Firejail, interesting... You mention sandboxing on the language level, but I don't think it is the way. Apparently sandboxing within Python itself is a particularly nasty rabbit hole that is ultimately unfruitful because of Python's introspection capabilities. You will find many dire warnings on that path.
- cryptonector 3y agoThis is a better take. Though I'm sure people can obfuscate in source code just as much as in build configuration code.
- duped 3y agoWhat would that accomplish? It certainly wouldn't have stopped this attack. > 99.9% of all projects have the capability to be built with trivial declarative rulesets. Only if you forbid bootstrapping, which all projects ultimately rely on at some point in their supply chain.
- kibwen 3y ago> What would that accomplish? It certainly wouldn't have stopped this attack. We could write an entire PhD thesis on the number of dire technical failings that would need to be addressed to stop this attack, so while this alone wouldn't have stopped it, it would have required the actor to come up with another vector of code injection which would have been easier to find. > Only if you forbid bootstrapping Codebases that bootstrap are the 0.1%. Those can be built via `bash build.sh` rather than deceptively hiding a Turing-complete environment behind a declarative one. Even if you need to have these in your trusted computing base somewhere, we can focus auditing resources there, especially once we've reduced the amount of auditing that we need to do on the other 99.9% of codebases now that we've systematically limited the build-time shenanigans that they can get up to.
- duped 3y agoConcretely, what security issues are solved by forcing the build specification language to be Turing incomplete? My guess is the answer is "none." At worst, you're actually creating more holes. The reason autoconf/automake exist and M4 scripts are innocuous in the first place is because the build system uses an underpowered language and developers have to turn to code generation to get around it. If you kneecap the build system's language you're not solving problems. You're creating them. > it would have required the actor to come up with another vector of code injection which would have been easier to find. If make was standardized and could programmatically determine the environment its run under and write full programs then the attack vector wouldn't exist in the first place. > Codebases that bootstrap are the 0.1%. We have different experiences, because ime it's close to 100% especially when you include transitive dependencies. When you care about supply chain security you care about being able to bootstrap from sources for your code and all your dependencies, and it's almost guaranteed that one of your dependencies needs to be bootstrapped.
- Dalewyn 3y ago[flagged]
- Sammi 3y agoJavascript is pretty much guaranteed to be permanent. It is the language of the web. (There's webassembly too, but that doesn't remove js)
- davedx 3y agoNonsensical argument. JavaScript and TypeScript projects are developed with patches of the original source not some compressed artefact. Take your trolling elsewhere
- Dalewyn 3y ago[flagged]
- mardifoufs 3y agoJavascript is almost always executed in sandboxed, unprivileged environments. The issue here is that this type of obfuscation is easy to add in core os libraries. The JavaScript ecosystem, for all the hate that it gets, makes it super easy to sandbox any running code. It doesn't matter if it's minified or obfuscated because you basically have to run unknown, untrusted code everywhere while browsing the web with JavaScript turned on. So the ecosystem and tooling is extremely resilient to most forms of malicious attacks no matter how minified or obfuscated the js you're running is. The complete opposite is true for bash and shell scripting in general
- rurban 3y agoGreat, let the cmake dummies wander off into their own little dreamworld, and keep the professionals dealing with the core stuff.
- tomxor 3y agoThat would be an improvement for sure... but this is not fundamentally a technical problem. Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscated way, that misses the point - once an attacker has gained maintainer-ship you are basically screwed - because they will find a way to insert something, somehow, eventually, no matter how many easy targets for obfuscation are removed. Is the real problem here in handing over substantial trust to an anonymous contributor? If a person has more to lose than maintainership then would this have happened? That someone can weave their way into a project under a pseudonym and eventually gain maintainership without ever risking their real reputation seems to set up quite a risk free target for attackers.
- flockonus 3y agoAgreed it's mainly a social engineering problem, BUT also can be viewed as a technical problem, if a sufficiently advanced fuzzer could catch issues like this. It could also be called an industry problem, where we rely on other's code without running proper checks. This seems to be an emerging realization, with services like socket.dev starting to emerge.
- usefulcat 3y ago> Is the real problem here handing over substantial trust to an anonymous contributor? Unless there's some practical way of comprehensively solving The Real Problem Here, it makes a lot of sense to consider all reasonable mitigations, technical, social or otherwise. > If a person has more to lose than maintainership then would this have happened? I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable.
- tomxor 3y ago> I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable. It's not necessary to accept liability, that's waived in all FOSS licenses. What I'm suggesting would only risk reputation of non-malicious contributors, and from what I've seen, most of the major FOSS contributors and maintainers freely use their real identity or associate it with their pseudonym anyway, since that attribution comes with real life perks. Disallowing anonymous pseudonyms would raise the bar quite a bit and require more effort from attackers to construct or steal plausible looking online identities for each attack, especially when they need to hold up for a long time as with this attack.
- pera 3y agoWhile I do agree that M4 is not great I don't believe any alternative would have prevented this attack: you could try translating that build-to-host file to say python with all the evals and shell oneliners and it would still not be immediately obvious for a distro package maintainer doing a casual review after work: for them it would look just like your average weekend project hacky code. Even if you also translated the oneliners it wouldn't be immediately obvious if you were not suspicious. My point is, you could write similarly obfuscated code in any language.
- ok123456 3y agoWith all the CI tooling and containerization, it seems to be going in the opposite direction.
- patmorgan23 3y agoThat and we need to pay open source maintainers and find new ways to support them. And all code that gets linked into security critical applications/libraries needs to be covered by under some sort of security focused code review. So no patching the compression code that openSSL links to with random junk distribution maintainers.
- orthecreedence 3y agoThis doesn't read as a technical failure to me. This was 99% social engineering. I understand that the build system was used a vector, but eliminating that vector doesn't mean all doors are closed. The attacker took advantage of someone having trouble.
- cryptonector 3y agoWhat's inscrutable code? Was it m4 or sh or the combination of the two? Who will pay for all the rewriting you want done? Or even just for the new frameworks that are "scrutable"? How do we guarantee that the result is not inscrutable to you or others? There is so much knee-jerking in this xz debacle. (And I say this / ask these questions with no love for autoconf/m4/sh.)
- intelVISA 3y agoFunctionality like IFUNC is completely inexcusable outside of dev/debug builds imo. It's rot.
- ncruces 3y agoThat's an improvement, but ultimately not enough, actually. The article hits at this, and you should definitely read it: Being able to send xz patches to the Linux kernel would have been a nice point of leverage for Jia Tan's future work. We're not at trusting trust [1] levels yet, but it would be one step closer. [1]: https://research.swtch.com/nih https://research.swtch.com/nih
- 1letterunixname 3y agoAutotools-based build infra is always crufty, fragile, slow, and fugly. In a lot of cases, it sucks because it's what we have right now without doing the work to replace it with something equally flexible. Build infrastructure should be minimal, standardized, and not subject to endless special, undocumented fragility. cmake, just, conan, meson, and bazel (and forks) exist. But I've still yet to see a proper build system that does feature detection in parallel and concurrently, or supports live, incremental, continuous, cached rebuilding.
- hinkley 3y agoI do think we have enough eyeballs at this point that we should stop entertaining the Dancing Bear in low level libraries and start insisting on crisp, self-explaining code. There are a lot of optimizations pushed into compilers these days, and there are a lot of architectural changes that can make things fast without making them inscrutable. We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).
- l33t7332273 3y ago>start insisting on crisp, self-explaining code That’s the beauty of FOSS: if you don’t like what’s written, you can write your own replacement.
- hinkley 3y agoAnd now we have 14 standards.