7 ms·
I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you
by echoangle 3y ago
I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.
- kimixa 3y agoI think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python. You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
- echoangle 3y agoThe problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.
- kimixa 3y agoLots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal. I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation. That's a lot of extra code to review, before you even get to any kind of language differences.
- eru 3y ago> Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. That's why you don't commit auto-generated code. You commit the generating code, and review that. Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.
- lukaslalinsky 3y agoBut then you have the problem that enabled this backdoor. It's normal to have uncommitted autogenerated unreadable shell code in the tarball. Nobody is going to review test, it was just generated by automake, right? That makes it so easy to sneak in a line that something slightly different. At least with cmake, you have none of this nonsense, people need cmake to build the project, it doesn't try to save users from it by generating a ton of unreadable shell code.
- eru 3y ago> Nobody is going to review test, it was just generated by automake, right? Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is. > It's normal to have uncommitted autogenerated unreadable shell code in the tarball. You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
- owlbite 3y agoI think the main issue is auto tools tries to support so many different shells/versions all with their own foibles, so the resulting cross-compatible code looks obfuscated to a modern user. Something built on python won't cover quite as wide a range of (obsolete?) hardware.
- eru 3y agoPython actually covers quite a lot of hardware. Of course, it does that via an autotools nightmare generated configure script. Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software. The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever). The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
- freedomben 3y agoThis probably varies widely, because unreadable bash is absolutely not considered normal, nor would pass code review in any of my projects. On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
- echoangle 3y agoWhat if your application is written in Rust or C? Would you write your build scripts in these languages, too? I would much prefer a simpler scripting language for this. If you’re already using a scripting language as the main language, you don’t necessarily need to pull in another language just for scripts, of course.
- ycombinatrix 3y agobuild.rs is a thing FYI
- dgsb 3y agoor make.go, for some project it makes sense to not add another language for scripting and building tasks. It way easier for every one to have to master multiple language.
- eru 3y agoWriting a build script in Rust is fine-ish. Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C. Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C. In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system. However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too. In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too. There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
- salawat 3y ago...The main problem is some asshat trying to install a backdoor. I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions. I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing. There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!! As a tester, nothing makes me angrier! A pox upon them, and may their treachery be returned upon them 7-fold!
- eru 3y ago> Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing. If your hammer is repurposed shoe, it's fair to blame the tools.
- Terr_ 3y ago> pair it with a comment A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
- salawat 3y agoNothing short of reading the damn code is a defense against malice. I have yet to have any luck in getting people to actually do that.
- cqqxo4zV46cp 3y ago“Why can’t humans just be better computers!” is a naive exercise in futility. This is not the answer, and completely ignores the fact that you yourself certainly make just as many mistakes as everyone else.
- giantrobot 3y agoWe'll write build scripts in highly obfuscated Perl. There, now no one is happy.
- sanderjd 3y agoMan, this sounds right, but I dunno ... I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written. Just comparing my dotfiles - every bit of which I understood when I wrote them - to the gnarliest sections of a big python app I work on ... I just really feel like at least some of this inscrutability issue can truly be laid at the feet of shell / bash as a language.
- clnhlzmn 3y agoThis is totally true and the bash apologists are delusional.
- lobocinza 3y agoIt depends on the use case. Bash code can be elegant, Python code can be ugly. I'm not saying those are the average cases but complex code regardless of the language often is ugly even with effort to make it more readable.
- clnhlzmn 3y agoOh I get it. Sometimes bash is the right tool for the job. I think that’s just mostly an unfortunate historical artifact though. It’s hard to argue it’s intuitive or “clean” or ${POSITIVE_ADJECTIVE} in the average case.
- TeMPOraL 3y agoSo are some Python advocates, too. The thing that's worse than a bash script made of Perlish line noise, is a piece of "clean code" dead simple Python that's 80% __language__.boilerplate, 20% logic, smeared over 10x the lines because small functions calling small functions are cool. No one has enough working memory to keep track of what's going on there. Instead, your eyes glaze over it and you convince yourself you understand what's going on. Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
- deanishe 3y agoPython makes it relatively hard to write inscrutable code, and more importantly, it's very non-idiomatic, and there would be pushback. WTFing at shell scripts is normal.
- egorfine 3y ago> WTFing at shell scripts is normal "WTFing". This is brilliant.
- Terr_ 3y agoI always think of this now-old comic: https://www.osnews.com/story/19266/wtfsm/ https://www.osnews.com/story/19266/wtfsm/
- cqqxo4zV46cp 3y agoYep. Python is my day job. If anyone on my team put something up that was as unreadable as a typical bash script, without really good justification, I’d immediately hit “request changes”. This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically. In bash land? “What the hell is [[?” And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language. And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
- Etheryte 3y agoThis feels a bit like saying you can run as fast as Usain Bolt. Theoretically many things are possible, but I don't think I've ever seen a readable bash script beyond a trivial oneliner and I've seen a lot of bash in my life. Or to maybe explain from a different perspective, ask a room full of developers to write a bash if-else without looking at the docs and you'll probably come back with more different options than developers in the room. Ask the same for a language such as Python and you'll mostly get one thing.
- cqqxo4zV46cp 3y agoYeah, nah. I’m all for “every tool has its place”, “religious wars over languages are silly”, etc. Don’t get me wrong. But anyone that claims that all but the 1% most simple bash scripts are “straightforward” and “readable” is, to be blunt, showing their age. The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable. Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
- sunshowers 3y agoIt's rather easy to monkeypatch Python into doing spooky things. For something like this you really want a language that can't be monkeypatched, like I think Starlark.
- echoangle 3y agoWhere are you going to hide your monkey patching though? As long as your code is public, stuff like this is always going to stand out, because no one writes weird magic one liners in python.
- sunshowers 3y agoI'm not a security-oriented professional, but to me a place I could hide this logic is by secretly evaling the contents of some file (like the "corrupt archive" used in xz) somewhere in the build process, hiding it behind a decorator or similar.
- echoangle 3y agoI’m not a security professional either, but that doesn’t sound very plausible to me. If you assume a maintainer who checks every commit added to the codebase, he’s hopefully blocking you the second he sees an eval call in your build script. And even a code audit should find weird stuff like that, if the code is pythonic and simple to read. And if it’s not, it should not be trusted and should be treated as malicious.
- sunshowers 3y agoWell, the threat model here is that a maintainer themselves is the saboteur.
- cjbprime 3y agoThat was true for this project, which was almost orphaned to begin with. We'll run out of nearly-unmaintained critical infrastructure projects sometime. Larger projects with healthier maintenance situations are also at risk, and it's worth reasoning about how a group of honest developers could discover the actions of one malicious developer (with perhaps a malicious reviewer involved too).
- matheusmoreira 3y agoAny language can become a turing tarpit if you try hard enough.
- eru 3y agoSome languages make you try harder, some less so. And not all languages are Turing complete in the first place. Not even all useful languages.
- heavyset_go 3y agoThe things that Bash is good at can wind up obfuscated in Python code due to the verbosity and complexity that it translates to in Python. Bash is great at dealing with files, text, running other programs, job handling, parallelism and IPC. Those things in combination can end up being more complex in Python, which creates more opportunities for obfuscation.
- vmfunction 3y agoCliffy is also pretty good: https://cliffy.io https://cliffy.io And type safe.
- DarkNova6 3y agoYet it is python code which has the highest amount of security vulnerabilities found in public repos. And the most often times that pre-compiled code is commited as well.
- unhammer 3y ago> In the beginning of Unix, m4 was created. This has made many people very angry and has been widely regarded as a bad move. autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes https://mbreen.com/m4.html#quotes If autoconf used m4 to generate python scripts, they would also look like https://pyobfusc.com/#winners https://pyobfusc.com/#winners
- AtlasBarfed 3y agoJust wait for AI to start pumping spaghetti code everywhere. That's AI phase one Hey, I phase 2 is even better disguised exploit code hiding behind acres of seemingly plausible AI generated code