14 ms·

As a longtime autotools-hater I would say this justifies my position, but any build system complex enough to be multiplatform is going to be inscrutable enough

As a longtime autotools-hater I would say this justifies my position, but any build system complex enough to be multiplatform is going to be inscrutable enough to let somebody slip something in like this. But it really is a problem that so much software is built with what's essentially become a giant cargo cult-style shell script whose pieces no one person understands all of.

echoangle 3y ago
I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.
- kimixa 3y ago
  I think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python. You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
  - echoangle 3y ago
    The problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.
    - kimixa 3y ago
      Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal. I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation. That's a lot of extra code to review, before you even get to any kind of language differences.
      - eru 3y ago
        > Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. That's why you don't commit auto-generated code. You commit the generating code, and review that. Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.
        lukaslalinsky 3y ago
        But then you have the problem that enabled this backdoor. It's normal to have uncommitted autogenerated unreadable shell code in the tarball. Nobody is going to review test, it was just generated by automake, right? That makes it so easy to sneak in a line that something slightly different. At least with cmake, you have none of this nonsense, people need cmake to build the project, it doesn't try to save users from it by generating a ton of unreadable shell code.
        eru 3y ago
        > Nobody is going to review test, it was just generated by automake, right? Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is. > It's normal to have uncommitted autogenerated unreadable shell code in the tarball. You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
    - owlbite 3y ago
      I think the main issue is auto tools tries to support so many different shells/versions all with their own foibles, so the resulting cross-compatible code looks obfuscated to a modern user. Something built on python won't cover quite as wide a range of (obsolete?) hardware.
      - eru 3y ago
        Python actually covers quite a lot of hardware. Of course, it does that via an autotools nightmare generated configure script. Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software. The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever). The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
    - freedomben 3y ago
      This probably varies widely, because unreadable bash is absolutely not considered normal, nor would pass code review in any of my projects. On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
      - echoangle 3y ago
        What if your application is written in Rust or C? Would you write your build scripts in these languages, too? I would much prefer a simpler scripting language for this. If you’re already using a scripting language as the main language, you don’t necessarily need to pull in another language just for scripts, of course.
        ycombinatrix 3y ago
        build.rs is a thing FYI
        dgsb 3y ago
        or make.go, for some project it makes sense to not add another language for scripting and building tasks. It way easier for every one to have to master multiple language.
        eru 3y ago
        Writing a build script in Rust is fine-ish. Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C. Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C. In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system. However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too. In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too. There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
    - salawat 3y ago
      ...The main problem is some asshat trying to install a backdoor. I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions. I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing. There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!! As a tester, nothing makes me angrier! A pox upon them, and may their treachery be returned upon them 7-fold!
      - eru 3y ago
        > Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing. If your hammer is repurposed shoe, it's fair to blame the tools.
      - Terr_ 3y ago
        > pair it with a comment A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
        salawat 3y ago
        Nothing short of reading the damn code is a defense against malice. I have yet to have any luck in getting people to actually do that.
        cqqxo4zV46cp 3y ago
        “Why can’t humans just be better computers!” is a naive exercise in futility. This is not the answer, and completely ignores the fact that you yourself certainly make just as many mistakes as everyone else.
  - giantrobot 3y ago
    We'll write build scripts in highly obfuscated Perl. There, now no one is happy.
  - sanderjd 3y ago
    Man, this sounds right, but I dunno ... I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written. Just comparing my dotfiles - every bit of which I understood when I wrote them - to the gnarliest sections of a big python app I work on ... I just really feel like at least some of this inscrutability issue can truly be laid at the feet of shell / bash as a language.
    - clnhlzmn 3y ago
      This is totally true and the bash apologists are delusional.
      - lobocinza 3y ago
        It depends on the use case. Bash code can be elegant, Python code can be ugly. I'm not saying those are the average cases but complex code regardless of the language often is ugly even with effort to make it more readable.
        clnhlzmn 3y ago
        Oh I get it. Sometimes bash is the right tool for the job. I think that’s just mostly an unfortunate historical artifact though. It’s hard to argue it’s intuitive or “clean” or ${POSITIVE_ADJECTIVE} in the average case.
      - TeMPOraL 3y ago
        So are some Python advocates, too. The thing that's worse than a bash script made of Perlish line noise, is a piece of "clean code" dead simple Python that's 80% __language__.boilerplate, 20% logic, smeared over 10x the lines because small functions calling small functions are cool. No one has enough working memory to keep track of what's going on there. Instead, your eyes glaze over it and you convince yourself you understand what's going on. Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
  - deanishe 3y ago
    Python makes it relatively hard to write inscrutable code, and more importantly, it's very non-idiomatic, and there would be pushback. WTFing at shell scripts is normal.
    - egorfine 3y ago
      > WTFing at shell scripts is normal "WTFing". This is brilliant.
      - Terr_ 3y ago
        I always think of this now-old comic: https://www.osnews.com/story/19266/wtfsm/ https://www.osnews.com/story/19266/wtfsm/
    - cqqxo4zV46cp 3y ago
      Yep. Python is my day job. If anyone on my team put something up that was as unreadable as a typical bash script, without really good justification, I’d immediately hit “request changes”. This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically. In bash land? “What the hell is [[?” And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language. And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
  - Etheryte 3y ago
    This feels a bit like saying you can run as fast as Usain Bolt. Theoretically many things are possible, but I don't think I've ever seen a readable bash script beyond a trivial oneliner and I've seen a lot of bash in my life. Or to maybe explain from a different perspective, ask a room full of developers to write a bash if-else without looking at the docs and you'll probably come back with more different options than developers in the room. Ask the same for a language such as Python and you'll mostly get one thing.
  - cqqxo4zV46cp 3y ago
    Yeah, nah. I’m all for “every tool has its place”, “religious wars over languages are silly”, etc. Don’t get me wrong. But anyone that claims that all but the 1% most simple bash scripts are “straightforward” and “readable” is, to be blunt, showing their age. The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable. Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
- sunshowers 3y ago
  It's rather easy to monkeypatch Python into doing spooky things. For something like this you really want a language that can't be monkeypatched, like I think Starlark.
  - echoangle 3y ago
    Where are you going to hide your monkey patching though? As long as your code is public, stuff like this is always going to stand out, because no one writes weird magic one liners in python.
    - sunshowers 3y ago
      I'm not a security-oriented professional, but to me a place I could hide this logic is by secretly evaling the contents of some file (like the "corrupt archive" used in xz) somewhere in the build process, hiding it behind a decorator or similar.
      - echoangle 3y ago
        I’m not a security professional either, but that doesn’t sound very plausible to me. If you assume a maintainer who checks every commit added to the codebase, he’s hopefully blocking you the second he sees an eval call in your build script. And even a code audit should find weird stuff like that, if the code is pythonic and simple to read. And if it’s not, it should not be trusted and should be treated as malicious.
        sunshowers 3y ago
        Well, the threat model here is that a maintainer themselves is the saboteur.
        cjbprime 3y ago
        That was true for this project, which was almost orphaned to begin with. We'll run out of nearly-unmaintained critical infrastructure projects sometime. Larger projects with healthier maintenance situations are also at risk, and it's worth reasoning about how a group of honest developers could discover the actions of one malicious developer (with perhaps a malicious reviewer involved too).
- matheusmoreira 3y ago
  Any language can become a turing tarpit if you try hard enough.
  - eru 3y ago
    Some languages make you try harder, some less so. And not all languages are Turing complete in the first place. Not even all useful languages.
- heavyset_go 3y ago
  The things that Bash is good at can wind up obfuscated in Python code due to the verbosity and complexity that it translates to in Python. Bash is great at dealing with files, text, running other programs, job handling, parallelism and IPC. Those things in combination can end up being more complex in Python, which creates more opportunities for obfuscation.
- vmfunction 3y ago
  Cliffy is also pretty good: https://cliffy.io https://cliffy.io And type safe.
- DarkNova6 3y ago
  Yet it is python code which has the highest amount of security vulnerabilities found in public repos. And the most often times that pre-compiled code is commited as well.
- unhammer 3y ago
  > In the beginning of Unix, m4 was created. This has made many people very angry and has been widely regarded as a bad move. autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes https://mbreen.com/m4.html#quotes If autoconf used m4 to generate python scripts, they would also look like https://pyobfusc.com/#winners https://pyobfusc.com/#winners
- AtlasBarfed 3y ago
  Just wait for AI to start pumping spaghetti code everywhere. That's AI phase one Hey, I phase 2 is even better disguised exploit code hiding behind acres of seemingly plausible AI generated code
JonChesterfield 3y ago
Right there with you. It's really tempting to blame this entire thing existing on m4 but that's the trauma talking.
rgmerk 3y ago
Yeah. I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly. That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.
shp0ngle 3y ago
Every time someone explains to me autotools, the individual pieces sort of make sense, yet the result is always this inscrutable unreadable mess. I don't know why.
cyanydeez 3y ago
Just the fact that you have multiple platforms suggests few people will fully understand the entire complex.
jongjong 3y ago
The devil is in the detail and nothing obscures details like complexity. Same reason why I don't like TypeScript in its current form. It's not worth the extra complexity it brings.
dpkirchner 3y ago
Agreed, and I think this leads to the question: how much risk do we face because we want to support such a wide range of platforms that a complex system is required? And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?
- cesarb 3y ago
  > And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation? The project in question contained a compression library, but was not limited to it; it also contained a set of command line tools (the "xz" command and several others). And a modern compression library needs more than just "math and memory allocation"; it also needs threads (to make use of all the available cores), which is historically not portable. You need to detect whether threads are available, and which threading library should be used (pthreads is not always the available option). And not only that, a modern compression library often needs hand-optimized assembly code, with several variants depending on the exact CPU type, the correct one possibly being known only at runtime (and it was exactly in the code to select the correct variant for the current CPU that this backdoor was hidden). And that's before considering that this is a library. Building a dynamic library is something which has a lot of variation between operating systems. You have Windows with its DLLs, MacOS with its frameworks, modern Linux with its ELF stuff, and historically it was even worse (like old a.out-based Linux with its manually pre-allocated base address for every dynamic library in the whole system). So yeah, if you restrict yourself to modern Linux and perhaps a couple of the BSDs, and require the correct CPU type to be selected at compilation time, you could get away with just a couple of pages of simple Makefile declarations. But once you start porting to a more diverse set of systems, you'll see it get more and more complicated. Add cross-compilation to the mix (a non-trivial amount of autotools complexity is there to make cross-compilation work well) and it gets even more complicated.
jonhohle 3y ago
The issue was that built artifacts weren’t immutable during the test phase and/or the test phase wasn’t sandboxed from the built artifacts. The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.
- edflsafoiewq 3y ago
  The malicious .o is extracted from data in binary test files, but the backdoor is inserted entirely in the build phase. Running any test phase is not necessary.
  - rafaelmn 3y ago
    So if you ran build without test files it would fail. I get that this is hindsight thinking - but maybe removing all non essential files when packaging/building libraries reduces the surface area.
salawat 3y ago
Let me reword that for you: >No one has any business saying they know what something does until they've actually read it. Beneath the placid surface of abstraction is the den of the devil.
kjellsbells 3y ago
I'm glad to see I'm not a minority of one here. Looking at autoconf and friends I'm reminded of the dreck that used to fill OpenSSL's code simply because they were trying to account for every eventuality. Autotools feels like the same thing. You end up with a ton of hard to read code (autogenerated bash, not exactly poetry) and that feels very inimical to safety.
livrem 3y ago
I see what everyone is saying about autotools, and I never envied those that maintained the config scripts, but as an end-user I miss the days when installing almost any software was as simple as ./configure && make && make install.
- humanrebar 3y ago
  Running that command is simple, sure. The development workflows when that doesn't work is full stop horrible. Also the basic workflows for the alternative build systems have maybe ten more characters to type. It's not bad.
frankohn 3y ago
I do not agree with your generalisation, the Meson build system is well thought, it has a clear declarative syntax that let you just express what you want in a direct way. The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities. In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need. Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.
- humanrebar 3y ago
  Meson is way better than autotools, but if you're too level to depend on python, you're probably needing to customize your build scripts in the ways you mention. I don't see meson being a silver bullet there. Also, meson's build dependencies (muon, python) are a lot for some of these projects.
  - pknopf 3y ago
    If a projects build system can't depend on python, that let's leave them in the dust, ffs..
    - humanrebar 3y ago
      What if the project is python or a dependency of python?
- Brian_K_White 3y ago
  This just results in 500x worse build.sh on top of meson/ninja/whatever.
humanrebar 3y ago
Autotools need to go away, but most of the underlying use cases that cause build system complexity comes down to dependency management and detection. The utter lack of best practices for those workflows is the root cause of complexity. There is no way to have a mybuild.toml build config on the face of those challenges.

echoangle 3y ago

I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.

kimixa 3y ago
I think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python. You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
- echoangle 3y ago
  The problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.
  - kimixa 3y ago
    Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal. I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation. That's a lot of extra code to review, before you even get to any kind of language differences.
    - eru 3y ago
      > Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly. That's why you don't commit auto-generated code. You commit the generating code, and review that. Same reason we don't stick compiled binaries in our repositories. Binary executables are just auto-generated machine code.
      - lukaslalinsky 3y ago
        But then you have the problem that enabled this backdoor. It's normal to have uncommitted autogenerated unreadable shell code in the tarball. Nobody is going to review test, it was just generated by automake, right? That makes it so easy to sneak in a line that something slightly different. At least with cmake, you have none of this nonsense, people need cmake to build the project, it doesn't try to save users from it by generating a ton of unreadable shell code.
        eru 3y ago
        > Nobody is going to review test, it was just generated by automake, right? Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is. > It's normal to have uncommitted autogenerated unreadable shell code in the tarball. You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
  - owlbite 3y ago
    I think the main issue is auto tools tries to support so many different shells/versions all with their own foibles, so the resulting cross-compatible code looks obfuscated to a modern user. Something built on python won't cover quite as wide a range of (obsolete?) hardware.
    - eru 3y ago
      Python actually covers quite a lot of hardware. Of course, it does that via an autotools nightmare generated configure script. Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software. The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever). The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
  - freedomben 3y ago
    This probably varies widely, because unreadable bash is absolutely not considered normal, nor would pass code review in any of my projects. On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
    - echoangle 3y ago
      What if your application is written in Rust or C? Would you write your build scripts in these languages, too? I would much prefer a simpler scripting language for this. If you’re already using a scripting language as the main language, you don’t necessarily need to pull in another language just for scripts, of course.
      - ycombinatrix 3y ago
        build.rs is a thing FYI
        dgsb 3y ago
        or make.go, for some project it makes sense to not add another language for scripting and building tasks. It way easier for every one to have to master multiple language.
      - eru 3y ago
        Writing a build script in Rust is fine-ish. Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C. Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C. In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system. However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too. In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too. There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
  - salawat 3y ago
    ...The main problem is some asshat trying to install a backdoor. I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions. I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing. There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!! As a tester, nothing makes me angrier! A pox upon them, and may their treachery be returned upon them 7-fold!
    - eru 3y ago
      > Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing. If your hammer is repurposed shoe, it's fair to blame the tools.
    - Terr_ 3y ago
      > pair it with a comment A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
      - salawat 3y ago
        Nothing short of reading the damn code is a defense against malice. I have yet to have any luck in getting people to actually do that.
        cqqxo4zV46cp 3y ago
        “Why can’t humans just be better computers!” is a naive exercise in futility. This is not the answer, and completely ignores the fact that you yourself certainly make just as many mistakes as everyone else.
- giantrobot 3y ago
  We'll write build scripts in highly obfuscated Perl. There, now no one is happy.
- sanderjd 3y ago
  Man, this sounds right, but I dunno ... I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written. Just comparing my dotfiles - every bit of which I understood when I wrote them - to the gnarliest sections of a big python app I work on ... I just really feel like at least some of this inscrutability issue can truly be laid at the feet of shell / bash as a language.
  - clnhlzmn 3y ago
    This is totally true and the bash apologists are delusional.
    - lobocinza 3y ago
      It depends on the use case. Bash code can be elegant, Python code can be ugly. I'm not saying those are the average cases but complex code regardless of the language often is ugly even with effort to make it more readable.
      - clnhlzmn 3y ago
        Oh I get it. Sometimes bash is the right tool for the job. I think that’s just mostly an unfortunate historical artifact though. It’s hard to argue it’s intuitive or “clean” or ${POSITIVE_ADJECTIVE} in the average case.
    - TeMPOraL 3y ago
      So are some Python advocates, too. The thing that's worse than a bash script made of Perlish line noise, is a piece of "clean code" dead simple Python that's 80% __language__.boilerplate, 20% logic, smeared over 10x the lines because small functions calling small functions are cool. No one has enough working memory to keep track of what's going on there. Instead, your eyes glaze over it and you convince yourself you understand what's going on. Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
- deanishe 3y ago
  Python makes it relatively hard to write inscrutable code, and more importantly, it's very non-idiomatic, and there would be pushback. WTFing at shell scripts is normal.
  - egorfine 3y ago
    > WTFing at shell scripts is normal "WTFing". This is brilliant.
    - Terr_ 3y ago
      I always think of this now-old comic: https://www.osnews.com/story/19266/wtfsm/ https://www.osnews.com/story/19266/wtfsm/
  - cqqxo4zV46cp 3y ago
    Yep. Python is my day job. If anyone on my team put something up that was as unreadable as a typical bash script, without really good justification, I’d immediately hit “request changes”. This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically. In bash land? “What the hell is [[?” And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language. And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
- Etheryte 3y ago
  This feels a bit like saying you can run as fast as Usain Bolt. Theoretically many things are possible, but I don't think I've ever seen a readable bash script beyond a trivial oneliner and I've seen a lot of bash in my life. Or to maybe explain from a different perspective, ask a room full of developers to write a bash if-else without looking at the docs and you'll probably come back with more different options than developers in the room. Ask the same for a language such as Python and you'll mostly get one thing.
- cqqxo4zV46cp 3y ago
  Yeah, nah. I’m all for “every tool has its place”, “religious wars over languages are silly”, etc. Don’t get me wrong. But anyone that claims that all but the 1% most simple bash scripts are “straightforward” and “readable” is, to be blunt, showing their age. The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable. Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
sunshowers 3y ago
It's rather easy to monkeypatch Python into doing spooky things. For something like this you really want a language that can't be monkeypatched, like I think Starlark.
- echoangle 3y ago
  Where are you going to hide your monkey patching though? As long as your code is public, stuff like this is always going to stand out, because no one writes weird magic one liners in python.
  - sunshowers 3y ago
    I'm not a security-oriented professional, but to me a place I could hide this logic is by secretly evaling the contents of some file (like the "corrupt archive" used in xz) somewhere in the build process, hiding it behind a decorator or similar.
    - echoangle 3y ago
      I’m not a security professional either, but that doesn’t sound very plausible to me. If you assume a maintainer who checks every commit added to the codebase, he’s hopefully blocking you the second he sees an eval call in your build script. And even a code audit should find weird stuff like that, if the code is pythonic and simple to read. And if it’s not, it should not be trusted and should be treated as malicious.
      - sunshowers 3y ago
        Well, the threat model here is that a maintainer themselves is the saboteur.
        cjbprime 3y ago
        That was true for this project, which was almost orphaned to begin with. We'll run out of nearly-unmaintained critical infrastructure projects sometime. Larger projects with healthier maintenance situations are also at risk, and it's worth reasoning about how a group of honest developers could discover the actions of one malicious developer (with perhaps a malicious reviewer involved too).
matheusmoreira 3y ago
Any language can become a turing tarpit if you try hard enough.
- eru 3y ago
  Some languages make you try harder, some less so. And not all languages are Turing complete in the first place. Not even all useful languages.
heavyset_go 3y ago
The things that Bash is good at can wind up obfuscated in Python code due to the verbosity and complexity that it translates to in Python. Bash is great at dealing with files, text, running other programs, job handling, parallelism and IPC. Those things in combination can end up being more complex in Python, which creates more opportunities for obfuscation.
vmfunction 3y ago
Cliffy is also pretty good: https://cliffy.io https://cliffy.io And type safe.
DarkNova6 3y ago
Yet it is python code which has the highest amount of security vulnerabilities found in public repos. And the most often times that pre-compiled code is commited as well.
unhammer 3y ago
> In the beginning of Unix, m4 was created. This has made many people very angry and has been widely regarded as a bad move. autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes https://mbreen.com/m4.html#quotes If autoconf used m4 to generate python scripts, they would also look like https://pyobfusc.com/#winners https://pyobfusc.com/#winners
AtlasBarfed 3y ago
Just wait for AI to start pumping spaghetti code everywhere. That's AI phase one Hey, I phase 2 is even better disguised exploit code hiding behind acres of seemingly plausible AI generated code

JonChesterfield 3y ago

Right there with you. It's really tempting to blame this entire thing existing on m4 but that's the trauma talking.

rgmerk 3y ago

Yeah. I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly. That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.

shp0ngle 3y ago

Every time someone explains to me autotools, the individual pieces sort of make sense, yet the result is always this inscrutable unreadable mess. I don't know why.

cyanydeez 3y ago

Just the fact that you have multiple platforms suggests few people will fully understand the entire complex.

jongjong 3y ago

The devil is in the detail and nothing obscures details like complexity. Same reason why I don't like TypeScript in its current form. It's not worth the extra complexity it brings.

dpkirchner 3y ago

Agreed, and I think this leads to the question: how much risk do we face because we want to support such a wide range of platforms that a complex system is required? And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?

cesarb 3y ago
> And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation? The project in question contained a compression library, but was not limited to it; it also contained a set of command line tools (the "xz" command and several others). And a modern compression library needs more than just "math and memory allocation"; it also needs threads (to make use of all the available cores), which is historically not portable. You need to detect whether threads are available, and which threading library should be used (pthreads is not always the available option). And not only that, a modern compression library often needs hand-optimized assembly code, with several variants depending on the exact CPU type, the correct one possibly being known only at runtime (and it was exactly in the code to select the correct variant for the current CPU that this backdoor was hidden). And that's before considering that this is a library. Building a dynamic library is something which has a lot of variation between operating systems. You have Windows with its DLLs, MacOS with its frameworks, modern Linux with its ELF stuff, and historically it was even worse (like old a.out-based Linux with its manually pre-allocated base address for every dynamic library in the whole system). So yeah, if you restrict yourself to modern Linux and perhaps a couple of the BSDs, and require the correct CPU type to be selected at compilation time, you could get away with just a couple of pages of simple Makefile declarations. But once you start porting to a more diverse set of systems, you'll see it get more and more complicated. Add cross-compilation to the mix (a non-trivial amount of autotools complexity is there to make cross-compilation work well) and it gets even more complicated.

jonhohle 3y ago

The issue was that built artifacts weren’t immutable during the test phase and/or the test phase wasn’t sandboxed from the built artifacts. The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.

edflsafoiewq 3y ago
The malicious .o is extracted from data in binary test files, but the backdoor is inserted entirely in the build phase. Running any test phase is not necessary.
- rafaelmn 3y ago
  So if you ran build without test files it would fail. I get that this is hindsight thinking - but maybe removing all non essential files when packaging/building libraries reduces the surface area.

salawat 3y ago

Let me reword that for you: >No one has any business saying they know what something does until they've actually read it. Beneath the placid surface of abstraction is the den of the devil.

kjellsbells 3y ago

I'm glad to see I'm not a minority of one here. Looking at autoconf and friends I'm reminded of the dreck that used to fill OpenSSL's code simply because they were trying to account for every eventuality. Autotools feels like the same thing. You end up with a ton of hard to read code (autogenerated bash, not exactly poetry) and that feels very inimical to safety.

livrem 3y ago

I see what everyone is saying about autotools, and I never envied those that maintained the config scripts, but as an end-user I miss the days when installing almost any software was as simple as ./configure && make && make install.

humanrebar 3y ago
Running that command is simple, sure. The development workflows when that doesn't work is full stop horrible. Also the basic workflows for the alternative build systems have maybe ten more characters to type. It's not bad.

frankohn 3y ago

I do not agree with your generalisation, the Meson build system is well thought, it has a clear declarative syntax that let you just express what you want in a direct way. The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities. In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need. Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.

humanrebar 3y ago
Meson is way better than autotools, but if you're too level to depend on python, you're probably needing to customize your build scripts in the ways you mention. I don't see meson being a silver bullet there. Also, meson's build dependencies (muon, python) are a lot for some of these projects.
- pknopf 3y ago
  If a projects build system can't depend on python, that let's leave them in the dust, ffs..
  - humanrebar 3y ago
    What if the project is python or a dependency of python?
Brian_K_White 3y ago
This just results in 500x worse build.sh on top of meson/ninja/whatever.

humanrebar 3y ago

Autotools need to go away, but most of the underlying use cases that cause build system complexity comes down to dependency management and detection. The utter lack of best practices for those workflows is the root cause of complexity. There is no way to have a mybuild.toml build config on the face of those challenges.