4 ms·

I don't know enough about C and complex builds, but the proposed change appears to be kind of a red flag, even without the breaking dot. - check_include

I don't know enough about C and complex builds, but the proposed change appears to be kind of a red flag, even without the breaking dot. - check_include_file(linux/landlock.h HAVE_LINUX_LANDLOCK_H) ... + check_c_source_compiles(" + #include <linux/landlock.h Is a compilation-test a legitimate/common/typical method to go about this? Independently, of the breaking code, to me it seems accidental failing, or even accidentally not failing, would be in the nature of such an assessment... So, this commit seems to raise the question of "why?", even if you missed the dot, doesn't it? If a feature is formally available, but effectively broken somehow, wouldn't you want the compiler to complain, instead of the feature dropped silently? Is the reasoning in the code comment sound? Can you test, if syscalls are defined in another way?

ronsor 3y ago
> Is a compilation-test a legitimate/common/typical method to go about this? Yes—in fact, compilation tests are often the only way you can tell if a feature actually works. It's extremely common for C build systems to detect and work around weird systems.
- jijijijij 3y ago
  Is this by design, or by legacy? I mean, is there a better way to do this? Seems really flawed to me.
  - ninkendo 3y ago
    It’s by design. The job of autotools is to find “ground truth” about whatever environment you’re compiling against. It’s meant to discover if you can use a feature by actually seeing whether it works, not just by allow-listing a known set of compiler or library versions. This is because the whole point is to allow porting code to any environment where it’ll work, even on compilers you don’t know about. Think back to a time when there were several dozen Unix vendors, and just as many compilers. You don’t want your build script to report it can’t compile something just because it isn’t aware of your particular Unix vendor… you want it to only fail if the thing it’s trying to do actually doesn’t work. The only way to do this is by just testing if certain code compiles and produces the expected result.
    - Aeolun 3y ago
      I don’t know. In my code I’d always compile and check at runtime?
      - Denvercoder9 3y ago
        Not everything is detectable at runtime, such as syscall numbers (which is what's being tested for here).
        manwe150 3y ago
        Runtime syscall number detection is very common in practice, since the kernel returns ENOSYS to enable that exact ability for glibc and other shim libraries
        Denvercoder9 3y ago
        That's syscall for _availability_ detection, not its number. There's no way to ask the kernel "what's the number of the syscall commonly known as read()".
      - ninkendo 3y ago
        Some of the checks are there to tell whether or not the compiler even supports your code. You may not be able to compile your code at all, and the job of the build system is sometimes to just emit useful errors to help the person building the code to understand that they need a compiler which supports [language feature X]. Again, this is intended to be portable software. It is designed to work on lots of OS’s, with lots of compilers, in a lot of future environments that don’t even exist yet. If you have a security feature for example, which uses the pledge() syscall on OpenBSD, but you can only use that feature on OpenBSD systems, you have two choices: - Conditionally compile it based on whether you’ve detected that this is an OpenBSD target at build time, or, - Conditionally compile it based on whether some sample code that uses pledge() builds successfully. You can’t defer this decision until runtime, because it would require linking to pledge() symbols even though they may not exist on this system, which would cause the executable to fail to link at runtime, unless you completely rearchitected to use a plugin model, which is overkill. So given the above are your main two options, the latter is preferred mainly because it allows new systems to come in and be compatible with old ones (maybe someone adds pledge() support to Linux one day) without having to fudge the uname command or something. This was super important in the early Unix days… perhaps less so now, but it’s still a good way to write portable software that still can take advantage of platform-specific features.
  - secondcoming 3y ago
    Back in the day when people compiled source from tarballs on their personal machines, the autoconf script would query your system to see what functionality it supported. It did this by trying to compile small programs for each feature. If the compilation failed it assumes the feature is unavailable on your system and a flag is set/unset for the rest of the build.
  - kccqzy 3y ago
    Have you ever tried to manually build something from a release tarball by starting with ./configure? If so, have you observed how many times the compiler is invoked in this configure phase before you even run make?
  - db48x 3y ago
    It’s by “design”, in the sense that C and C++ provide no better way to really know for sure that the functions you want to call really exist. In more modern languages we rely on metadata and semver, but none of that exists for C and C++.
    - uecker 3y ago
      I never test for this in this way in my C projects. I also rely on version tests only.
      - db48x 3y ago
        Yea, it’s not impossible to do, just not standardized. If you use a library and it provides useful version information, then definitely use it. It’s just that the language or the tooling doesn’t force libraries to have that kind of metadata. Compare that with Rust, where every library you use must come with a standardized manifest that includes a version number, and where they tell library authors up front that they are expected to follow the semver convention. The fact is that things have good easier over the last 10 or 20 years. It used to be the case that any program targeting Unix had to either spend a lot of time and energy tracking the precise differences between dozens of different commercial Unices, or use autoconf. Autoconf was the project that combined all of that lore into a single place, so that most people didn’t have to know every single detail. But these days most of the Unices are dead and buried, and 99% of all new projects just target Linux. Kinda sucks if you prefer one of the BSDs, or OpenSolaris/Illumos/SmartOS, but it does mean that new Linux developers never have to jump through those hoops and simply never learn about autoconf. And while on the one hand that represents a loss of knowledge and competency for the community (making this type of supply–chain attack much easier), on the other hand autoconf is in practice an abomination. It is (or at least was) extremely useful, but it was implemented in M4 and reading the source code will literally damage your brain.
        acdha 3y ago
        > It used to be the case that any program targeting Unix had to either spend a lot of time and energy tracking the precise differences between dozens of different commercial Unices, or use autoconf. Autoconf was the project that combined all of that lore into a single place, so that most people didn’t have to know every single detail. As a data point, the place I worked for in the mid-90s had a single codebase with something over 600 permutations of supported OS and compiler when you included the different versions. One thing we take for granted now is how easy it is to get and install updates – back then you might find that, say, a common API was simply broken on one particular operating system version but your customers would have to wait for a patch to be released, sometimes purchased, put onto floppy disks or CD-ROM, and then manually installed in a way which had enough risk involved that people often put it off as long as they could. Some vendors also did individual patches which could be combined by the sysadmin so you had to test for that specific feature rather than just saying “is it greater than 1.2.3?”, and it wasn’t uncommon to find cases where they’d compiled a common library with some weird patches so you had to test whether the specific features you needed functioned. Part of why Linux annihilated them was cost but much of it was having package managers designed by grownups - I remember as late as the mid-2000s bricking brand new Sun servers by running the new Solaris updater, which left them in an unbootable state.
  - cryptonector 3y ago
    It's not by design, unlike what siblings say. It's by accident (so, "legacy", as you put it). The problem is that already in the 80s there was tons of variability from on Unix system (or version of it) to the next, but there was no standard way of representing what features/standards/APIs/libraries a system supported or had installed. When faced with such a mess people wrote code to detect features are present on the target host. This then got made into tools with libraries of detection code. Think autoconf/autotools. Now we also have pkgconfig, but it's too late and it was not enough anyways. Some things you might only detect at run-time, provided that their ABIs are stable enough that you can copy their headers into your application.
- Nezghul 3y ago
  That "check_c_source_compiles" function should first test if the provided code snipped is "valid C code in general" and only then check if it compiles in given system.

ronsor 3y ago

> Is a compilation-test a legitimate/common/typical method to go about this? Yes—in fact, compilation tests are often the only way you can tell if a feature actually works. It's extremely common for C build systems to detect and work around weird systems.

jijijijij 3y ago
Is this by design, or by legacy? I mean, is there a better way to do this? Seems really flawed to me.
- ninkendo 3y ago
  It’s by design. The job of autotools is to find “ground truth” about whatever environment you’re compiling against. It’s meant to discover if you can use a feature by actually seeing whether it works, not just by allow-listing a known set of compiler or library versions. This is because the whole point is to allow porting code to any environment where it’ll work, even on compilers you don’t know about. Think back to a time when there were several dozen Unix vendors, and just as many compilers. You don’t want your build script to report it can’t compile something just because it isn’t aware of your particular Unix vendor… you want it to only fail if the thing it’s trying to do actually doesn’t work. The only way to do this is by just testing if certain code compiles and produces the expected result.
  - Aeolun 3y ago
    I don’t know. In my code I’d always compile and check at runtime?
    - Denvercoder9 3y ago
      Not everything is detectable at runtime, such as syscall numbers (which is what's being tested for here).
      - manwe150 3y ago
        Runtime syscall number detection is very common in practice, since the kernel returns ENOSYS to enable that exact ability for glibc and other shim libraries
        Denvercoder9 3y ago
        That's syscall for _availability_ detection, not its number. There's no way to ask the kernel "what's the number of the syscall commonly known as read()".
    - ninkendo 3y ago
      Some of the checks are there to tell whether or not the compiler even supports your code. You may not be able to compile your code at all, and the job of the build system is sometimes to just emit useful errors to help the person building the code to understand that they need a compiler which supports [language feature X]. Again, this is intended to be portable software. It is designed to work on lots of OS’s, with lots of compilers, in a lot of future environments that don’t even exist yet. If you have a security feature for example, which uses the pledge() syscall on OpenBSD, but you can only use that feature on OpenBSD systems, you have two choices: - Conditionally compile it based on whether you’ve detected that this is an OpenBSD target at build time, or, - Conditionally compile it based on whether some sample code that uses pledge() builds successfully. You can’t defer this decision until runtime, because it would require linking to pledge() symbols even though they may not exist on this system, which would cause the executable to fail to link at runtime, unless you completely rearchitected to use a plugin model, which is overkill. So given the above are your main two options, the latter is preferred mainly because it allows new systems to come in and be compatible with old ones (maybe someone adds pledge() support to Linux one day) without having to fudge the uname command or something. This was super important in the early Unix days… perhaps less so now, but it’s still a good way to write portable software that still can take advantage of platform-specific features.
- secondcoming 3y ago
  Back in the day when people compiled source from tarballs on their personal machines, the autoconf script would query your system to see what functionality it supported. It did this by trying to compile small programs for each feature. If the compilation failed it assumes the feature is unavailable on your system and a flag is set/unset for the rest of the build.
- kccqzy 3y ago
  Have you ever tried to manually build something from a release tarball by starting with ./configure? If so, have you observed how many times the compiler is invoked in this configure phase before you even run make?
- db48x 3y ago
  It’s by “design”, in the sense that C and C++ provide no better way to really know for sure that the functions you want to call really exist. In more modern languages we rely on metadata and semver, but none of that exists for C and C++.
  - uecker 3y ago
    I never test for this in this way in my C projects. I also rely on version tests only.
    - db48x 3y ago
      Yea, it’s not impossible to do, just not standardized. If you use a library and it provides useful version information, then definitely use it. It’s just that the language or the tooling doesn’t force libraries to have that kind of metadata. Compare that with Rust, where every library you use must come with a standardized manifest that includes a version number, and where they tell library authors up front that they are expected to follow the semver convention. The fact is that things have good easier over the last 10 or 20 years. It used to be the case that any program targeting Unix had to either spend a lot of time and energy tracking the precise differences between dozens of different commercial Unices, or use autoconf. Autoconf was the project that combined all of that lore into a single place, so that most people didn’t have to know every single detail. But these days most of the Unices are dead and buried, and 99% of all new projects just target Linux. Kinda sucks if you prefer one of the BSDs, or OpenSolaris/Illumos/SmartOS, but it does mean that new Linux developers never have to jump through those hoops and simply never learn about autoconf. And while on the one hand that represents a loss of knowledge and competency for the community (making this type of supply–chain attack much easier), on the other hand autoconf is in practice an abomination. It is (or at least was) extremely useful, but it was implemented in M4 and reading the source code will literally damage your brain.
      - acdha 3y ago
        > It used to be the case that any program targeting Unix had to either spend a lot of time and energy tracking the precise differences between dozens of different commercial Unices, or use autoconf. Autoconf was the project that combined all of that lore into a single place, so that most people didn’t have to know every single detail. As a data point, the place I worked for in the mid-90s had a single codebase with something over 600 permutations of supported OS and compiler when you included the different versions. One thing we take for granted now is how easy it is to get and install updates – back then you might find that, say, a common API was simply broken on one particular operating system version but your customers would have to wait for a patch to be released, sometimes purchased, put onto floppy disks or CD-ROM, and then manually installed in a way which had enough risk involved that people often put it off as long as they could. Some vendors also did individual patches which could be combined by the sysadmin so you had to test for that specific feature rather than just saying “is it greater than 1.2.3?”, and it wasn’t uncommon to find cases where they’d compiled a common library with some weird patches so you had to test whether the specific features you needed functioned. Part of why Linux annihilated them was cost but much of it was having package managers designed by grownups - I remember as late as the mid-2000s bricking brand new Sun servers by running the new Solaris updater, which left them in an unbootable state.
- cryptonector 3y ago
  It's not by design, unlike what siblings say. It's by accident (so, "legacy", as you put it). The problem is that already in the 80s there was tons of variability from on Unix system (or version of it) to the next, but there was no standard way of representing what features/standards/APIs/libraries a system supported or had installed. When faced with such a mess people wrote code to detect features are present on the target host. This then got made into tools with libraries of detection code. Think autoconf/autotools. Now we also have pkgconfig, but it's too late and it was not enough anyways. Some things you might only detect at run-time, provided that their ABIs are stable enough that you can copy their headers into your application.
Nezghul 3y ago
That "check_c_source_compiles" function should first test if the provided code snipped is "valid C code in general" and only then check if it compiles in given system.