6 ms·

There is no gatekeeping. To actually achieve POSIX-compliance is not an easy task and requires lots of testing. If you don't do it then your replacement will br

There is no gatekeeping. To actually achieve POSIX-compliance is not an easy task and requires lots of testing. If you don't do it then your replacement will break everyone's scripts. A distro maintainer will also want to retain compatibility with their supported GNU/BSD extensions so that's more work to add on. I would still agree with his statement at least as far as BSD is concerned. Outside of Redox I haven't seen any serious projects to implement POSIX compatibility. The Rust coreutils project is a good start but from their readme it looks like they are more aiming to achieve GNU compatibility on Microsoft Windows while being MIT licensed — I don't think anyone is seriously using it to build a BSD (or even a GNU) distro. If I'm wrong about that I'd love to hear it though. Rust is a good choice to write these things in but let's be realistic about the time frame required to rewrite everything. Plus I just downloaded those Rust coreutils and tried to build it and now I'm waiting for 400 (!!!) Rust dependencies to download and compile. Is this really appropriate for a core system component to have this many dependencies? Or is there something I'm missing here? I admit I am not familiar with best practices in Rust. As the project stabilizes I assume they will want to start eliminating the dependencies or vendoring them upstream? At what point do we decide to put these into a separate library like libbsd or gnulib?

notaplumber 6y ago
> Is this really appropriate for a core system component to have this many dependencies? Absolutely not. The entire OpenBSD base system can be built without an internet connection.
- em-bee 6y ago
  after you are done downloading all the source.
  - hedora 6y ago
    You can check that the source hasn’t been tampered with before starting the build, and know you’re building the right code base. Builds that curl junk and use a dozen language specific package managers don’t have that property.
    - damnyou 6y ago
      That is exactly the set of properties you get with Rust and Cargo: that the source hasn't been tampered with, and that you know you're building the right codebase. It would be deeply irresponsible to build a package management system without those properties.
      - Beldin 6y ago
        True enough... but you still don't know what the dependencies do. E.g., one added ads. Another added use tracking. This could even be legitimately in line with the dependency's purpose. Unless you're manually vetting all the code, 400 dependencies means trusting a lot of third parties.
        hedora 6y ago
        Trusting a lot of third parties is almost as irresponsible as not bothering with checksumming, etc. See npm and pip for examples. I suspect cargo will have its share of incidents over time, unless it is somehow curated by a small group that tests and maintains the packages, but if it is, then it’s comparable to apt for c/c++.
      - burntsushi 6y ago
        This is one of those things that's strictly true, but is kind of misleading because it doesn't capture the full picture. In particular, there is actually no guarantee that the source code you see on GitHub matches the source code you compile from crates.io. Now, most responsible maintainers tag each release and publish exactly that tag to crates.io. Which is good. But that's just a convention. There's nothing enforcing it. If you want to actually review the source code, then you'd have to download the crate archive itself and review the code there. There is some tooling for this (notably, cargo-crev), but it's definitely not a normal part of development in the Rust open source ecosystem as it stands right now. For the most part, trust and reputation hold the system together. For example, imagine what would happen if someone found malicious code in one of my crates that I was duplicitous about (i.e., published it on crates.io but kept those code changes off of GitHub).
- steveklabnik 6y ago
  You can also build Rust and its ecosystem without an internet connection. It works the same way; you need to get a copy of everything first, but the build itself does not require the internet. (This has been a hard constraint for effectively forever, because both Firefox and Linux distributions have this requirement.)
  - Beldin 6y ago
    I understand that if you have all dependencies locally, you don't need to download more dependencies. I understood NotAPlumber's remark as: openbsd has no third party dependencies. In other words: there is one party to trust, not 400. If I misunderstood the BSD case, I'd love to hear it.
dathinab 6y ago
I don't think you miss anything. POSIX compliance is a headache which is not only hard to get right but also strongly limits your interfaces and internal tooling. Many scripts still will run with non full compliance, but that doesn't help if you try to build for OpenBSD. Also I'm not sure how they ended up with 400 deps for coreutils. I would have expected much less. But yes this means at least until better code signing and so on is the default this isn't at all appropriate for the base of any OS. I just which the author would have written it a bit less mean in it's wording. (Probably was just annoyed, but sill).
- lonelappde 6y ago
  The author is from the 1990s era of Linux / FOSS, before kindness was part of the culture, when the culture had a tough competitive streak where mistakes and humility were seen as signs of weakness and disrespected. High expectations wee tightly bound with low levels of patience. And many people were scared off from contributing.
- saagarjha 6y ago
  > POSIX compliance is a headache which is not only hard to get right but also strongly limits your interfaces and internal tooling. Platforms which are POSIX compliant usually offer extensions and then allow for selecting actual POSIX-compliant behavior. It's not a very strong limitation if you're already fairly close to that anyways.
throwaway894345 6y ago
Most scripts don’t depend on the POSIX standards anyway, they depend on GNU or BSD programs. CPython’s configure script, for example, blows up if the system grep is BSD and not GNU. The whole C build ecosystem is duct tape anyway and things rarely build without tons of manual effort (figure out what versions of which dependencies need to be installed to which locations in the file tree and then maybe things will build, but none of it will be documented and good luck reverse engineering those Autotools/CMake scripts!), so citing concerns about breaking things seems really disingenuous.
- afiori 6y ago
  I believe a main point of the article is that while linux might be fine with this state of affairs, OpenBSD is not and actually care about having a specific standard.
deleted 6y ago
[deleted]
oefrha 6y ago
To replace anything in OpenBSD, you need way more than POSIX-compliance -- you need OpenBSD-compliance. Which means every non-POSIX option has to be implemented down to edge cases, every departure from or controversial interpretation of POSIX (POSIX is vague in many places) has to behave exactly the same. Or you'll definitely break scripts.
- anthk 6y ago
  These nuts don't understand OpenBSD's base doesn't map to GNU coreutils, it ships SEVERAL more tools. They are so GNU and Rust blind they don't get anything. .ls -1 /bin/ /sbin/ /usr/bin/ /usr/sbin/ | sort -u | wc -l 671 Good luck reimplementing that.
DougBTX 6y ago
> Or is there something I'm missing here? There may be some platform-specific dependencies, but at least when building on Windows, I saw 67 dependencies downloaded. One thing that may be skewing the count is that internally coreutils is packaged with one crate per command. There are about 100 commands, so you'll have seen at least that many separate crates building. They're listed here: https://github.com/uutils/coreutils/blob/6e8c901204934029c883553bf8c69b9c43577714/Cargo.toml#L217 https://github.com/uutils/coreutils/blob/6e8c901204934029c88... My impression is that ecosystems where tracking dependencies is hard tend to discourage them, while ecosystems where tracking dependencies is easy encourage them. The coreutils crate depends on separate md5, sha1, sha2, and sha3 crates, if managing external dependencies was hard I imagine those would be bundled together. But it is easy, so it makes sense to organise the crates so that each just provides one specific piece of functionality. Think single responsibility principle applied to packaging.
- gmueckl 6y ago
  Urgh, that's just a verification nightmare waiting to happen. Who is going to wade through all that mess, review every package, pin its version etc...? You need reproducability before you can get safety and security.
  - AgentME 6y ago
    Cargo uses lockfiles by default to pin all dependencies of an application, so reproducibility shouldn't be much of an issue.
    - hedora 6y ago
      No one reads lockfiles. In python, people have successfully backdoored projects by sending a PR that upgrades a dependency and regenerates the lockfiles. Somewhere in the 1MB lockfile patch, they pin a dependency to a version with a known remotely exploitable flaw (sometimes a zero day, sometimes not).
- hedora 6y ago
  If you care about security, api stability and legal compliance, you should minimize the number of developers and packages you trust. My rule of thumb is that, for a team of ~10 developers, each package dependency adds a week or two of developer maintenance time each year or so. Having a dozen packages where one will do, and having them do the same thing recursively means you need to spend 144 times longer dealing with such things.
zozbot234 6y ago
Rust dependencies are just compilation units. Do people complain when a coreutils-scale project ultimately depends on 400 .c (and/or .o) files?
dathinab 6y ago
Actually you did miss something. Thinks like `cargo tree | wc -l` or the number of dependencies cargo builds are not appropriate as they count _internal_ dependencies (from the same repo). Given that it's split into many "sub-crates" in the same workspace this will make the numbers unnatural high. Or with other words _most_ of the 417 "dependencies" for building the "unix" feature set are internal ones in the same repository. I.e. the project is just split up into many small parts. Additionally many deps are reused between many internal deps. Removing duplicate dependencies, dev-dependencies (e.g. testing tools) and build dependencies less then 100 deps are left. Looking through them many are rust versions a C/C++ libs which are "available per default", like e.g. unix_socket, libc and similar. Then some are thinks which come from rust deps being split up more, e.g. there are md5, sha1, sha2, sha3 as separate crate. There are also a bunch of "internal" deps of deps e.g. backtrace-sys. Anyway still a lot of deps but most are quite reasonable and could be "taken care of" in some way or another if they want to ship a distro based on this.
zamalek 6y ago
His statement was that there was "no attempt" to write POSIX tools. As a "statement of fact." Irrespective of how much work there is still to do for coreutils in terms of compliance, they are an attempt. They may be be completely unsuitable as a drop-in replacement right now, but that was not the "fact" that he asserted.
michaelcampbell 6y ago
Somewhat academic as I have no dog in the fight, but ... > To actually achieve POSIX-compliance is not an easy task and requires lots of testing. Why are the existing tests not enough for a new implementation?
- ncmncm 6y ago
  Sweet summer child, imagining OpenBSD has comprehensive tests? This is UNIX, old-school. If it seems to work, it works.
deleted 6y ago
[deleted]

notaplumber 6y ago

> Is this really appropriate for a core system component to have this many dependencies? Absolutely not. The entire OpenBSD base system can be built without an internet connection.

em-bee 6y ago
after you are done downloading all the source.
- hedora 6y ago
  You can check that the source hasn’t been tampered with before starting the build, and know you’re building the right code base. Builds that curl junk and use a dozen language specific package managers don’t have that property.
  - damnyou 6y ago
    That is exactly the set of properties you get with Rust and Cargo: that the source hasn't been tampered with, and that you know you're building the right codebase. It would be deeply irresponsible to build a package management system without those properties.
    - Beldin 6y ago
      True enough... but you still don't know what the dependencies do. E.g., one added ads. Another added use tracking. This could even be legitimately in line with the dependency's purpose. Unless you're manually vetting all the code, 400 dependencies means trusting a lot of third parties.
      - hedora 6y ago
        Trusting a lot of third parties is almost as irresponsible as not bothering with checksumming, etc. See npm and pip for examples. I suspect cargo will have its share of incidents over time, unless it is somehow curated by a small group that tests and maintains the packages, but if it is, then it’s comparable to apt for c/c++.
    - burntsushi 6y ago
      This is one of those things that's strictly true, but is kind of misleading because it doesn't capture the full picture. In particular, there is actually no guarantee that the source code you see on GitHub matches the source code you compile from crates.io. Now, most responsible maintainers tag each release and publish exactly that tag to crates.io. Which is good. But that's just a convention. There's nothing enforcing it. If you want to actually review the source code, then you'd have to download the crate archive itself and review the code there. There is some tooling for this (notably, cargo-crev), but it's definitely not a normal part of development in the Rust open source ecosystem as it stands right now. For the most part, trust and reputation hold the system together. For example, imagine what would happen if someone found malicious code in one of my crates that I was duplicitous about (i.e., published it on crates.io but kept those code changes off of GitHub).
steveklabnik 6y ago
You can also build Rust and its ecosystem without an internet connection. It works the same way; you need to get a copy of everything first, but the build itself does not require the internet. (This has been a hard constraint for effectively forever, because both Firefox and Linux distributions have this requirement.)
- Beldin 6y ago
  I understand that if you have all dependencies locally, you don't need to download more dependencies. I understood NotAPlumber's remark as: openbsd has no third party dependencies. In other words: there is one party to trust, not 400. If I misunderstood the BSD case, I'd love to hear it.

dathinab 6y ago

I don't think you miss anything. POSIX compliance is a headache which is not only hard to get right but also strongly limits your interfaces and internal tooling. Many scripts still will run with non full compliance, but that doesn't help if you try to build for OpenBSD. Also I'm not sure how they ended up with 400 deps for coreutils. I would have expected much less. But yes this means at least until better code signing and so on is the default this isn't at all appropriate for the base of any OS. I just which the author would have written it a bit less mean in it's wording. (Probably was just annoyed, but sill).

lonelappde 6y ago
The author is from the 1990s era of Linux / FOSS, before kindness was part of the culture, when the culture had a tough competitive streak where mistakes and humility were seen as signs of weakness and disrespected. High expectations wee tightly bound with low levels of patience. And many people were scared off from contributing.
saagarjha 6y ago
> POSIX compliance is a headache which is not only hard to get right but also strongly limits your interfaces and internal tooling. Platforms which are POSIX compliant usually offer extensions and then allow for selecting actual POSIX-compliant behavior. It's not a very strong limitation if you're already fairly close to that anyways.

throwaway894345 6y ago

Most scripts don’t depend on the POSIX standards anyway, they depend on GNU or BSD programs. CPython’s configure script, for example, blows up if the system grep is BSD and not GNU. The whole C build ecosystem is duct tape anyway and things rarely build without tons of manual effort (figure out what versions of which dependencies need to be installed to which locations in the file tree and then maybe things will build, but none of it will be documented and good luck reverse engineering those Autotools/CMake scripts!), so citing concerns about breaking things seems really disingenuous.

afiori 6y ago
I believe a main point of the article is that while linux might be fine with this state of affairs, OpenBSD is not and actually care about having a specific standard.

deleted 6y ago

[deleted]

oefrha 6y ago

To replace anything in OpenBSD, you need way more than POSIX-compliance -- you need OpenBSD-compliance. Which means every non-POSIX option has to be implemented down to edge cases, every departure from or controversial interpretation of POSIX (POSIX is vague in many places) has to behave exactly the same. Or you'll definitely break scripts.

anthk 6y ago
These nuts don't understand OpenBSD's base doesn't map to GNU coreutils, it ships SEVERAL more tools. They are so GNU and Rust blind they don't get anything. .ls -1 /bin/ /sbin/ /usr/bin/ /usr/sbin/ | sort -u | wc -l 671 Good luck reimplementing that.

DougBTX 6y ago

> Or is there something I'm missing here? There may be some platform-specific dependencies, but at least when building on Windows, I saw 67 dependencies downloaded. One thing that may be skewing the count is that internally coreutils is packaged with one crate per command. There are about 100 commands, so you'll have seen at least that many separate crates building. They're listed here: https://github.com/uutils/coreutils/blob/6e8c901204934029c883553bf8c69b9c43577714/Cargo.toml#L217 https://github.com/uutils/coreutils/blob/6e8c901204934029c88... My impression is that ecosystems where tracking dependencies is hard tend to discourage them, while ecosystems where tracking dependencies is easy encourage them. The coreutils crate depends on separate md5, sha1, sha2, and sha3 crates, if managing external dependencies was hard I imagine those would be bundled together. But it is easy, so it makes sense to organise the crates so that each just provides one specific piece of functionality. Think single responsibility principle applied to packaging.

gmueckl 6y ago
Urgh, that's just a verification nightmare waiting to happen. Who is going to wade through all that mess, review every package, pin its version etc...? You need reproducability before you can get safety and security.
- AgentME 6y ago
  Cargo uses lockfiles by default to pin all dependencies of an application, so reproducibility shouldn't be much of an issue.
  - hedora 6y ago
    No one reads lockfiles. In python, people have successfully backdoored projects by sending a PR that upgrades a dependency and regenerates the lockfiles. Somewhere in the 1MB lockfile patch, they pin a dependency to a version with a known remotely exploitable flaw (sometimes a zero day, sometimes not).
hedora 6y ago
If you care about security, api stability and legal compliance, you should minimize the number of developers and packages you trust. My rule of thumb is that, for a team of ~10 developers, each package dependency adds a week or two of developer maintenance time each year or so. Having a dozen packages where one will do, and having them do the same thing recursively means you need to spend 144 times longer dealing with such things.

zozbot234 6y ago

Rust dependencies are just compilation units. Do people complain when a coreutils-scale project ultimately depends on 400 .c (and/or .o) files?

dathinab 6y ago

Actually you did miss something. Thinks like `cargo tree | wc -l` or the number of dependencies cargo builds are not appropriate as they count _internal_ dependencies (from the same repo). Given that it's split into many "sub-crates" in the same workspace this will make the numbers unnatural high. Or with other words _most_ of the 417 "dependencies" for building the "unix" feature set are internal ones in the same repository. I.e. the project is just split up into many small parts. Additionally many deps are reused between many internal deps. Removing duplicate dependencies, dev-dependencies (e.g. testing tools) and build dependencies less then 100 deps are left. Looking through them many are rust versions a C/C++ libs which are "available per default", like e.g. unix_socket, libc and similar. Then some are thinks which come from rust deps being split up more, e.g. there are md5, sha1, sha2, sha3 as separate crate. There are also a bunch of "internal" deps of deps e.g. backtrace-sys. Anyway still a lot of deps but most are quite reasonable and could be "taken care of" in some way or another if they want to ship a distro based on this.

zamalek 6y ago

His statement was that there was "no attempt" to write POSIX tools. As a "statement of fact." Irrespective of how much work there is still to do for coreutils in terms of compliance, they are an attempt. They may be be completely unsuitable as a drop-in replacement right now, but that was not the "fact" that he asserted.

michaelcampbell 6y ago

Somewhat academic as I have no dog in the fight, but ... > To actually achieve POSIX-compliance is not an easy task and requires lots of testing. Why are the existing tests not enough for a new implementation?

ncmncm 6y ago
Sweet summer child, imagining OpenBSD has comprehensive tests? This is UNIX, old-school. If it seems to work, it works.

deleted 6y ago

[deleted]