5 ms·

You cut out a key word: > Linux is really hurt here by the total lack of any unit testing or UI scripting standards. > standards I've been very impressed rea

You cut out a key word: > Linux is really hurt here by the total lack of any unit testing or UI scripting standards. > standards I've been very impressed reading how the Rust developers handle this. They have a tool called crater[1], which runs regression tests for the compiler against all Rust code ever released on crates.io or GitHub. Every front-facing change that is even slightly risky must pass a crater run. https://github.com/rust-lang/crater https://github.com/rust-lang/crater Surely Microsoft has internal tools for Windows that do the same thing: run a battery of tests across popular apps and make sure changes in the OS don't break any user apps. Where's the similar test harness for Linux you can run that tests hundreds of popular apps across Wayland/X11 and Gnome/KDE/XFCE and makes sure everything still works?

dataangel 4y ago
This still has nothing to do with Linux. Unit testing isn't standardized in most languages. Even in Rust people have custom frameworks! The Linux Kernel does have such a project doing batteries of tests. Userspace may not, but that's not a "unit test" problem. In fact it's the opposite, it's integration tests.
leeter 4y ago
> Surely Microsoft has internal tools for Windows that do the same thing: run a battery of tests across popular apps and make sure changes in the OS don't break any user apps. And hardware, they actually deploy to hardware they buy locally from retailers to verify things still work too last I checked. Because there is always that "one popular laptop" that has stupid quirks. I know they try to focus on a spectrum of commonly used models based on the telemetry too.
Ygg2 4y ago
And crater costs a bunch, runs for a week, and it's not a guarantee things won't break. I'm not sure it runs every crate or just top 1 million. It used to, but I could see that changing, if And in case of closed source software, that isn't publicly available, crates wouldn't work.
- kibwen 4y ago
  Crater's an embarrassingly parallel problem though, it's only a matter of how much hardware you throw at it. Microsoft already donates the hardware used by Crater, it would have no problem allocating 10x as much for its own purposes.
  - Ygg2 4y ago
    How much of crates are ran on crater? All of them? Also I think there are magnitudes more C libs/apps, than Rust crates.
    - kibwen 4y ago
      There are certainly more things written in C than in Rust--the advantage of being fifty years old--but the standardization of the build system in Rust means that it would be difficult for any C compiler (or OS, or libc, or etc.) to produce a comparable corpus of C code to automatically test against (crates.io currently has 90,000 crates). But that's fine, because for the purpose of this thread that just means that Microsoft's theoretical Crater-like run for Windows compatibility just takes even less time and resources to run.
      - jcranmer 4y ago
        If you want to compile a large fraction of C/C++ code, just take a distro and rebuild it from scratch--Debian actually does this reasonably frequently. All of the distros have to somehow solve the problem of figuring out how to compile and install everything they package, although some are better at letting you change the build environment for testing than others. (From what I understand, Debian and Nix are the best bets here.) But what that doesn't solve is making sure that the resulting builds actually works. Cargo, for Rust, makes running some form of tests relatively easy, and Rust is new enough that virtually every published package is going to contain some amount of unit tests. But for random open-source packages? Not really. Pick a random numerics library--for something like an linear programming solver, this is the most comprehensive automated test suite I've seen: https://github.com/coin-or/Clp/tree/master/test https://github.com/coin-or/Clp/tree/master/test

dataangel 4y ago

This still has nothing to do with Linux. Unit testing isn't standardized in most languages. Even in Rust people have custom frameworks! The Linux Kernel does have such a project doing batteries of tests. Userspace may not, but that's not a "unit test" problem. In fact it's the opposite, it's integration tests.

leeter 4y ago

> Surely Microsoft has internal tools for Windows that do the same thing: run a battery of tests across popular apps and make sure changes in the OS don't break any user apps. And hardware, they actually deploy to hardware they buy locally from retailers to verify things still work too last I checked. Because there is always that "one popular laptop" that has stupid quirks. I know they try to focus on a spectrum of commonly used models based on the telemetry too.

Ygg2 4y ago

And crater costs a bunch, runs for a week, and it's not a guarantee things won't break. I'm not sure it runs every crate or just top 1 million. It used to, but I could see that changing, if And in case of closed source software, that isn't publicly available, crates wouldn't work.

kibwen 4y ago
Crater's an embarrassingly parallel problem though, it's only a matter of how much hardware you throw at it. Microsoft already donates the hardware used by Crater, it would have no problem allocating 10x as much for its own purposes.
- Ygg2 4y ago
  How much of crates are ran on crater? All of them? Also I think there are magnitudes more C libs/apps, than Rust crates.
  - kibwen 4y ago
    There are certainly more things written in C than in Rust--the advantage of being fifty years old--but the standardization of the build system in Rust means that it would be difficult for any C compiler (or OS, or libc, or etc.) to produce a comparable corpus of C code to automatically test against (crates.io currently has 90,000 crates). But that's fine, because for the purpose of this thread that just means that Microsoft's theoretical Crater-like run for Windows compatibility just takes even less time and resources to run.
    - jcranmer 4y ago
      If you want to compile a large fraction of C/C++ code, just take a distro and rebuild it from scratch--Debian actually does this reasonably frequently. All of the distros have to somehow solve the problem of figuring out how to compile and install everything they package, although some are better at letting you change the build environment for testing than others. (From what I understand, Debian and Nix are the best bets here.) But what that doesn't solve is making sure that the resulting builds actually works. Cargo, for Rust, makes running some form of tests relatively easy, and Rust is new enough that virtually every published package is going to contain some amount of unit tests. But for random open-source packages? Not really. Pick a random numerics library--for something like an linear programming solver, this is the most comprehensive automated test suite I've seen: https://github.com/coin-or/Clp/tree/master/test https://github.com/coin-or/Clp/tree/master/test