13 ms·

There is an interesting approach to this in Rust: if a potentially breaking change (e.g. a soundness fix) is being proposed, they usually test it against all pu

by dthul 3y ago

There is an interesting approach to this in Rust: if a potentially breaking change (e.g. a soundness fix) is being proposed, they usually test it against all publicly available Rust code.

eesmith 3y ago
I'm not sure that's the flex you think it might be. At least, I interpret it as saying there isn't much publicly available Rust code, and only a few places to find Rust code. I have a hard time even estimating how long it would take to test a change against all publicly available C++ code. FWIW, the C++ standards developers use do use code search tools to help identify possible breakage.
- lifthrasiir 3y ago
  Note that this tool (crater) is mainly used to catch regressions, not to judge whether it's worth to intentionally break things (which is what editions are for). If many crates depend on the bug which devs want to fix crater will probably detect that and devs will consider other alternatives.
- Entalpi 3y ago
  Pretty sure from a build system perspective its quite a flex .. to be fair.
  - eesmith 3y ago
    I understand the intent of the flex, but if true, it suggests there's very little public Rust outside of packages that can be downloaded from crates.io and a smallish list of alternatives. By comparison, there's so much publicly available Python code, from so many sources, that no one can honestly say they can even find it all. The same for C++. I've seen papers where the source code was included in the paper itself (eg, the FORTRAN code in Sibson's 1973 "SLINK" paper), or only distributed as a zip file from the author's web site, or in the supplementary data (eg, https://scholar.google.com/scholar?q=%22source+code+in+the+supplementary%22 https://scholar.google.com/scholar?q=%22source+code+in+the+s... ) . Personally, I don't think it's true. I suspect Rust changes - just like new proposed C++ changes - are checked against only easily and "well-known" accessible package.
    - dralley 3y ago
      >if true, it suggests there's very little public Rust outside of packages that can be downloaded from crates.io and a smallish list of alternatives. You seem to be suggesting that it's a good thing that the public code is spread across so many different places that it cannot all be found. I don't see how that's an inherently good thing. It says less about the total amount of code than it does about the lack of any central resource that can be consulted.
      - eesmith 3y ago
        > that it cannot all be found. Do you think you can find all public Rust code? Like, if I'm teaching a Rust course, and put a hello-world.rs program on my department's public GitLab instance, under an MIT license, do you think I should also put that on GitHub? And register it as a crate? > the lack of any central resource that can be consulted. And you say that like it's a good thing. You want everything to be centralized on GitHub? If so, you want to force all research software developers to agree to the GitHub's terms, including those who are ardent free software advocates. You also prevent 12 years olds from publishing their Rust source code. (GitHub's terms of service don't allow that.) Or, do you also allow BitBucket [1], and GitLab [2]? [1] https://bitbucket.org/project_samar/samar_lite/src/master/ https://bitbucket.org/project_samar/samar_lite/src/master/ contains two Rust programs, neither on crates.io [2] https://gitlab.com/rouault-team-public/analysis/umaprs https://gitlab.com/rouault-team-public/analysis/umaprs What about department instances of GitLab? [3] https://gitlab.anu.edu.au/mu/mu-impl-fast/-/tree/rtmu-dev https://gitlab.anu.edu.au/mu/mu-impl-fast/-/tree/rtmu-dev It really doesn't seem like it's all that easy to find all publicly available Rust code.
        dralley 3y ago
        What bearing does any of this have on the previous thread of discussion? Why do you think a 12 year old needs to publish their "hello world" programs because of Crater? The purpose of Crater is uncovering subtle compiler regressions. If "hello world" is ever broken then it would likely be discovered by the standard test suite or generally long before the Crater run. This isn't a matter of "allowing" anything. It's just a statement that yes a Crater run does test all meaningful publicly available code, where "meaningful" at the very least means code which is consumed via crates.io. Sure, there is very likely public code that exists elsewhere which Crater cannot find, and that's OK. The point is that a Crater run coming back clean means something, because a very very wide swath of code was tested.
- aldanor 3y ago
  It's exactly the flex you might think it is. The point is not in the number, but rather in the fact that you can build and test almost all crates available on almost all supported platforms, regardless of how many there are, and with no human intervention. For C++, there's no registry to start with, but even if there was: there's no standard way of building and testing projects. So, a crater build like this is simply impossible at a scale.
  - lionkor 3y ago
    > For C++, there's no registry to start with That's not right, there are multiple. There's no single registry. You can use the vcpkg registry, for example - that holds all small and large libraries I've ever needed (even one of my own). They also come with a standard way to build them, of course (CMake targets). Have you done C++ development recently? I fear a large part of the C++ crowd may not be aware that package managers and build systems are available, they're just not preinstalled with C++.
    - aldanor 3y ago
      I know about vcpkg but there's only 2k packages there. The point of the grandparent comment was that there's SO much C++ code / so many packages you can build in a crater run that it would be physically unfeasible. The problem with vcpkg, just like with any other "ports" package manager, is that it's not maintained by the original authors of the code but rather by a separate community. It's a bunch of "ports", trying to standardise the builds and installs to a common format. Out of wonder, I looked at a few recipes, they seem to just install things for you, but you have no automated way to do a crater run still - since most of those libraries are header-only you will need to write library-specific code in each case to actually use each package; at least a single include. The tests are seemingly also not being run.
      - lionkor 3y ago
        That's fair - but since this is about core language changes, compiling the non-header-only libraries already covers most of the commonly used libraries. Libraries like boost will also use most existing C++ features. I can't think of a single feature boost doesn't use, actually.
3836293648 3y ago
But soundness fixes are typically considered more important than compat and go ahead anyway

by dthul 3y ago

There is an interesting approach to this in Rust: if a potentially breaking change (e.g. a soundness fix) is being proposed, they usually test it against all publicly available Rust code.

eesmith 3y ago

I'm not sure that's the flex you think it might be. At least, I interpret it as saying there isn't much publicly available Rust code, and only a few places to find Rust code. I have a hard time even estimating how long it would take to test a change against all publicly available C++ code. FWIW, the C++ standards developers use do use code search tools to help identify possible breakage.

lifthrasiir 3y ago
Note that this tool (crater) is mainly used to catch regressions, not to judge whether it's worth to intentionally break things (which is what editions are for). If many crates depend on the bug which devs want to fix crater will probably detect that and devs will consider other alternatives.
Entalpi 3y ago
Pretty sure from a build system perspective its quite a flex .. to be fair.
- eesmith 3y ago
  I understand the intent of the flex, but if true, it suggests there's very little public Rust outside of packages that can be downloaded from crates.io and a smallish list of alternatives. By comparison, there's so much publicly available Python code, from so many sources, that no one can honestly say they can even find it all. The same for C++. I've seen papers where the source code was included in the paper itself (eg, the FORTRAN code in Sibson's 1973 "SLINK" paper), or only distributed as a zip file from the author's web site, or in the supplementary data (eg, https://scholar.google.com/scholar?q=%22source+code+in+the+supplementary%22 https://scholar.google.com/scholar?q=%22source+code+in+the+s... ) . Personally, I don't think it's true. I suspect Rust changes - just like new proposed C++ changes - are checked against only easily and "well-known" accessible package.
  - dralley 3y ago
    >if true, it suggests there's very little public Rust outside of packages that can be downloaded from crates.io and a smallish list of alternatives. You seem to be suggesting that it's a good thing that the public code is spread across so many different places that it cannot all be found. I don't see how that's an inherently good thing. It says less about the total amount of code than it does about the lack of any central resource that can be consulted.
    - eesmith 3y ago
      > that it cannot all be found. Do you think you can find all public Rust code? Like, if I'm teaching a Rust course, and put a hello-world.rs program on my department's public GitLab instance, under an MIT license, do you think I should also put that on GitHub? And register it as a crate? > the lack of any central resource that can be consulted. And you say that like it's a good thing. You want everything to be centralized on GitHub? If so, you want to force all research software developers to agree to the GitHub's terms, including those who are ardent free software advocates. You also prevent 12 years olds from publishing their Rust source code. (GitHub's terms of service don't allow that.) Or, do you also allow BitBucket [1], and GitLab [2]? [1] https://bitbucket.org/project_samar/samar_lite/src/master/ https://bitbucket.org/project_samar/samar_lite/src/master/ contains two Rust programs, neither on crates.io [2] https://gitlab.com/rouault-team-public/analysis/umaprs https://gitlab.com/rouault-team-public/analysis/umaprs What about department instances of GitLab? [3] https://gitlab.anu.edu.au/mu/mu-impl-fast/-/tree/rtmu-dev https://gitlab.anu.edu.au/mu/mu-impl-fast/-/tree/rtmu-dev It really doesn't seem like it's all that easy to find all publicly available Rust code.
      - dralley 3y ago
        What bearing does any of this have on the previous thread of discussion? Why do you think a 12 year old needs to publish their "hello world" programs because of Crater? The purpose of Crater is uncovering subtle compiler regressions. If "hello world" is ever broken then it would likely be discovered by the standard test suite or generally long before the Crater run. This isn't a matter of "allowing" anything. It's just a statement that yes a Crater run does test all meaningful publicly available code, where "meaningful" at the very least means code which is consumed via crates.io. Sure, there is very likely public code that exists elsewhere which Crater cannot find, and that's OK. The point is that a Crater run coming back clean means something, because a very very wide swath of code was tested.
aldanor 3y ago
It's exactly the flex you might think it is. The point is not in the number, but rather in the fact that you can build and test almost all crates available on almost all supported platforms, regardless of how many there are, and with no human intervention. For C++, there's no registry to start with, but even if there was: there's no standard way of building and testing projects. So, a crater build like this is simply impossible at a scale.
- lionkor 3y ago
  > For C++, there's no registry to start with That's not right, there are multiple. There's no single registry. You can use the vcpkg registry, for example - that holds all small and large libraries I've ever needed (even one of my own). They also come with a standard way to build them, of course (CMake targets). Have you done C++ development recently? I fear a large part of the C++ crowd may not be aware that package managers and build systems are available, they're just not preinstalled with C++.
  - aldanor 3y ago
    I know about vcpkg but there's only 2k packages there. The point of the grandparent comment was that there's SO much C++ code / so many packages you can build in a crater run that it would be physically unfeasible. The problem with vcpkg, just like with any other "ports" package manager, is that it's not maintained by the original authors of the code but rather by a separate community. It's a bunch of "ports", trying to standardise the builds and installs to a common format. Out of wonder, I looked at a few recipes, they seem to just install things for you, but you have no automated way to do a crater run still - since most of those libraries are header-only you will need to write library-specific code in each case to actually use each package; at least a single include. The tests are seemingly also not being run.
    - lionkor 3y ago
      That's fair - but since this is about core language changes, compiling the non-header-only libraries already covers most of the commonly used libraries. Libraries like boost will also use most existing C++ features. I can't think of a single feature boost doesn't use, actually.

3836293648 3y ago

But soundness fixes are typically considered more important than compat and go ahead anyway