4 ms·

There's a kind of ecosystem resilience that could be improved (and I think gradually is improving) here. We have utilities to determine whether a package is ba

by jka 5y ago

There's a kind of ecosystem resilience that could be improved (and I think gradually is improving) here. We have utilities to determine whether a package is backwards-compatible, as far as we know -- unit tests, QA tests, integration tests, etc -- we can (under many practical circumstances) set resource bounds on computation, and we can review and detect many categories of security errors (that are arguably simply a subset of the more general category of programming and design errors -- also possible to detect). Part of the problem, I reckon, is that the time and computation cost to run the tests in particular is often too expensive for it to be worthwhile to run them at an individual commit level. Progress on incremental tests (determine the data-flow graph from the area of code that a commit affects, determine the set of inputs and outputs potentially affected, and run a subset or relevant tests) could help - as could providing more of these tools to individual developers, and reducing the time and expense of running costs in general.

dane-pgp 5y ago
That's a good point. It would be nice if NPM/Microsoft committed to running backwards compatibility tests on packages that had them (for packages that were significant enough and tests that were efficient enough). This would require a convention that packages indicate in their metadata where the backwards compatibility tests are, and for each package's code to be run against the tests in the previous version of the package (or all previous versions with the same major version number?). A slightly different approach would be for packages to write tests for the specific functionality of their dependencies that they rely on, and NPM could collect that distributed test suite and warn the test writer if a newly published version of one of their dependencies wasn't backwards compatible. NPM could even store such packages in a staging area (for a probationary period) where the dependency solving algorithm of npm wouldn't look by default.

by jka 5y ago

dane-pgp 5y ago

That's a good point. It would be nice if NPM/Microsoft committed to running backwards compatibility tests on packages that had them (for packages that were significant enough and tests that were efficient enough). This would require a convention that packages indicate in their metadata where the backwards compatibility tests are, and for each package's code to be run against the tests in the previous version of the package (or all previous versions with the same major version number?). A slightly different approach would be for packages to write tests for the specific functionality of their dependencies that they rely on, and NPM could collect that distributed test suite and warn the test writer if a newly published version of one of their dependencies wasn't backwards compatible. NPM could even store such packages in a staging area (for a probationary period) where the dependency solving algorithm of npm wouldn't look by default.