3 ms·

The tests don't prevent bugs, but they do make fixes a lot faster to get out, since you can be confident your fix doesn't break any functionality.

drikerf 4y ago
Any functionality that is tested :)
- rozenmd 4y ago
  "if it's not tested, it's not functionality"
  - jeffreygoesto 4y ago
    Hyrum would want to have a word...
    - ithkuil 4y ago
      If only there was a way to run all the tests of all the software that depend on your change... Fwiw, that's what monorepos are good for. Sadly it's hard to make a "world monorepo"
      - franga2000 4y ago
        If you pin your dependency versions and the dependency maintainers run tests before releasing, you're already almost there (excluding bugs that might arise only in your specific environment). If dependencies don't have tests or test automation, you can always contribute them. As for the platform-specific bugs, monorepos only help if the way to run tests in all the components is standardised. But this is something you can just as easily implement across repos. Could be as simple as having a testme.sh in the root.
        ithkuil 4y ago
        > > "if it's not tested, it's not functionality" This suggests you may have not 100% test coverage in your tests. But 100% coverage if what? What is the specification you're defining your behaviour against? The comment above suggests that you could treat your tests as if they were the ones that actually define your contract. > Hyrum would like to have a word. This is a reference to "Hyrum's law" which says: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." This comment, in response to the previous about defining tests as the source of truth for your contract, remarks that sadly you can't do that because no matter what contract you wish you define, ultimately the behaviour of your existing software becomes its effective contract. > [My comment about monorepos] Here I suggest that if you extend the notion of what is the test corpus to include the test corpus of all of the software that depend on you (not your dependencies! The code whom your code is a dependency) then you could detect if a (yet unmerged) change you're making is actually going to affect any existing code.
        franga2000 4y ago
        Totally misread the "depend on you" as "you depend on" part, sorry. Interesting idea, but monorepos still don't help on their own. You need a way to detect which modules depend on yours and a way to run and interpret their tests. Whether they're in an adjacent folder or another repo changes very little. GitHub actually already has a dependency scanning thingy that builds cross-repo dependency graphs [0] and package managers have been doing that for years. The missing parts wouldn't be that much effort to build, but getting everyone to set up their repos to work with it would probably be prohibitively difficult. [0] https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/exploring-the-dependencies-of-a-repository#dependents-view https://docs.github.com/en/code-security/supply-chain-securi...
        ithkuil 4y ago
        Monorepos are useless unless you have good tooling, including a build system that is aware of module dependencies inside the monorepo. An example build system that can do that is https://bazel.build https://bazel.build
      - YoshiRulz 4y ago
        That's exactly what the Nixpkgs Hydra instance does—caveat being your library and its consumers all need to be in Nixpkgs. I'm hopeful that there will be a way to keep the tracking part of that going forward, as Flakes are adopted and the community decentralises.
deepsun 4y ago
Not if a simple test requires rewriting hundreds of tests (aka "fragile tests"). Yes, one shouldn't be writing fragile tests, but usually from what I seen at projects with great test coverage, is that it often slows bugfix releases, and especially any bigger changes, as it's very wearisome to also change hundreds of tests. So I believe there should be some balance between tests/code ratio, as well as attention paid to tests brittleness.
- 2rsf 4y ago
  Definitely, some real world cases research show that this balance stands around 80% for unit tests- you don't earn more quality as you add more test above that.
  - deepsun 4y ago
    Last time I heard 80% ratio is optimal for Java, but any dynamic typing language (JS, Python, Ruby) it's around 100% test LOC / code LOC.

drikerf 4y ago

Any functionality that is tested :)

rozenmd 4y ago
"if it's not tested, it's not functionality"
- jeffreygoesto 4y ago
  Hyrum would want to have a word...
  - ithkuil 4y ago
    If only there was a way to run all the tests of all the software that depend on your change... Fwiw, that's what monorepos are good for. Sadly it's hard to make a "world monorepo"
    - franga2000 4y ago
      If you pin your dependency versions and the dependency maintainers run tests before releasing, you're already almost there (excluding bugs that might arise only in your specific environment). If dependencies don't have tests or test automation, you can always contribute them. As for the platform-specific bugs, monorepos only help if the way to run tests in all the components is standardised. But this is something you can just as easily implement across repos. Could be as simple as having a testme.sh in the root.
      - ithkuil 4y ago
        > > "if it's not tested, it's not functionality" This suggests you may have not 100% test coverage in your tests. But 100% coverage if what? What is the specification you're defining your behaviour against? The comment above suggests that you could treat your tests as if they were the ones that actually define your contract. > Hyrum would like to have a word. This is a reference to "Hyrum's law" which says: "With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody." This comment, in response to the previous about defining tests as the source of truth for your contract, remarks that sadly you can't do that because no matter what contract you wish you define, ultimately the behaviour of your existing software becomes its effective contract. > [My comment about monorepos] Here I suggest that if you extend the notion of what is the test corpus to include the test corpus of all of the software that depend on you (not your dependencies! The code whom your code is a dependency) then you could detect if a (yet unmerged) change you're making is actually going to affect any existing code.
        franga2000 4y ago
        Totally misread the "depend on you" as "you depend on" part, sorry. Interesting idea, but monorepos still don't help on their own. You need a way to detect which modules depend on yours and a way to run and interpret their tests. Whether they're in an adjacent folder or another repo changes very little. GitHub actually already has a dependency scanning thingy that builds cross-repo dependency graphs [0] and package managers have been doing that for years. The missing parts wouldn't be that much effort to build, but getting everyone to set up their repos to work with it would probably be prohibitively difficult. [0] https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/exploring-the-dependencies-of-a-repository#dependents-view https://docs.github.com/en/code-security/supply-chain-securi...
        ithkuil 4y ago
        Monorepos are useless unless you have good tooling, including a build system that is aware of module dependencies inside the monorepo. An example build system that can do that is https://bazel.build https://bazel.build
    - YoshiRulz 4y ago
      That's exactly what the Nixpkgs Hydra instance does—caveat being your library and its consumers all need to be in Nixpkgs. I'm hopeful that there will be a way to keep the tracking part of that going forward, as Flakes are adopted and the community decentralises.

deepsun 4y ago

Not if a simple test requires rewriting hundreds of tests (aka "fragile tests"). Yes, one shouldn't be writing fragile tests, but usually from what I seen at projects with great test coverage, is that it often slows bugfix releases, and especially any bigger changes, as it's very wearisome to also change hundreds of tests. So I believe there should be some balance between tests/code ratio, as well as attention paid to tests brittleness.

2rsf 4y ago
Definitely, some real world cases research show that this balance stands around 80% for unit tests- you don't earn more quality as you add more test above that.
- deepsun 4y ago
  Last time I heard 80% ratio is optimal for Java, but any dynamic typing language (JS, Python, Ruby) it's around 100% test LOC / code LOC.