4 ms·

Yup, it's why I built `just-tap` [1] which trys to minimise as much magic that a lot of these frameworks try to "help" you with. 1. https://github.com/markwyld

Yup, it's why I built `just-tap` [1] which trys to minimise as much magic that a lot of these frameworks try to "help" you with. 1. https://github.com/markwylde/just-tap https://github.com/markwylde/just-tap

autarch 4y ago
TAP is better than some things, but it has some serious issues that I wrote about on my blog a while back - https://blog.urth.org/2017/01/21/tap-is-great-except-when-its-not/ https://blog.urth.org/2017/01/21/tap-is-great-except-when-it... Basically it's nearly impossible to fully parse it correctly.
- hedora 4y ago
  Is test2 a flag for TAP? If you have to pick one or the other, then you're breaking the common flow (human debugging code before pushing) so that management can have better reports. The right solution would be to add a environment variable or CLI parameter that told tap to produce machine readable output, preferably with a separate tool that could convert the machine readable junk to whatever TAP currently writes to stdout/stderr.
  - autarch 4y ago
    Test2 is a Perl distribution that replaces a bunch of older test stuff. See https://metacpan.org/pod/Test2 https://metacpan.org/pod/Test2 But unlike TAP, it's fairly Perl-specific as opposed to just being an output format. I imagine you could adapt the ideas in it to Node but it'd be more complex than simply implement TAP in JS. And yes, I think the idea of having different output formats makes sense. With Test2, the test _harness_ produces TAP from the underlying machine-readable format, rather than having the test code itself directly product TAP. The harness is a separate program that executes the tests.
    - sitkack 4y ago
      What is this madness? Nothing should have to be parsed. Write test results to sqlite, done. You can generate reports directly off those test databases using anything of your choice. your-program test-re.sqlite output.html
      - hedora 4y ago
        Yeah, but sqlite doesn't scale, and SQL isn't a functional language: https://web.archive.org/web/20110114031716/https://browsertoolkit.com/fault-tolerance.png https://web.archive.org/web/20110114031716/https://browserto...
hedora 4y ago
Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to class inheritance, etc. Now, you have a tree of hooks. Document whether it is walked in breadth first or depth first order, then never change the decision (or disallow having trees of hooks, either by detecting them at runtime, or by picking a hook registration mechanism that makes them inexpressible).
- jasonhansel 4y ago
  I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected order.
  - zmj 4y ago
    I’ll disagree with this. Every time I’ve seen that, the interference between tests was also possible between requests in production. I’d rather my test framework give me a 2% chance of noticing the bug than 0%.
    - regularfry 4y ago
      What's annoying is not being able to reproduce the 2% cases so you can't fix it even when you've noticed them. Sensible test tools give you the random seed they used to order the tests so you can reproduce the sequence.
  - thomaslangston 4y ago
    Test order should be random, so that the ability to run them in parallel and distribute them across multiple hosts is not lost by missing enforcement of test isolation.

autarch 4y ago

TAP is better than some things, but it has some serious issues that I wrote about on my blog a while back - https://blog.urth.org/2017/01/21/tap-is-great-except-when-its-not/ https://blog.urth.org/2017/01/21/tap-is-great-except-when-it... Basically it's nearly impossible to fully parse it correctly.

hedora 4y ago
Is test2 a flag for TAP? If you have to pick one or the other, then you're breaking the common flow (human debugging code before pushing) so that management can have better reports. The right solution would be to add a environment variable or CLI parameter that told tap to produce machine readable output, preferably with a separate tool that could convert the machine readable junk to whatever TAP currently writes to stdout/stderr.
- autarch 4y ago
  Test2 is a Perl distribution that replaces a bunch of older test stuff. See https://metacpan.org/pod/Test2 https://metacpan.org/pod/Test2 But unlike TAP, it's fairly Perl-specific as opposed to just being an output format. I imagine you could adapt the ideas in it to Node but it'd be more complex than simply implement TAP in JS. And yes, I think the idea of having different output formats makes sense. With Test2, the test _harness_ produces TAP from the underlying machine-readable format, rather than having the test code itself directly product TAP. The harness is a separate program that executes the tests.
  - sitkack 4y ago
    What is this madness? Nothing should have to be parsed. Write test results to sqlite, done. You can generate reports directly off those test databases using anything of your choice. your-program test-re.sqlite output.html
    - hedora 4y ago
      Yeah, but sqlite doesn't scale, and SQL isn't a functional language: https://web.archive.org/web/20110114031716/https://browsertoolkit.com/fault-tolerance.png https://web.archive.org/web/20110114031716/https://browserto...

hedora 4y ago

Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to class inheritance, etc. Now, you have a tree of hooks. Document whether it is walked in breadth first or depth first order, then never change the decision (or disallow having trees of hooks, either by detecting them at runtime, or by picking a hook registration mechanism that makes them inexpressible).

jasonhansel 4y ago
I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected order.
- zmj 4y ago
  I’ll disagree with this. Every time I’ve seen that, the interference between tests was also possible between requests in production. I’d rather my test framework give me a 2% chance of noticing the bug than 0%.
  - regularfry 4y ago
    What's annoying is not being able to reproduce the 2% cases so you can't fix it even when you've noticed them. Sensible test tools give you the random seed they used to order the tests so you can reproduce the sequence.
- thomaslangston 4y ago
  Test order should be random, so that the ability to run them in parallel and distribute them across multiple hosts is not lost by missing enforcement of test isolation.