3 ms·
All of those are supported by TAP, they're just not top-level keywords. Just start your test's description with your "CUSTOM-KEYWORD", then aggregate on that,
by avar 3y ago
All of those are supported by TAP, they're just not top-level keywords.
Just start your test's description with your "CUSTOM-KEYWORD", then aggregate on that, instead of "ok/not ok".
You can even easily plug into prove(1) to emit your own custom summary.
You can aggregate on anything, e.g. you could consider all tests taking longer than 500ms failures, regardless of top-level "ok" status.
Your use-case is rather niche, so it's not a failure of the protocol that it provides a boolean "ok/not ok" at the top-level, and not "failed known flaky" or whatever (although that's usually marked as "TODO" test).
It's intentionally minimal exactly to make it easy to support the sort of use cases you're describing here.
- o11c 3y agoIf all the standard-following tooling doesn't know about or support it automatically, it's not reasonable to claim it's "supported by TAP". If the only controllable level of granularity is "run a whole executable file", you don't have control. Actually running all tests is about the least interesting thing you can do when testing.
- pastage 3y agoTap can be emitted by anything, when you start modules etc. The reason you have 1...3 at the start is to be able to check when tests fail to run, you can have multiple of these.