4 ms·
Nice list, but I think it would be simpler to ask that frameworks support custom outcomes, with user-defined behavior. Most (all?) of your list of outcomes can
by SrslyJosh 3y ago
Nice list, but I think it would be simpler to ask that frameworks support custom outcomes, with user-defined behavior. Most (all?) of your list of outcomes can be boiled down to a few behaviors:
1. Should this result be treated as a pass or a fail?
2. Should a warning of some sort be emitted? (Ex: for FAST and SLOW)
3. Should the result be counted as part of the main test corpus, or as part of some auxiliary grouping?
- o11c 3y agoThe reason enumerations are better than free text is to so tools can handle them in a uniform way. This is a major problem that is keeping TAP from being very useful. For example, if you get MISSING outputs from tests under CI, you know something is wrong with your CI configuration. If you get either kind of WIP output, you shouldn't publish (FSVO publish - some projects might require this for every commit, some only for commits on the main branch, some only on tags). It also makes it easier to diff test results meaningfully.