4 ms·
"Can someone describe a real life production scenario in which this flaky behavior is desirable?" I think so? My company develops a processor that is meant to
by ghettoimp 11y ago
"Can someone describe a real life production scenario in which this flaky behavior is desirable?"
I think so? My company develops a processor that is meant to be compatible with processors from other vendors. To try to ensure compatibility, we have a test suite that compares our "golden model" of intended behavior against the observed behavior of competitors' chips.
We sometimes have run into cases where a competitor's product "randomly" gets wrong results, but when we run the instruction again, it gets the right answer. This happens frequently enough that we've arranged the test suite to automatically try again to see if a failure is reproducible before bothering a human with it.
- danso 11y agoYeah I figured these kind of errors fall into the system interoperability/integration category...but I also figured they would fall into their own kind of test suite, one that is much more geared toward the measurements of thresholds and probabilities. Adding a "flaky" plugin to the standard test framework to do this kind of non-standard testing...feels like designing the workflow in a slightly backwards way, like monkeypatching a basic data object for very specific behavior needed in a few niche libraries.