What you need is a form of moderation. An actual group of workers who's job is to collect data, validate a test case is reproducible, or at least that there's

What you need is a form of moderation. An actual group of workers who's job is to collect data, validate a test case is reproducible, or at least that there's telemetry of a problem having happened. In edge cases they could at least collect some data and try to isolate where a problem might exist so that better tests / data collection can occur.