For feature analysis (should we enable this new feature) I think Bayesian approaches are far better. I work in a slightly different space, which is qualificati

For feature analysis (should we enable this new feature) I think Bayesian approaches are far better. I work in a slightly different space, which is qualification of builds (this assumes you have feature flags and code, and the code shouldn't enable features, but it may include refactoring, as well as the code that will later be enabled by a flag etc.) For this, I originally wanted to push my org to do things in a more Bayesian, but it didn't work. What did work was forcing org leaders to sit down and actually decide how costly true positives were, and therefore how much developer time they were willing to sink into chasing ghosts. If you can say "we expect to spend 4 hours of developer time investigating an alert here", and 1/10 alerts will be real at our current sensitivity, then you can decide if these things are alright. Ultimately we can't control all the variables (our sensitivity requirements are informed by things like SLOs that we don't directly control), but it does help make informed decisions about prioritization. The type m/s stuff is really neat, although in my space we don't care too much about those, although that may be because we have conventionally gargantuan sample sizes.