Honestly that's my experience with ML in general. And I know any organization that does a lot of ML has things like backtests and holdouts to supposedly measure

Honestly that's my experience with ML in general. And I know any organization that does a lot of ML has things like backtests and holdouts to supposedly measure impact on key metrics but I've seen too many cases where adding X has a +10% impact in some top line metric and then removing it later (and returning to the initial state) has another 5% increase in that same top line metric. Maybe we need to revise the famous phrase to: Lies, damned lies, statistics and machine learning.