> who is to say the change will positively impact future predictions (arguably all that matters to the business)? My point is that you are the person who can d

> who is to say the change will positively impact future predictions (arguably all that matters to the business)? My point is that you are the person who can do this. Not up front, but you have the ability to look at your past prediction of future outcomes and see if what you're doing is actually working or not. You already had the predictions. You now know the actual outcomes. How does the performance metrics of your model's predictions from back then on your validation set compare to the performance of your models when they were used for real? How would the prior day's model do. Or the prior year's?