I once wasted several months of schedule and payroll wrestling to improve a DNN model - which resulted in that model getting much much better thanks to tons of

I once wasted several months of schedule and payroll wrestling to improve a DNN model - which resulted in that model getting much much better thanks to tons of creative and interesting work. We swapped the learned model for a perfect a oracle and realized the actual problem was elsewhere in the system and was easily fixed. None of those model enhancements were needed.