In my day job I manage ~8 deep learning models that make decisions automatically. I adjust 3 inputs, and the models output a lot of data that I use to make adju

In my day job I manage ~8 deep learning models that make decisions automatically. I adjust 3 inputs, and the models output a lot of data that I use to make adjustments the next day. The issue is.. its incredibly muddy. Making the same change across models, according to documentation, will produce similar results. Having only had this job for a couple months I honestly cannot tell if the input -> output is just very very muddy, or if the documentation is wrong and the models all behave differently. Similar to Crawl I can't see behind the curtain.. my inputs are generic "factor 1", "factor 2" and my outputs have more meaning but are subject to unknowable fluctuations from outside sources. > holding the system constant enough that you can evaluate your changes over time I do not know if I can learn from the system because I'm not sure if its trustworthy. Good thing there are directions on how to test it included in the post :)