I disagree (greatly) about testing machine learning code. I test ruthlessly my pure functions and even full stack stuff, but of course my full-data experiments

I disagree (greatly) about testing machine learning code. I test ruthlessly my pure functions and even full stack stuff, but of course my full-data experiments require interpretation. The difference is huge! Without very solid separation between those two phases, and then without very solid actual testing, you cannot achieve trust or repeatability in your experiments. It's actually turning into a pretty interesting problem (that I've patched together solutions using Fabric and Org-mode) to track experiments to versions of code while maintaining good testing practices.