There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This the

There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This then breaks models when the framework updates. One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues. I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.