3 ms·

Is this an issue of code stability or do these libraries have leaky abstractions that cause issues with models trained on previous version, because the library

by xmprt 4y ago

Is this an issue of code stability or do these libraries have leaky abstractions that cause issues with models trained on previous version, because the library implementation changed and since ML has low explainability, it's hard to tell how the change impacts the model performance? eg. making a library run faster or use a slightly different parameter internally causes some floating point imprecision that cascades into wildly different results?

alfalfasprout 4y ago
Unfortunately it's all of the above. Sometimes the code isn't stable (eg; a library method changed), sometimes the behavior has changed (eg; same library method but now it has a different behavior), and sometimes it has some other unwanted effect (the computation produces a totally different result, it runs way slower, or something else). With a service it's all much easier because you can have pretty robust integration tests and you're good to go. With an ML model it's a lot trickier because even if you have sample payloads and responses there might be other payloads you didn't think about that could cause issues. Or it might rear its ugly head when you finally retrain the model and suddenly model performance is crap.
dotnet00 4y ago
There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This then breaks models when the framework updates. One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues. I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.

by xmprt 4y ago

alfalfasprout 4y ago

Unfortunately it's all of the above. Sometimes the code isn't stable (eg; a library method changed), sometimes the behavior has changed (eg; same library method but now it has a different behavior), and sometimes it has some other unwanted effect (the computation produces a totally different result, it runs way slower, or something else). With a service it's all much easier because you can have pretty robust integration tests and you're good to go. With an ML model it's a lot trickier because even if you have sample payloads and responses there might be other payloads you didn't think about that could cause issues. Or it might rear its ugly head when you finally retrain the model and suddenly model performance is crap.

dotnet00 4y ago

There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This then breaks models when the framework updates. One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues. I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.