13 ms·
You mean, comparing the performance after training a model?
by mjhirn 11y ago
You mean, comparing the performance after training a model?
- reality_hacker 11y agoPerformance in terms of model accuracy. How accurate will be those models.
- hobofan 11y agoNo accuracy is not included, since that is something that should be constant for each model across frameworks. These benchmarks aim to highlight the performance differences in terms of speed/memory usage across frameworks and machine configurations. There is also the practical hurdle that training imagenet models to maximum accuracy takes 1 week+.
- reality_hacker 11y agoSo, are you saying that output of all frameworks are exactly and always the same? Sorry, if I am asking stupid questions.
- hobofan 11y agoNot exactly, no. But not even the output of the same framework will always be the same since you are usually randomly initalizing the weights in a network and randomly picking the samples used in SGD[1] (the seed for the RNG could of course be a fixed one to mitigate this somewhat). But in the end, if you are using the same model, the same solver and the same RNG, yes the output of all frameworks should be the same. In practice this also mostly holds true, since the stochastic processes involved are geared towards finding a good local minimum, which is the same given a model and a dataset. [1]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent https://en.wikipedia.org/wiki/Stochastic_gradient_descent
- reality_hacker 11y agoSo, there is a strong chance that you are comparing very different things, and frameworks which perform not good in terms of speed can be far superior in terms of models quality.