3 ms·
There's extra randomness added accidentally in practice: inference is a massively parallelized set of matrix multiplications, and floating point math is not com
by TeMPOraL 2y ago
There's extra randomness added accidentally in practice: inference is a massively parallelized set of matrix multiplications, and floating point math is not commutative - the randomness in execution order gets converted into a random FP error, so even setting temperature to 0 doesn't guarantee repeatable results.
- HeatrayEnjoyer 2y agoOnly if the inference software doesn't guarantee concurrency, which is CS 101
- pizza 2y agoThis sort of nondeterministic scheduling of non associative floating point ops is essentially running at the level of GPU firmware, so, I would imagine that in this case, Nvidia is aware.