3 ms·
Correct me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
by spencerchubb 2y ago
Correct me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
- jebarker 2y agoI don't think that's always correct when you're talking about operators in neural nets. E.g. the sin and cos in rope embeddings would get more precise, large sums like softmax would become more precise, potentially attention too due to dot products