3 ms·
> well-understood imprecision that applies evenly over results I'm sorry, I have some bad news for you... especially the "evenly" part. I had a friend who wor
by crabbone 2y ago
> well-understood imprecision that applies evenly over results
I'm sorry, I have some bad news for you... especially the "evenly" part. I had a friend who worked at IBM research on the problem of "normalizing" some matrix multiplication irregularities which occur due to numerical underflow or overflow. And, it's anything but a solved or an easy problem.
- dzaima 2y agoThere a maximum error of 0.5 ulp for all of add/sub/mul/div/sqrt on all inputs (ok sin/cos & co typically have a couple ulp of error but close enough). And indeed it's well-understood that it can get problematic if you compose a bunch of such, esp. add/sub with a large magnitude difference. Indeed, especially matrix multiplication.