3 ms·
Who actually cares about FP64 for AI? All I've seen has been going the other way, ie. bfloat16 or even 8-bit floats. Since neural transition functions are larg
by FeepingCreature 5y ago
Who actually cares about FP64 for AI? All I've seen has been going the other way, ie. bfloat16 or even 8-bit floats.
Since neural transition functions are largely characterized by a single point of inflection, it seems hard to see how you'd make 32 bits of mantissa pay rent even in principle.
- philjohn 5y agoThese aren't for AI workloads, they're for things like simulating atomic blasts.
- MayeulC 5y agoOr space simulations: https://spaceengine.org/news/blog210611/ https://spaceengine.org/news/blog210611/ > FP64 is extremely slow on Nvidia and AMD GPU, thanks to market segmentation: full performance with FP64 is enabled only on “professional” cards like Quadro. So for consumer hardware, we implemented FP64 emulation using two FP32 numbers. Performance is still not great, but it’s much better than using hardware FP64.
- touisteur 5y agoI think they market those fp64 things for the simulation market, which is basically pretty much every supercomputer's mission (I'm simplifying). The A100 also has specific silicium for fp64 (so called dp tensor cores) that you can't find in lower end datacenter stuff. FP64 really stings even on (e.g) mid-end nvidia hw, what with the 1:32 handbrake. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#arithmetic-instructions__throughput-native-arithmetic-instructions https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.... for reference