3 ms·
If you can train with 8 bit ints, then FP8 should be fine: https://arxiv.org/abs/2009.13108 https://arxiv.org/abs/2009.13108 A group at IBM has been claiming f
by sbierwagen 4y ago
If you can train with 8 bit ints, then FP8 should be fine: https://arxiv.org/abs/2009.13108 https://arxiv.org/abs/2009.13108
A group at IBM has been claiming for years that you can train with as little as FP4: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8cb45877d577-Paper.pdf https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...
- modeless 4y agoThere have been a lot of papers on techniques for training in low precision, even as low as binary. But the largest models that would benefit most aren't using those techniques, which indicates to me that they don't work as well as their paper abstracts suggest.
- sdenton4 4y agoMy experience with quantization is that a little is easy, and a lot takes work. Quantization aware training helps, but it's finicky and adds extra steps to the training process, which in turn slows down iteration speed.