4 ms·

What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698 https://

What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698 https://youtu.be/Y2F8yisiS6E?t=4698

sipjca 3y ago
yes
deleted 3y ago
[deleted]
fancyfredbot 3y ago
That's right.There was mention of a precision aware transformer engine which might make it easier to use fp4, but it's not 30x faster in a like for like way. This shouldn't be surprising since it's more or less two hoppers next to one another on a slightly improved process node. 2.5x seems more likely in cases where you don't exploit a new feature like that or the increased memory.
Havoc 3y ago
>bit misleading. Only partially, because in LLMs FP4 isn't half as useful as FP8. So if you have gear that crushes at FP4 then that's what you use and you benefit from that increased speed (at minimal accuracy loss). Definitely some marketing creativity in there, but its not entirely wrong as a measure of real world usage
- jxy 3y ago
  curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16bit or INT8 for actual computations.
  - buildbot 3y ago
    Llama.cpp and many others support 4 bit weights and lower
  - creshal 3y ago
    Half the reason why they move back up to 8/16 bit is that current hardware doesn't properly support 4 bit floats, and you get better performance from the conversion. I think once this hardware hits, most of the computation will shift to native 4 bit just for efficency's sake. ...assuming the recent 1.58b paper doesn't render the entire float quantization approach obsolete by then.
    - imtringued 3y ago
      The 1.58b approach is good for everyone including for quantization. It means that current quantization schemes have room for improvement.
      - creshal 3y ago
        It's good for everyone on the software side, but it's not so good for the hardware side, because it means that whatever you're designing now for tapeout in 6 months and release in 12 is going to be obsolete in 2 weeks.
s_m_t 3y ago
How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?
- deleted 3y ago
  [deleted]
- wongarsu 3y ago
  For training FP4 sounds pretty niche, but for inference it might be very useful.
- WhitneyLand 3y ago
  - Training isn’t done at 4-bits, to date this small size has only been for inference. - Research for a while now has been finding that smaller weights are surprisingly effective. It’s kind of a counterintuitive result, but one way to think about it is there are billions of weights working together. So taken as a whole you still have a large amount of information.
  - acchow 3y ago
    Intuitively, there is a ton of redundancy and we still have a long way we can still compress things.
    - imtringued 3y ago
      Each token is represented by a vector of 4096 floats. Of course there is redundancy.
  - coffeebeqn 3y ago
    Maybe the rounding errors are noise that is somewhat useful in a big enough neutral net. Image generators also generate noise to work on
  - tmalsburg2 3y ago
    > - Training isn’t done at 4-bits, to date this small size has only been for inference. Wasn't there a paper from Microsoft two weeks ago or so where they trained on log₂(3) bits? Edit: https://arxiv.org/pdf/2402.17764.pdf https://arxiv.org/pdf/2402.17764.pdf
wongarsu 3y ago
It's 4 bit floating point, at twice the speed of 8 bit floating point. There's also FP6, doesn't offer faster compute than FP8 but manages to take advantage of the better memory bandwith and cache use of the 6 bit format. Apparently some people are drawing connections to this paper [1] on 4 bit LLMs, which has one NVIDIA employee among its contributors 1: https://arxiv.org/pdf/2310.16836.pdf https://arxiv.org/pdf/2310.16836.pdf
opcode84 3y ago
https://arxiv.org/pdf/2310.10537.pdf https://arxiv.org/pdf/2310.10537.pdf Discussed in a previous post https://news.ycombinator.com/item?id=37930663 https://news.ycombinator.com/item?id=37930663
- buildbot 3y ago
  Yes this is it!

What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698 https://youtu.be/Y2F8yisiS6E?t=4698

sipjca 3y ago

yes

deleted 3y ago

[deleted]

fancyfredbot 3y ago

That's right.There was mention of a precision aware transformer engine which might make it easier to use fp4, but it's not 30x faster in a like for like way. This shouldn't be surprising since it's more or less two hoppers next to one another on a slightly improved process node. 2.5x seems more likely in cases where you don't exploit a new feature like that or the increased memory.

Havoc 3y ago

>bit misleading. Only partially, because in LLMs FP4 isn't half as useful as FP8. So if you have gear that crushes at FP4 then that's what you use and you benefit from that increased speed (at minimal accuracy loss). Definitely some marketing creativity in there, but its not entirely wrong as a measure of real world usage

jxy 3y ago
curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16bit or INT8 for actual computations.
- buildbot 3y ago
  Llama.cpp and many others support 4 bit weights and lower
- creshal 3y ago
  Half the reason why they move back up to 8/16 bit is that current hardware doesn't properly support 4 bit floats, and you get better performance from the conversion. I think once this hardware hits, most of the computation will shift to native 4 bit just for efficency's sake. ...assuming the recent 1.58b paper doesn't render the entire float quantization approach obsolete by then.
  - imtringued 3y ago
    The 1.58b approach is good for everyone including for quantization. It means that current quantization schemes have room for improvement.
    - creshal 3y ago
      It's good for everyone on the software side, but it's not so good for the hardware side, because it means that whatever you're designing now for tapeout in 6 months and release in 12 is going to be obsolete in 2 weeks.

s_m_t 3y ago

How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?

deleted 3y ago
[deleted]
wongarsu 3y ago
For training FP4 sounds pretty niche, but for inference it might be very useful.
WhitneyLand 3y ago
- Training isn’t done at 4-bits, to date this small size has only been for inference. - Research for a while now has been finding that smaller weights are surprisingly effective. It’s kind of a counterintuitive result, but one way to think about it is there are billions of weights working together. So taken as a whole you still have a large amount of information.
- acchow 3y ago
  Intuitively, there is a ton of redundancy and we still have a long way we can still compress things.
  - imtringued 3y ago
    Each token is represented by a vector of 4096 floats. Of course there is redundancy.
- coffeebeqn 3y ago
  Maybe the rounding errors are noise that is somewhat useful in a big enough neutral net. Image generators also generate noise to work on
- tmalsburg2 3y ago
  > - Training isn’t done at 4-bits, to date this small size has only been for inference. Wasn't there a paper from Microsoft two weeks ago or so where they trained on log₂(3) bits? Edit: https://arxiv.org/pdf/2402.17764.pdf https://arxiv.org/pdf/2402.17764.pdf

wongarsu 3y ago

It's 4 bit floating point, at twice the speed of 8 bit floating point. There's also FP6, doesn't offer faster compute than FP8 but manages to take advantage of the better memory bandwith and cache use of the 6 bit format. Apparently some people are drawing connections to this paper [1] on 4 bit LLMs, which has one NVIDIA employee among its contributors 1: https://arxiv.org/pdf/2310.16836.pdf https://arxiv.org/pdf/2310.16836.pdf

opcode84 3y ago

https://arxiv.org/pdf/2310.10537.pdf https://arxiv.org/pdf/2310.10537.pdf Discussed in a previous post https://news.ycombinator.com/item?id=37930663 https://news.ycombinator.com/item?id=37930663

buildbot 3y ago
Yes this is it!