4 ms·

For a more digestible introduction, see the companion blog post: https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandb

For a more digestible introduction, see the companion blog post: https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandbytes-integration

xrd 3y ago
This is really interesting. Can someone help me understand why this is important: This means that in BF16 we can retain the same dynamic range as FP32. But we lose 3 bits of precision with respect to FP16. Now there is absolutely no problem with huge numbers, but the precision is worse than FP16 here. Why is it important with LLMs and transformers that you can keep track of very big numbers and very small numbers? It feels like you should be able to deal with only big numbers or only tiny numbers when running the models? Why does precision matter? If I've used anything incorrectly here, please correct me.
- colejohnson66 3y ago
  I don't think they're claiming precision loss is a bad thing, but instead is just pointing out that BF16, in exchange for more "dynamic range" (eight bit exponent instead of five), you lose some precision. After all, BF16 is just F32, but with a truncated significand.
  - xrd 3y ago
    Yes, that much I do understand. I'm just confused why that is important when predicting the next token? What about the calculation requires that you keep track of really big numbers (and don't overflow) and requires that you have very precise numbers (and don't truncate the precision)?
    - xeonmc 3y ago
      Because AI is an alchemical art involving heuristically selecting arcane “activation functions” without critically thinking about precision distribution or symmetry, so they rely on band-aiding over it via altering number formats rather than properly matching the function to the appropriate numeric domain.
    - deleted 3y ago
      [deleted]
- Lerc 3y ago
  I can see areas where being able to adjust weights to a wide range of sensitivities is important. Take the original transformer diagram for example. https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/06/Transformer.png https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/06... See the arrows coming in from the sides of the "Add & Norm" boxes. These come from bypassing the layer earlier. This gives back-propagation a easier path to adjust the weights earlier in the model. The results of the inputs are added, if the sensitivity were comparable to both inputs to the 'Add and Norm' a strong bypass signal would swamp the information from the path with more processing. The bypass allows for the easy early training to travel back through the bypass channels, but as the network develops more nuance, the longer path will provide additional accuracy, as this improves, the strength of the bypass signal can be reduced and while still remaining part of the overall signal it probably ends up way less significant than the longer path. A larger range of weights would allow this combination of shouting and whispers. note: not an expert, grain of salt etc.

For a more digestible introduction, see the companion blog post: https://huggingface.co/blog/hf-bitsandbytes-integration https://huggingface.co/blog/hf-bitsandbytes-integration

xrd 3y ago

This is really interesting. Can someone help me understand why this is important: This means that in BF16 we can retain the same dynamic range as FP32. But we lose 3 bits of precision with respect to FP16. Now there is absolutely no problem with huge numbers, but the precision is worse than FP16 here. Why is it important with LLMs and transformers that you can keep track of very big numbers and very small numbers? It feels like you should be able to deal with only big numbers or only tiny numbers when running the models? Why does precision matter? If I've used anything incorrectly here, please correct me.

colejohnson66 3y ago
I don't think they're claiming precision loss is a bad thing, but instead is just pointing out that BF16, in exchange for more "dynamic range" (eight bit exponent instead of five), you lose some precision. After all, BF16 is just F32, but with a truncated significand.
- xrd 3y ago
  Yes, that much I do understand. I'm just confused why that is important when predicting the next token? What about the calculation requires that you keep track of really big numbers (and don't overflow) and requires that you have very precise numbers (and don't truncate the precision)?
  - xeonmc 3y ago
    Because AI is an alchemical art involving heuristically selecting arcane “activation functions” without critically thinking about precision distribution or symmetry, so they rely on band-aiding over it via altering number formats rather than properly matching the function to the appropriate numeric domain.
  - deleted 3y ago
    [deleted]
Lerc 3y ago
I can see areas where being able to adjust weights to a wide range of sensitivities is important. Take the original transformer diagram for example. https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/06/Transformer.png https://i0.wp.com/bdtechtalks.com/wp-content/uploads/2020/06... See the arrows coming in from the sides of the "Add & Norm" boxes. These come from bypassing the layer earlier. This gives back-propagation a easier path to adjust the weights earlier in the model. The results of the inputs are added, if the sensitivity were comparable to both inputs to the 'Add and Norm' a strong bypass signal would swamp the information from the path with more processing. The bypass allows for the easy early training to travel back through the bypass channels, but as the network develops more nuance, the longer path will provide additional accuracy, as this improves, the strength of the bypass signal can be reduced and while still remaining part of the overall signal it probably ends up way less significant than the longer path. A larger range of weights would allow this combination of shouting and whispers. note: not an expert, grain of salt etc.