4 ms·
Google uses int8 for LLM training
- ianbicking 3y agoThis makes me think: the math behind neural nets is kind of arbitrary. The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want. That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals.
- throwbadubadu 3y ago> built on "math" or "numbers" but could be built on bits That reads strange, what is the difference besides the range of numbers?
- Hydraulix989 3y agoMaybe bitwise operations instead of arithmetic?
- riscy 3y agoarithmetic is implemented as bitwise operations so i’m still confused. the lower the precision / smaller range of values you choose to represent means fewer bitwise operations are required.
- Hydraulix989 3y agoMaybe something like this: https://arxiv.org/abs/1601.06071 https://arxiv.org/abs/1601.06071
- blastonico 3y agoI would call it applied statistics :-D
- david-gpu 3y agoI used to design low-precision hardware to train AIs. > The actually important thing is (a) getting a result out, and (b) being able to nudge the weights to get the result closer to what you want That is accurate. > That doesn't have to be built on "math" or "numbers" but could be built on bits directly so long as the operations satisfy the goals Some people have published research where the parameters or activations are binary, sometimes in a stochastic fashion. Typically either the AI doesn't converge or it converges to a rather suboptimal solution. Also, just because the weights or activations are binary it doesn't mean there's no math involved. You need some sort of a mathematical model to figure out how to tweak the parameters of the model in order to improve its fitness to the data.
- sigmoid10 3y agoThe keyword the original comment was missing is differentiability. This is what a) and b) imply because it is how you ensure that you can train your model using gradient descent or whatever. So you need a particular kind of math to train these models using the methods we have today. There's quite a bit of research into other methods for training neural networks that don't come with this constraint, because it's obvious the human brain doesn't work that way. But this is early research and very far from being as useful.
- stevenhuang 3y agoI wouldn't say it's obvious the brain doesn't work this way. See the free energy principle, one of the leading theories of how our brain works. > The free energy principle is a theoretical framework suggesting that the brain reduces surprise or uncertainty by making predictions based on internal models and updating them using sensory input. It highlights the brain's objective of aligning its internal model with the external world to enhance prediction accuracy Making predictions and then correcting for them in some way is a pretty low bar and it's often achieved through differentiability. I wouldn't be surprised if relevant neurological structures exist in our brains that do this.
- sigmoid10 3y ago
- _nalply 3y agoProbably even 4 bit values suffice. Invent some binary operations that exhibit some smoothness or continuousness such that gradient descent still works, because mostly a small change should have only a small effect. I think mathematicians would say functions that are differentiable.
- tayo42 3y agoQuantization is done with 4 bits, unless you meant something else?
- _nalply 3y agoPerhaps quantization with 4 bits is a special case of what I meant. With 4 bits you have 16 different weight values but these values must not be quantized. They could be anything provided gradient descent is possible. Have some combination of binary operations instead of mathematical operations. What these operations do exactly, does not really matter, as long as gradient descent/backpropagation stays possible. In other words, the model will find a way to use the weights correctly.
- wilg 3y agoElon's always prattling on about how Tesla does this for FSD and it is a big advantage but I'm too dumb to know if it matters. Thoughts?
- bananapub 3y agogiven FSD doesn't actually exist still, I can't think why trying to decode whatever lies the CEO says about the process of failing to make that product is a good use of time
- wilg 3y agoi would like to understand the pros and cons of int8?
- reitzensteinm 3y agoDo you have a link to Tesla training with int8?
- wilg 3y agoOh I guess it’s inference. Not an expert here at all. https://x.com/elonmusk/status/1695376562055774664?s=46 https://x.com/elonmusk/status/1695376562055774664?s=46
- ganzuul 3y agoI want to see an example where high bit precision is needed. Could audio be it? Touch?
- laughingcurve 3y agoAcademic in the field: My answer is usually highly complex scientific simulations, military hardware, and non-euclidean mathematics. There's a trade-off space between precision and efficiency, so it depends on the constraints of the solution space.
- juliangoldsmith 3y agoHigh precision would matter near the input and output layers, but further into the model you could always substitute more parameters for precision.
- xrd 3y agoCan someone explain what is missing in my mental model of the significance of this? Why does it matter than one is an int and one is a float? I understand that this means a different range of values and that floats are manipulated differently and with different costs on GPU and CPU because of the way they are stored. Aren't LLMs basically just probability stages, so why does it matter if the values are floats or ints, other than to note that more space for the data gives you more fine grained probabilities? Is this difference mainly to suggest that Google uses different hardware that is more suitable for processing ints and that is a competitive advantage somehow?