4 ms·

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after. Is the ~13 digit limitation due to the model itself (how/how well

by kloch 4y ago

> you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after. Is the ~13 digit limitation due to the model itself (how/how well it was trained) or simply the use of double precision in the model weights (which maxes out around 15 digits of precision).

kristjansson 4y ago
Additional evidence against the second hypothesis: almost nothing in LLM-land is double precision anyway, weights are generally half-precision (or something like bfloat16 with more range by less precision than IEEE float16).
famouswaffles 4y ago
It's hard to say for sure but the second is pretty unlikely.
tel 4y ago
In order for it to be the second you'd need to assume that at least some part of the critical reasoning of the LLM involves storing the data of the number in a single model activation. This is pretty unlikely, as models tend to store information across many activations simultaneously. I don't know this for a fact, you'd need to do brain surgery on GPT-4 and it'd be hard even in that case, but most studies of ANN processing would suggest that the null hypothesis is to assume that the information is widely distributed and not sigfig limited in that way.
- jychang 4y ago
  There are exceptions https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a...
  - tel 4y ago
    Sort of. I hadn't read that one and just skimmed it, but I think it falls into the same category of "while we can find compelling activation patterns even at a single neuron level, it's hard to rule out other representations". On the flip side is Chris Olah's research (e.g. https://distill.pub/2017/feature-visualization/ https://distill.pub/2017/feature-visualization/) which maps a single neuron's activation into the output space (by optimizing for outputs which would have arising from high activations of given neurons). If we take this seriously and call these outputs the "favored" output of an individual neuron, we might believe we've characterized that neuron, but it's really still the entire downstream computation from that neuron and there are still substantial overlaps and redundancies. These are exactly where I'm getting the sense that the standard rule should be that neural nets have very distributed representations. Even more to the point, these analyses tend to focus on "binary" questions, either a feature is present or not. Or, said another way, they implicitly treat activation as a "vote", where greater activation implies a greater component weighting of that which that neuron represents. This, to my mind, flies in the face of GPT-n doing arithmetic, though. I'd lean much closer to the argument that each individual quantity is represented as a neuron as opposed to each quantity is represented continuously by the actual activation _value_. This "voting" representation is also reinforced by the common use of tanh-like activation functions. There's a long history of having neuron activations be probabilistic/logit-like quantities. Of course, I cannot speak for certain about any neural net and especially not for GPT-3/4. I just have a hard time taking as the null hypothesis that it is storing magnitudes of numbers in a way that is directly affected by the float precision in use.
shagie 4y ago
While its more token costly, the tokenization of numbers gets weird. Taking "123" it becomes [123]. "1234" becomes [12][34]. "12345" becomes [123][45]. Consider using spaces between each digit to force them to be distinct tokens. "1 2 3 4 5" is tokenized as [1][ 2][ 3][ 4][ 5] It may then be possible to work with the digits more easily because they're not getting tokenized in potentially odd ways that are harder to manipulate.

by kloch 4y ago

kristjansson 4y ago

Additional evidence against the second hypothesis: almost nothing in LLM-land is double precision anyway, weights are generally half-precision (or something like bfloat16 with more range by less precision than IEEE float16).

famouswaffles 4y ago

It's hard to say for sure but the second is pretty unlikely.

tel 4y ago

In order for it to be the second you'd need to assume that at least some part of the critical reasoning of the LLM involves storing the data of the number in a single model activation. This is pretty unlikely, as models tend to store information across many activations simultaneously. I don't know this for a fact, you'd need to do brain surgery on GPT-4 and it'd be hard even in that case, but most studies of ANN processing would suggest that the null hypothesis is to assume that the information is widely distributed and not sigfig limited in that way.

jychang 4y ago
There are exceptions https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-a...
- tel 4y ago
  Sort of. I hadn't read that one and just skimmed it, but I think it falls into the same category of "while we can find compelling activation patterns even at a single neuron level, it's hard to rule out other representations". On the flip side is Chris Olah's research (e.g. https://distill.pub/2017/feature-visualization/ https://distill.pub/2017/feature-visualization/) which maps a single neuron's activation into the output space (by optimizing for outputs which would have arising from high activations of given neurons). If we take this seriously and call these outputs the "favored" output of an individual neuron, we might believe we've characterized that neuron, but it's really still the entire downstream computation from that neuron and there are still substantial overlaps and redundancies. These are exactly where I'm getting the sense that the standard rule should be that neural nets have very distributed representations. Even more to the point, these analyses tend to focus on "binary" questions, either a feature is present or not. Or, said another way, they implicitly treat activation as a "vote", where greater activation implies a greater component weighting of that which that neuron represents. This, to my mind, flies in the face of GPT-n doing arithmetic, though. I'd lean much closer to the argument that each individual quantity is represented as a neuron as opposed to each quantity is represented continuously by the actual activation _value_. This "voting" representation is also reinforced by the common use of tanh-like activation functions. There's a long history of having neuron activations be probabilistic/logit-like quantities. Of course, I cannot speak for certain about any neural net and especially not for GPT-3/4. I just have a hard time taking as the null hypothesis that it is storing magnitudes of numbers in a way that is directly affected by the float precision in use.

shagie 4y ago

While its more token costly, the tokenization of numbers gets weird. Taking "123" it becomes [123]. "1234" becomes [12][34]. "12345" becomes [123][45]. Consider using spaces between each digit to force them to be distinct tokens. "1 2 3 4 5" is tokenized as [1][ 2][ 3][ 4][ 5] It may then be possible to work with the digits more easily because they're not getting tokenized in potentially odd ways that are harder to manipulate.