6 ms·

I feel like there should be a LLM architecture which includes "scratch space" - tokens the model can write to and read from which do not constitute part of its

by Imnimo 4y ago

I feel like there should be a LLM architecture which includes "scratch space" - tokens the model can write to and read from which do not constitute part of its output. The trouble with current architectures is that they can only do a finite amount of computation per output token - they get one forward pass and then have to output something. Chain-of-thought reasoning allows the model to devote more computation to finding the answer, storing intermediate results in its output tokens. But this is silly - most of the intermediate tokens are not providing useful information towards solving the problem, they're just wasted computation: >There are 16 balls in total. >Half of the balls are golf balls. >That means that there are 8 golf balls. >Half of the golf balls are blue. >That means that there are 4 blue golf balls. For the number of forward passes being done to generate this text, only a few tokens are actually helpful - most are grammatical filler. Further, the model is losing information by being forced to project its state down to a single output token. Even more, the most probable one-step output may not even be the most informative or helpful! It'd be much nicer if the model could write arbitrary, continuous-valued tokens to a private scratch space and then attend to those tokens as though they were words in the prompt while generating the actual output, potentially performing several forward passes per output token when necessary. In short, if chain-of-thought prompting is such a good idea, we should bake it into the model. Obviously all of this is FAR easier said than done.

heyitsguay 4y ago
Is this something one could try to quickly implement alongside NanoGPT? Seems like a pretty straightforward, concrete idea, once you decide where you want those tokens to fit into downstream attention layer inputs. Evaluating relative performance on a small scale could give indication of if it's worth trying at larger scales, unless it's one of those things that doesn't help until your model is huge.
skybrian 4y ago
On the other hand, if it represents scratch space in English, it's a lot easier to see how it justifies its answer and to tell where it's gone wrong. Debuggability seems pretty important? Maybe it just needs more training at "thinking out loud" so it does it without prompting?
rcme 4y ago
The model already contains “scratch space” via its billions of parameters.
- imtringued 4y ago
  Parameters are not updated during inference. Training ANNs is still a single shot exercise.
  - rcme 4y ago
    Sure, parameters are not updated, but ANNs are universal approximates, so they can model whatever it is you envision this “scratch space” doing. Think about it like this: whatever gets put in to the scratch space would need to be deterministic based on the inputs, i.e. it would just be a store if some intermediate value computed by the network. So how would it fundamentally differ from the network itself. I guess what I’m saying is that I’d want an explanation how how this scratch space fundamentally differs from the network itself. It’s almost like you’re assuming the network is “thinking” and that giving it a pad of paper would help it reason better.
dwohnitmok 4y ago
> arbitrary, continuous-valued tokens to a private scratch space I'm with skybrian. Please don't use private scratch spaces. The one saving grace of current LLMs when it comes to understand them is that they still generally need to "think out loud" by outputting more text. Remove that functionality and you end up with a truly inscrutable black box and that has very terrible implications for AI interpretability with knock-on effects for AI safety.
- echelon 4y ago
  > AI safety Is it really that big of a deal if AI leapfrogs us? Everyone else in the field is worried about safety, alignment, and bias. Google used this excuse to execute slowly. Now they've got the "deer in headlights" look, with their single biggest cash cow clearly in the cross hairs. And here I am excited by the possibility of AI out-evolving us.
  - jstummbillig 4y ago
    If everyone else in a field is worried, and you have no unfair advantage or special insight, and you are not willing to move into conspiracy theories, I think there's a good hint as to what approach is most reasonable right now. Might of course turn out to have been completely off, later. Still, maybe one of those occasions where you really don't want to "oops" it.
    - StarlaAtNight 4y ago
      wait, what?
  - travisjungroth 4y ago
    The experience of any group on Earth that runs into a more capable peer has generally not been good. Humans wiping out megafauna. Civilizations colonizing other civilizations. Invasive species of plants and animals. It is not a situation I would hope humanity to get thrown into carelessly.
  - ben_w 4y ago
nestorD 4y ago
The scratchspace could be in natural language, preserving some debugeability and letting us know about the model mental process. This is doable but it introduce a sequential dependency which would make the training significantly slower.
blackbear_ 4y ago
You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.
cma 4y ago
Didn't Facebook's Galatica model use scratch space?
- acallaha 4y ago
  Yes, IIUC it had something like a separate scratch space, and training examples training it to "think" in terms of symbolic expressions and python programs. See section 3.1.1 here: https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf Example from the paper below: Question: A needle 35 mm long rests on a water surface at 20◦C. What force over and above the needle’s weight is required to lift the needle from contact with the water surface? σ = 0.0728m. <work> σ = 0.0728 N/m σ = F/L 0.0728 = F/(2 × 0.035) F = 0.0728(2 × 0.035) calculate.py ‘‘‘ f = 0.0728*(2*0.035) with open("output.txt", "w") as file: file.write(str(round(f, 5))) ‘‘‘ «run: "calculate.py"> «read: "output.txt"» 0.0051 </work> Answer: F = 0.0051 N

by Imnimo 4y ago

heyitsguay 4y ago

Is this something one could try to quickly implement alongside NanoGPT? Seems like a pretty straightforward, concrete idea, once you decide where you want those tokens to fit into downstream attention layer inputs. Evaluating relative performance on a small scale could give indication of if it's worth trying at larger scales, unless it's one of those things that doesn't help until your model is huge.

skybrian 4y ago

On the other hand, if it represents scratch space in English, it's a lot easier to see how it justifies its answer and to tell where it's gone wrong. Debuggability seems pretty important? Maybe it just needs more training at "thinking out loud" so it does it without prompting?

rcme 4y ago

The model already contains “scratch space” via its billions of parameters.

imtringued 4y ago
Parameters are not updated during inference. Training ANNs is still a single shot exercise.
- rcme 4y ago
  Sure, parameters are not updated, but ANNs are universal approximates, so they can model whatever it is you envision this “scratch space” doing. Think about it like this: whatever gets put in to the scratch space would need to be deterministic based on the inputs, i.e. it would just be a store if some intermediate value computed by the network. So how would it fundamentally differ from the network itself. I guess what I’m saying is that I’d want an explanation how how this scratch space fundamentally differs from the network itself. It’s almost like you’re assuming the network is “thinking” and that giving it a pad of paper would help it reason better.

dwohnitmok 4y ago

> arbitrary, continuous-valued tokens to a private scratch space I'm with skybrian. Please don't use private scratch spaces. The one saving grace of current LLMs when it comes to understand them is that they still generally need to "think out loud" by outputting more text. Remove that functionality and you end up with a truly inscrutable black box and that has very terrible implications for AI interpretability with knock-on effects for AI safety.

echelon 4y ago
> AI safety Is it really that big of a deal if AI leapfrogs us? Everyone else in the field is worried about safety, alignment, and bias. Google used this excuse to execute slowly. Now they've got the "deer in headlights" look, with their single biggest cash cow clearly in the cross hairs. And here I am excited by the possibility of AI out-evolving us.
- jstummbillig 4y ago
  If everyone else in a field is worried, and you have no unfair advantage or special insight, and you are not willing to move into conspiracy theories, I think there's a good hint as to what approach is most reasonable right now. Might of course turn out to have been completely off, later. Still, maybe one of those occasions where you really don't want to "oops" it.
  - StarlaAtNight 4y ago
    wait, what?
- travisjungroth 4y ago
  The experience of any group on Earth that runs into a more capable peer has generally not been good. Humans wiping out megafauna. Civilizations colonizing other civilizations. Invasive species of plants and animals. It is not a situation I would hope humanity to get thrown into carelessly.
- ben_w 4y ago

nestorD 4y ago

The scratchspace could be in natural language, preserving some debugeability and letting us know about the model mental process. This is doable but it introduce a sequential dependency which would make the training significantly slower.

blackbear_ 4y ago

You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.

cma 4y ago

Didn't Facebook's Galatica model use scratch space?

acallaha 4y ago
Yes, IIUC it had something like a separate scratch space, and training examples training it to "think" in terms of symbolic expressions and python programs. See section 3.1.1 here: https://galactica.org/static/paper.pdf https://galactica.org/static/paper.pdf Example from the paper below: Question: A needle 35 mm long rests on a water surface at 20◦C. What force over and above the needle’s weight is required to lift the needle from contact with the water surface? σ = 0.0728m. <work> σ = 0.0728 N/m σ = F/L 0.0728 = F/(2 × 0.035) F = 0.0728(2 × 0.035) calculate.py ‘‘‘ f = 0.0728*(2*0.035) with open("output.txt", "w") as file: file.write(str(round(f, 5))) ‘‘‘ «run: "calculate.py"> «read: "output.txt"» 0.0051 </work> Answer: F = 0.0051 N