3 ms·

This is the problem with LLM researchers all but giving up on the problem of inspecting how the LLM actually works internally. As long as the LLM is a black bo

This is the problem with LLM researchers all but giving up on the problem of inspecting how the LLM actually works internally. As long as the LLM is a black box, its entirely possible that (a) the LLM does reason through the rules and understands what moves are legal or (b) was trained on a large set of legal moves and therefore only learned to make legal moves. You can claim either case is the real truth, but we have absolutely no way to know because we have absolutely no way to actually understand what the LLM was "thinking".

codeulike 2y ago
Here's an article where they teach an LLM Othello and then probe its internal state to assess whether it is 'modelling' the Othello board internally https://thegradient.pub/othello/ https://thegradient.pub/othello/ Associated paper: https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2210.13382
- deleted 2y ago
  [deleted]
mattmcknight 2y ago
It's weird because it is not a black box at the lowest level, we can see exactly what all of the weights are doing. It's just too complex for us to understand it. What is difficult is finding some intermediate pattern in between there which we can label with an abstraction that is compatible with human understanding. It may not exist. For example, it may be more like how our brain works to produce language than it is like a logical rule based system. We occasionally say the wrong word, skip a word, spell things wrong...violate the rules of grammar. The inputs and outputs of the model are human language, so at least there we know the system as a black box can be characterized, if not understood.
- _heimdall 2y ago
  > The inputs and outputs of the model are human language, so at least there we know the system as a black box can be characterized, if not understood. This is actually where the AI safety debates tend to lose. From where I sit we can't characterize the black box itself, we can only characterize the outputs themselves. More specifically, we can decide what we think the quality of the output for the given input and we can attempt to infer what might have happened in between. We really have no idea what happened in between, and though many of the "doomers" raise concerns that seem far fetched, we have absolutely no way of understanding whether they are completely off base or raising concerns of a system that just hasn't shown problems in the input/output pairs yet.
lukeschlather 2y ago
> (a) the LLM does reason through the rules and understands what moves are legal or (b) was trained on a large set of legal moves and therefore only learned to make legal moves. How can you learn to make legal moves without understanding what moves are legal?
- _heimdall 2y ago
  I'm spit balling here so definitely take this with a grain of salt. If I only see legal moves, I may not think outside the box come up with moves other than what I already saw. Humans run into this all the time, we see things done a certain and effectively learn that that's just how to do it and we don't innovate. Said differently, if the generative AI isn't actually being generative at all, meaning its just predicting based on the training set, it could be providing only legal moves without ever learning or understanding the rules of the game.
  - adelineJoOs 2y ago
    I am not a ML person, and know there is an mathematical explanation for what I am about to write, but here comes my informal reasoning: I fear this is not the case: 1) Either, the LLM (or other forms of deep neural networks) can reproduce exactly what it saw, but nothing new (then it would only produce legal moves, if it was trained on only legal ones) 2) Or, the LLM can produce moves that it did not exactly see, by outputting the "most probable" looking move in that situation (which it never has seen before). In effect, this is combining different situations and their output into a new output. As a result of this "mixing", it might output an illegal move (= the output move is illegal in this new situation), despite having been trained on only legal moves. In fact, I am not even sure if the deep neuronal networks we use in practice even can replicate their training data exactly - it seems to me that there is some kind of compression going on by embedding knowledge into the network, which will come with a loss. I am deeply convinced that LLMs will never be exact technology (but LLMs + other technology like proof assistants or compilers might be)
    - _heimdall 2y ago
raincole 2y ago
> we have absolutely no way to know To me, this means that it absolutely doesn't matter whether LLM does reason or not.
- _heimdall 2y ago
  It might if AI/LLM safety is a concern. We can't begin to really judge safety without understanding how they work internally.

codeulike 2y ago

Here's an article where they teach an LLM Othello and then probe its internal state to assess whether it is 'modelling' the Othello board internally https://thegradient.pub/othello/ https://thegradient.pub/othello/ Associated paper: https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2210.13382

deleted 2y ago
[deleted]

mattmcknight 2y ago

It's weird because it is not a black box at the lowest level, we can see exactly what all of the weights are doing. It's just too complex for us to understand it. What is difficult is finding some intermediate pattern in between there which we can label with an abstraction that is compatible with human understanding. It may not exist. For example, it may be more like how our brain works to produce language than it is like a logical rule based system. We occasionally say the wrong word, skip a word, spell things wrong...violate the rules of grammar. The inputs and outputs of the model are human language, so at least there we know the system as a black box can be characterized, if not understood.

_heimdall 2y ago
> The inputs and outputs of the model are human language, so at least there we know the system as a black box can be characterized, if not understood. This is actually where the AI safety debates tend to lose. From where I sit we can't characterize the black box itself, we can only characterize the outputs themselves. More specifically, we can decide what we think the quality of the output for the given input and we can attempt to infer what might have happened in between. We really have no idea what happened in between, and though many of the "doomers" raise concerns that seem far fetched, we have absolutely no way of understanding whether they are completely off base or raising concerns of a system that just hasn't shown problems in the input/output pairs yet.

lukeschlather 2y ago

> (a) the LLM does reason through the rules and understands what moves are legal or (b) was trained on a large set of legal moves and therefore only learned to make legal moves. How can you learn to make legal moves without understanding what moves are legal?

_heimdall 2y ago
I'm spit balling here so definitely take this with a grain of salt. If I only see legal moves, I may not think outside the box come up with moves other than what I already saw. Humans run into this all the time, we see things done a certain and effectively learn that that's just how to do it and we don't innovate. Said differently, if the generative AI isn't actually being generative at all, meaning its just predicting based on the training set, it could be providing only legal moves without ever learning or understanding the rules of the game.
- adelineJoOs 2y ago
  I am not a ML person, and know there is an mathematical explanation for what I am about to write, but here comes my informal reasoning: I fear this is not the case: 1) Either, the LLM (or other forms of deep neural networks) can reproduce exactly what it saw, but nothing new (then it would only produce legal moves, if it was trained on only legal ones) 2) Or, the LLM can produce moves that it did not exactly see, by outputting the "most probable" looking move in that situation (which it never has seen before). In effect, this is combining different situations and their output into a new output. As a result of this "mixing", it might output an illegal move (= the output move is illegal in this new situation), despite having been trained on only legal moves. In fact, I am not even sure if the deep neuronal networks we use in practice even can replicate their training data exactly - it seems to me that there is some kind of compression going on by embedding knowledge into the network, which will come with a loss. I am deeply convinced that LLMs will never be exact technology (but LLMs + other technology like proof assistants or compilers might be)
  - _heimdall 2y ago

raincole 2y ago

> we have absolutely no way to know To me, this means that it absolutely doesn't matter whether LLM does reason or not.

_heimdall 2y ago
It might if AI/LLM safety is a concern. We can't begin to really judge safety without understanding how they work internally.