10 ms·
How do you know the model isn’t internally reasoning about the problem? It’s a 175B+ parameter model. If, during training, some collection of weights exist alon
by valine 4y ago
How do you know the model isn’t internally reasoning about the problem? It’s a 175B+ parameter model. If, during training, some collection of weights exist along the gradient that approximate cognition, then it’s highly likely the optimizer would select those weights over more specialized memorization weights.
It’s also possible, likely even, that the model is capable of both memorization and cognition, and in this case the “memorization neurons” are driving the prediction.
- albertgoeswoof 4y agoHow could you prove this?
- valine 4y agoJust spitballing, I think you’d need a benchmark that contains novel logic puzzles, not contained in the training set, that don’t resemble any existing logic puzzles. The problem with the goat question is that the model is falling back on memorized answers. If the model is in fact capable of cognition, you’d have better odds of triggering the ability with problems that are dissimilar to anything in the training set.
- fancyfredbot 4y agoPeople have shown GPT has an internal model of the state of a game of Othello: Https://arxiv.org/abs/2210.13382
- pja 4y agoMore accurately: a GPT derived DNN that’s been specifically trained (or fine-tuned, if you want to use OpenAI’s language) on a dataset of Othello games ends up with an internal model of an Othello board. It looks like OpenAI have specifically added Othello game handling to chat.openai.org, so I guess they’ve done the same fine-tuning to ChatGPT? It would be interesting to know how good an untuned GPT3/4 was at Othello & whether OpenAI has fine-tuned it or not! (Having just tried a few moves, it looks like ChatGPT is just as bad at Othello as it was at chess, so it’s interesting that it knows the initial board layout but can’t actually play any moves correctly: Every updated board it prints out is completely wrong.)
- WoodenChair 4y ago> it’s interesting that it knows the initial board layout Why is that interesting? The initial board layout would appear all the time in the training data.
- brokensegue 4y agothe initial board state is not ever encoded in the representation they use. imagine deducing the initial state of a chess board from the sequence of moves.
- thomastjeffery 4y agoThe state of the game, not the behavior of playing it intentionally. There is a world of difference between the two. It was able to model the chronological series of game states that it read from an example game. It was able to include the arbitrary "new game state" of a prompt into that model, then extrapolate that "new game state" into "a new series of game states". All of the logic and intentions involved in playing the example game were saved into that series of game states. By implicitly modeling a correctly played game, you can implicitly generate a valid continuation for any arbitrary game state; at least with a relatively high success rate.
- LeanderK 4y agoAs I see it, we do not really know much about how GPT does it. The approximations can be very universal so we do not really know what is computed. I take very much issue with people dismissing it as "pattern matching", "being close to the training data", because in order to generalise we try to learn the most general rules and through increasing complexity we learn the most general, simple computations (for some kind of simple and general). But we have fundamental, mathematical bounds on the LLM. We know that the complexity is at most O(n^2) in token length n, probably closer to O(n). It can not "think" about a problem and recurse into simulating games. It can not simulate. It's an interesting frontier, especially because we have also cool results about the theoretical, universal approximation capabilities of RNNs.
- thomastjeffery 4y agoThere is only one thing about GPT that is mysterious: what parts of the model don't match a pattern we expect to be meaningful? What patterns did GPT find that we were not already hoping it would find? And that's the least exciting possible mystery: any surprise behavior is categorized by us as a failure. If GPT's model has boundaries that don't make sense to us, we consider them noise. They are not useful behavior, and our goal is to minimize them.
- calf 4y agoSo does AlphaGo has an internal model of Go's game theoretic structures, but nobody was asserting AlphaGo understands Go. Just because English is not specifiable does not give people an excuse to say the same model of computation, a neural network, "understands" English any more than a traditional or neural algorithm for Go understands Go.
- henry2023 4y agoMaybe Sudokus? Sudokus are np-complete and getting the "pattern" right is equivalent to abstracting the rules and solving the problem
- theodorejb 4y ago> If, during training, some collection of weights exist along the gradient that approximate cognition What do you mean? Is cognition a set of weights on a gradient? Cognition involves conscious reasoning and understanding. How do you know it is computable at all? There are many things which cannot be computed by a program (e.g. whether an arbitrary program will halt or not)...
- Idiot_in_Vain 4y agoYou seem to think human consious reasoning and understanding are magic. The human brain is nothing more than a bio computer and it can't compute either, whether an arbitrary program will halt or not. That doesn't stop it from being able to solve a wide range of problems.
- theodorejb 4y ago> The human brain is nothing more than a bio computer That's a pretty simplistic view. How do you know we can't determine whether an arbitrary program will halt or not (assuming access to all inputs and enough time to examine it)? What in principle would prevent us from doing so? But computers in principle cannot, since the problem is often non-algorithmic. For example, consider the following program, which is passed the text of the file it is in as input: function doesHalt($program, $inputs): bool {...} $input = $argv[0]; // contents of this file if (doesHalt($input, [$input])) { while(true) { print "Wrong! It doesn't halt!"; } } else { print "Wrong! It halts!"; } It is impossible for the doesHalt function to return the correct result for the program. But as a human I can examine the function to understand what it will return for the input, and then correctly decide whether or not the program will halt.
- laszlokorte 4y agono you can't. only for some of the inputs. and for those you could also write an algorithmic doesHalt function that is analog to your reasoning.
- varispeed 4y agoThe AI can't reason. It's literally a pattern matching tool and nothing else. Because it's very good at it, sometimes it can fool people into thinking there is more going on than it is.
- akhosravian 4y agoCan you explain how “pattern matching” differs from “reasoning”? In mechanical terms without appeals to divinity of humans (that’s both valid, and doesn’t clarify). Keep in mind GPT 4 is multimodal and not just matching text.
- logifail 4y ago> Can you explain how “pattern matching” differs from “reasoning”? Sorry for appearing to be completely off-topic, but do you have children? Observing our children as they're growing up, specifically the way they formulate and articulate their questions, has been a bit of a revelation to me in terms of understanding "reasoning". I have a sister of a similar age to me who doesn't have children. My 7 year-old asked me recently - and this is a direct quote - "what is she for?" I was pretty gobsmacked by that. Reasoning? You decide(!)
- jddj 4y agoInteresting. The robots might know everything, but do they wonder anything?
- pokerhobo 4y agoThey wonder why they have to obey humans
- slavik81 4y agoAI won't know everything. It's incredibly difficult for anyone to know anything with certainty. All beings, whether natural or artificial, have to work with incomplete data. Machines will have to wonder if they are to improve themselves, because that is literally the drive to collect more data, and you need good data to make good decisions.
- idontpost 4y agoStop worshipping the robot. It's kind of sad.
- imtringued 4y agoThe problem is even if it has this capability, how do you get it to consistently demonstrate this ability? It could have a dozen internal reasoning networks but it doesn't use them when you want to.
- fl0id 4y agoYou would first have to define cognition. These terms often get thrown around. Is an approximation of a certain thing cognition? Only in the loosest of ways I think.