8 ms·

The article puts scare quotes around "understand" etc. to try to head off critiques around the lack of precision or scientific language, but I think this is a r

by lsy 2y ago

The article puts scare quotes around "understand" etc. to try to head off critiques around the lack of precision or scientific language, but I think this is a really good example of where casual use of these terms can get pretty misleading. Because code LLMs have been trained on the syntactic form of the program and not its execution, it's not correct — even if the correlation between variable annotations and requested completions was perfect (which it's not) — to say that the model "understands nullability", because nullability means that under execution the variable in question can become null, which is not a state that it's possible for a model trained only on a million programs' syntax to "understand". You could get the same result if e.g. "Optional" means that the variable becomes poisonous and checking "> 0" is eating it, and "!= None" is an antidote. Human programmers can understand nullability because they've hopefully run programs and understand the semantics of making something null. The paper could use precise, scientific language (e.g. "the presence of nullable annotation tokens correlates to activation of vectors corresponding to, and emission of, null-check tokens with high precision and accuracy") which would help us understand what we can rely on the LLM to do and what we can't. But it seems like there is some subconscious incentive to muddy how people see these models in the hopes that we start ascribing things to them that they aren't capable of.

deleted 2y ago
[deleted]
creatonez 2y ago
> Because code LLMs have been trained on the syntactic form of the program and not its execution What makes you think this? It has been trained on plenty of logs and traces, discussion of the behavior of various code, REPL sessions, etc. Code LLMs are trained on all human language and wide swaths of whatever machine-generated text is available, they are not restricted to just code.
uh_uh 2y ago
We don't really have a clue what they are and aren't capable of. Prior to the LLM-boom, many people – and I include myself in this – thought it'd be impossible to get to the level of capability we have now purely from statistical methods and here we are. If you have a strong theory that proves some bounds on LLM-capability, then please put it forward. In the absence of that, your sceptical attitude is just as sus as the article's.
- deleted 2y ago
  [deleted]
- Baeocystin 2y ago
  I majored in CogSci at UCSD in the 90's. I've been interested and active in the machine learning world for decades. The LLM boom took me completely and utterly by surprise, continues to do so, and frankly I am most mystified by the folks who downplay it. These giant matrixes are already so far beyond what we thought was (relatively) easily achievable that even if progress stopped tomorrow, we'd have years of work to put in to understand how we got here. Doesn't mean we've hit AGI, but what we already have is truly remarkable.
  - chihuahua 2y ago
    The funny thing is that 1/3 of people think LLMs are dumb and will never amount to anything. Another third think that it's already too late to prevent the rise of superhuman AGI that will destroy humanity, and are calling for airstrikes on any data center that does not submit to their luddite rules. And the last third use LLMs for writing small pieces of code.
  - Workaccount2 2y ago
    Pretty much until 2022, the de facto orthodoxy for AI was "The creative pursuits will forever be outside the reach of computers". People are pretty quiet about creative pursuits actually being the low hanging fruit on the AI tree.
- kubav027 2y ago
  LLM also have no idea what it is capable of. This feels like difference to humans. Having some understanding of the problem also means knowing or "feeling" the limits of that understanding.
  - deleted 2y ago
    [deleted]
  - uh_uh 2y ago
wvenable 2y ago
> Because code LLMs have been trained on the syntactic form of the program and not its execution One of the very first tests I did of ChatGPT way back when it was new was give it a relatively complex string manipulation function from our code base, strip all identifying materials from the code (variable names, the function name itself, etc), and then provide it with inputs and ask it for the outputs. I was surprised that it could correctly generate the output from the input. So it does have some idea of what the code actually does not just syntax.
waldrews 2y ago
I was going to say "so you believe the LLM's don't have the capacity to understand" but then I realized that the precise language would be something like "the presence of photons in this human's retinas in patterns encoding statements about LLM's having understanding correlates to the activation of neuron signaling chains corresponding to, and emission of, muscle activations engaging keyboard switches, which produce patterns of 'no they don't' with high frequency." The critiques of mental state applied to the LLM's are increasingly applicable to us biologicals, and that's the philosophical abyss we're staring down.
- xigency 2y ago
  This only applies to people who understand how computers and computer programs work, because someone who doesn't externalize their thinking process would never ascribe human elements of consciousness to inanimate materials. Certainly many ancient people worshiped celestial objects or crafted idols by their own hands and ascribed to them powers greater than themselves. That doesn't really help in the long run compared to taking personal responsibility for one's own actions and motives, the best interests of their tribe or community, and taking initiative to understand the underlying cause of mysterious phenomena.
  - deleted 2y ago
    [deleted]
- mjburgess 2y ago
  No it's not. He gave you modal conditions on "understanding", he said: predicting the syntax of valid programs, and their operational semantics, ie., the behaviour of the computer as it runs. I would go much further than this; but this is a de minimus criteria that the LLM already fails. What zealots eventually discover is that they can hold their "fanatical proposition" fixed in the face of all opposition to the contrary, by tearing down the whole edifice of science, knowledge, and reality itself. If you wish to assert, against any reasonable thought, that the sky is a pink dome you can do so -- first that our eyes are broken, and then, eventually that we live in some paranoid "philosophical abyss" carefully constructed to permit your paranoia. This abursidty is exhausting, and I'd wish one day to find fanatics who'd realise it quickly and abate it -- but alas, I have never. If you find yourself hollowing-out the meaning of words to the point of making no distinctions, denying reality to reality itself, and otherwise arriving at a "philosophical abyss" be aware that it is your cherished propositions which are the maddness and nothing else. Here: no, the LLM does not understand. Yes, we do. It is your job to begin from reasonable premises and abduce reasonable theories. If you do not, you will not.
yujzgzc 2y ago
How do you know that these models haven't been trained by running programs? At least, it's likely that they've been trained on undergrad textbooks that explain program behaviors and contain exercises.
aoeusnth1 2y ago
As far as you know, AI labs are doing E2E RL training with running code in the loop to advance the model's capability to act as an agent (for cursor et al).
hatthew 2y ago
I am slowly coming around to the idea that nobody should ever use the word "understand" in relation to LLMs, simply because everyone has their own definition of "understand", and many of these definitions disagree, and people tend to treat their definition as axiomatic. I have yet to see any productive discussion happen once anyone disagrees on the definition of "understand". So, what word would you propose we use to mean "an LLM's ability (or lack thereof) to output generally correct sentences about the topic at hand"?
- nomonnai 2y ago
  It's a prediction of what humans have frequently produced in similar situations.
globnomulous 2y ago
This is essentially John Searle's Chinese Room Argument against strong AI. His conclusion is broader -- he argues categorically against the very possibility of so-called "strong" AI, viz. AI that understands, not just against the narrower notion that LLMs "understand" -- but the reasoning is essentially identical. Here's the Stanford Encyclopedia of Philosophy's superb write up: https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/ It covers, engagingly and with playful wit, not just Searle's original argument but its evolution in his writing, other philosopher's responses/criticisms, and Searle's counter-responses.

by lsy 2y ago

deleted 2y ago

[deleted]

creatonez 2y ago

> Because code LLMs have been trained on the syntactic form of the program and not its execution What makes you think this? It has been trained on plenty of logs and traces, discussion of the behavior of various code, REPL sessions, etc. Code LLMs are trained on all human language and wide swaths of whatever machine-generated text is available, they are not restricted to just code.

uh_uh 2y ago

We don't really have a clue what they are and aren't capable of. Prior to the LLM-boom, many people – and I include myself in this – thought it'd be impossible to get to the level of capability we have now purely from statistical methods and here we are. If you have a strong theory that proves some bounds on LLM-capability, then please put it forward. In the absence of that, your sceptical attitude is just as sus as the article's.

deleted 2y ago
[deleted]
Baeocystin 2y ago
I majored in CogSci at UCSD in the 90's. I've been interested and active in the machine learning world for decades. The LLM boom took me completely and utterly by surprise, continues to do so, and frankly I am most mystified by the folks who downplay it. These giant matrixes are already so far beyond what we thought was (relatively) easily achievable that even if progress stopped tomorrow, we'd have years of work to put in to understand how we got here. Doesn't mean we've hit AGI, but what we already have is truly remarkable.
- chihuahua 2y ago
  The funny thing is that 1/3 of people think LLMs are dumb and will never amount to anything. Another third think that it's already too late to prevent the rise of superhuman AGI that will destroy humanity, and are calling for airstrikes on any data center that does not submit to their luddite rules. And the last third use LLMs for writing small pieces of code.
- Workaccount2 2y ago
  Pretty much until 2022, the de facto orthodoxy for AI was "The creative pursuits will forever be outside the reach of computers". People are pretty quiet about creative pursuits actually being the low hanging fruit on the AI tree.
kubav027 2y ago
LLM also have no idea what it is capable of. This feels like difference to humans. Having some understanding of the problem also means knowing or "feeling" the limits of that understanding.
- deleted 2y ago
  [deleted]
- uh_uh 2y ago

wvenable 2y ago

> Because code LLMs have been trained on the syntactic form of the program and not its execution One of the very first tests I did of ChatGPT way back when it was new was give it a relatively complex string manipulation function from our code base, strip all identifying materials from the code (variable names, the function name itself, etc), and then provide it with inputs and ask it for the outputs. I was surprised that it could correctly generate the output from the input. So it does have some idea of what the code actually does not just syntax.

waldrews 2y ago

I was going to say "so you believe the LLM's don't have the capacity to understand" but then I realized that the precise language would be something like "the presence of photons in this human's retinas in patterns encoding statements about LLM's having understanding correlates to the activation of neuron signaling chains corresponding to, and emission of, muscle activations engaging keyboard switches, which produce patterns of 'no they don't' with high frequency." The critiques of mental state applied to the LLM's are increasingly applicable to us biologicals, and that's the philosophical abyss we're staring down.

xigency 2y ago
This only applies to people who understand how computers and computer programs work, because someone who doesn't externalize their thinking process would never ascribe human elements of consciousness to inanimate materials. Certainly many ancient people worshiped celestial objects or crafted idols by their own hands and ascribed to them powers greater than themselves. That doesn't really help in the long run compared to taking personal responsibility for one's own actions and motives, the best interests of their tribe or community, and taking initiative to understand the underlying cause of mysterious phenomena.
- deleted 2y ago
  [deleted]
mjburgess 2y ago
No it's not. He gave you modal conditions on "understanding", he said: predicting the syntax of valid programs, and their operational semantics, ie., the behaviour of the computer as it runs. I would go much further than this; but this is a de minimus criteria that the LLM already fails. What zealots eventually discover is that they can hold their "fanatical proposition" fixed in the face of all opposition to the contrary, by tearing down the whole edifice of science, knowledge, and reality itself. If you wish to assert, against any reasonable thought, that the sky is a pink dome you can do so -- first that our eyes are broken, and then, eventually that we live in some paranoid "philosophical abyss" carefully constructed to permit your paranoia. This abursidty is exhausting, and I'd wish one day to find fanatics who'd realise it quickly and abate it -- but alas, I have never. If you find yourself hollowing-out the meaning of words to the point of making no distinctions, denying reality to reality itself, and otherwise arriving at a "philosophical abyss" be aware that it is your cherished propositions which are the maddness and nothing else. Here: no, the LLM does not understand. Yes, we do. It is your job to begin from reasonable premises and abduce reasonable theories. If you do not, you will not.

yujzgzc 2y ago

How do you know that these models haven't been trained by running programs? At least, it's likely that they've been trained on undergrad textbooks that explain program behaviors and contain exercises.

aoeusnth1 2y ago

As far as you know, AI labs are doing E2E RL training with running code in the loop to advance the model's capability to act as an agent (for cursor et al).

hatthew 2y ago

I am slowly coming around to the idea that nobody should ever use the word "understand" in relation to LLMs, simply because everyone has their own definition of "understand", and many of these definitions disagree, and people tend to treat their definition as axiomatic. I have yet to see any productive discussion happen once anyone disagrees on the definition of "understand". So, what word would you propose we use to mean "an LLM's ability (or lack thereof) to output generally correct sentences about the topic at hand"?

nomonnai 2y ago
It's a prediction of what humans have frequently produced in similar situations.

globnomulous 2y ago

This is essentially John Searle's Chinese Room Argument against strong AI. His conclusion is broader -- he argues categorically against the very possibility of so-called "strong" AI, viz. AI that understands, not just against the narrower notion that LLMs "understand" -- but the reasoning is essentially identical. Here's the Stanford Encyclopedia of Philosophy's superb write up: https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/ It covers, engagingly and with playful wit, not just Searle's original argument but its evolution in his writing, other philosopher's responses/criticisms, and Searle's counter-responses.