7 ms·

Although I'm sympathetic to the author's argument, I don't think they've found the best way to frame it. I have two main objections i.e. points I guess LLM advo

by n4r9 2y ago

Although I'm sympathetic to the author's argument, I don't think they've found the best way to frame it. I have two main objections i.e. points I guess LLM advocates might dispute. Firstly: > LLMs are capable of appearing to have a theory about a program ... but it’s, charitably, illusion. To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. Secondly: > Theories are developed by doing the work and LLMs do not do the work Isn't this a little... anthropocentric? That's the way humans develop theories. In principle, could a theory not be developed by transmitting information into someone's brain patterns as if they had done the work?

IanCal 2y ago
Skipping that they say it's fallacious at the start, none of the arguments in the article are valid if you simply have models 1. Run code 2. Communicate with POs 3. Iteratively write code
- n4r9 2y ago
  I thought the fallacy bit was tongue-in-cheek. They're not actually arguing from authority in the article. The system you describe appears to treat programmers as mere cogs. Programmers do not simply write and iterate code as dictated by POs. That's a terrible system for all but the simplest of products. We could implement that system, then lose the ability to make broad architectural improvements, effectively adapt the model to new circumstances, or fix bugs that the model cannot.
  - IanCal 2y ago
    > The system you describe appears to treat programmers as mere cogs Not at all, it simply addresses key issues raised. That they cannot have a theory of the program because they are reading it and not actually writing it - so have them write code, fix problems and iterate. Have them communicate with others to get more understanding of the "why". > . Programmers do not simply write and iterate code as dictated by POs. Communicating with POs is not the same as writing code directed by POs.
    - n4r9 2y ago
      Oh, I think I see. You're imagining LLMs that learn from PO feedback as they go?
      - IanCal 2y ago
        This can be as simple as giving them search over communication with a PO, and giving them a place to store information that's searchable. How good they are at this is a different matter but the article claims it is impossible because they don't work on the code and build an understanding like people do and cannot gain that by just reading code.
Jensson 2y ago
> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. Human theory building works, we have demonstrated this, our science letting us build things on top of things proves it. LLM theory building so far doesn't, they always veer in a wrong direction after a few steps, you will need to prove that LLM can build theories just like we proved that humans can.
- jerf 2y ago
  You can't prove LLMs can build theories like humans can, because we can effectively prove they can't. Most code bases do not fit in a context window. And any "theory" an LLM might build about a code base, analogously to the recent reasoning models, itself has to carve a chunk out of the context window, at what would have to be a fairly non-trivial percentage expansion of tokens versus the underlying code base, and there's already not enough tokens. There's no way that is big enough to build a theory of a code base. "Building a theory" is something I expect the next generation of AIs to do, something that has some sort of memory that isn't just a bigger and bigger context window. As I often observe, LLMs != AI. The fact that an LLM by its nature can't build a model of a program doesn't mean that some future AI can't.
  - imtringued 2y ago
    This is correct. The model context is a form of short term memory. It turns out LLMs have an incredible short term memory, but simultaneously that is all they have. What I personally find perplexing is that we are still stuck at having a single context window. Everyone knows that turing machines with two tapes require significantly fewer operations than a single tape turning machine that needs to simulate multiple tapes. The reasoning stuff should be thrown into a separate context window that is not subject to training loss (only the final answer).
    - fouc 2y ago
      Or have at least 2 models. Each with their own dedicated context.
- 2y ago
ryandv 2y ago
> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. This idea has already been explored by thought experiments such as John Searle's so-called "Chinese room" [0]; an LLM cannot have a theory about a program, any more than the computer in Searle's "Chinese room" understands "Chinese" by using lookup tables to generate canned responses to an input prompt. One says the computer lacks "intentionality" regarding the topics that the LLM ostensibly appears to be discussing. Their words aren't "about" anything, they don't represent concepts or ideas or physical phenomena the same way the words and thoughts of a human do. The computer doesn't actually "understand Chinese" the way a human can. [0] https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
- CamperBob2 2y ago
  You're seriously still going to invoke the Chinese Room argument after what we've seen lately? Wow. The computer understands Chinese better than Searle (or anyone else) understood the nature and functionality of language.
  - ryandv 2y ago
    You're seriously going to invoke this braindead reddit-tier of "argumentation," or rather lack thereof, by claiming bewilderment and offering zero substantive points? Wow.
    - CamperBob2 2y ago
      Yes, because the Chinese Room was a weak test the day it was proposed, and it's a heap of smoldering rhetorical wreckage now. It's Searle who failed to offer any substantive points. How do you know you're not arguing with an LLM at the moment? You don't... any more than I do.
      - ryandv 2y ago
        > How do you know you're not arguing with an LLM at the moment? You don't. I wish I was right now. It would probably provide at least the semblance of greater insight into these topics. > the Chinese Room was a weak test the day it was proposed Why?
psychoslave 2y ago
> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. That burden of proof is on you, since you are presumably human and you are challenging the need of humans to have more than a mere appearance of having a theory when they claim to have one. Note that even when the only theoretical assumption we go with is that we will have a good laugh watching other people going crazy after random bullshits thrown at them, we still have a theory.
dcre 2y ago
I agree. Of course you can learn and use a theory without having developed it yourself!
jimbokun 2y ago
He doesn't prove the claim. But he does make a strong argument for why it's very unlikely that an LLM would have a theory of a program similar to what a human author of a program would have: > Theories are developed by doing the work and LLMs do not do the work. They ingest the output of work. And this is certainly a true statement about how LLMs are constructed. Maybe this latently induces in the LLM something very similar to what humans do when writing programs. But another possibility is that it's similar to the Brain Teasers that were popular for a long time in programming interviews. The idea was that if the interviewee could use logic to solve riddles, they were probably also likely to be good at writing programs. In reality, it was mostly a test of whether the interviewee had reviewed all the popular riddles commonly asked in these interviews. If they had, they could also produce a realistic chain of logic to simulate the process of solving the riddle from first principles. But if that same interviewee was given a riddle not similar to one they had previously reviewed, they probably wouldn't do nearly as well in solving it. It's very likely that LLMs are like those interviewees who crammed a lot of examples, again due to how LLMs are trained. They can reproduce programs similar to ones in their training set. They can even produce explanations for their "reasoning" based on examples they've seen of explanations of why a program was written in one way instead of another. But that is a very different kind of model than the one a person builds up writing a program from scratch over a long period of time. Having said all this, I'm not sure what experiments you would run to determine if the LLM is using one approach vs another.

by n4r9 2y ago

IanCal 2y ago

Skipping that they say it's fallacious at the start, none of the arguments in the article are valid if you simply have models 1. Run code 2. Communicate with POs 3. Iteratively write code

n4r9 2y ago
I thought the fallacy bit was tongue-in-cheek. They're not actually arguing from authority in the article. The system you describe appears to treat programmers as mere cogs. Programmers do not simply write and iterate code as dictated by POs. That's a terrible system for all but the simplest of products. We could implement that system, then lose the ability to make broad architectural improvements, effectively adapt the model to new circumstances, or fix bugs that the model cannot.
- IanCal 2y ago
  > The system you describe appears to treat programmers as mere cogs Not at all, it simply addresses key issues raised. That they cannot have a theory of the program because they are reading it and not actually writing it - so have them write code, fix problems and iterate. Have them communicate with others to get more understanding of the "why". > . Programmers do not simply write and iterate code as dictated by POs. Communicating with POs is not the same as writing code directed by POs.
  - n4r9 2y ago
    Oh, I think I see. You're imagining LLMs that learn from PO feedback as they go?
    - IanCal 2y ago
      This can be as simple as giving them search over communication with a PO, and giving them a place to store information that's searchable. How good they are at this is a different matter but the article claims it is impossible because they don't work on the code and build an understanding like people do and cannot gain that by just reading code.

Jensson 2y ago

> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. Human theory building works, we have demonstrated this, our science letting us build things on top of things proves it. LLM theory building so far doesn't, they always veer in a wrong direction after a few steps, you will need to prove that LLM can build theories just like we proved that humans can.

jerf 2y ago
You can't prove LLMs can build theories like humans can, because we can effectively prove they can't. Most code bases do not fit in a context window. And any "theory" an LLM might build about a code base, analogously to the recent reasoning models, itself has to carve a chunk out of the context window, at what would have to be a fairly non-trivial percentage expansion of tokens versus the underlying code base, and there's already not enough tokens. There's no way that is big enough to build a theory of a code base. "Building a theory" is something I expect the next generation of AIs to do, something that has some sort of memory that isn't just a bigger and bigger context window. As I often observe, LLMs != AI. The fact that an LLM by its nature can't build a model of a program doesn't mean that some future AI can't.
- imtringued 2y ago
  This is correct. The model context is a form of short term memory. It turns out LLMs have an incredible short term memory, but simultaneously that is all they have. What I personally find perplexing is that we are still stuck at having a single context window. Everyone knows that turing machines with two tapes require significantly fewer operations than a single tape turning machine that needs to simulate multiple tapes. The reasoning stuff should be thrown into a separate context window that is not subject to training loss (only the final answer).
  - fouc 2y ago
    Or have at least 2 models. Each with their own dedicated context.
2y ago

ryandv 2y ago

> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. This idea has already been explored by thought experiments such as John Searle's so-called "Chinese room" [0]; an LLM cannot have a theory about a program, any more than the computer in Searle's "Chinese room" understands "Chinese" by using lookup tables to generate canned responses to an input prompt. One says the computer lacks "intentionality" regarding the topics that the LLM ostensibly appears to be discussing. Their words aren't "about" anything, they don't represent concepts or ideas or physical phenomena the same way the words and thoughts of a human do. The computer doesn't actually "understand Chinese" the way a human can. [0] https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

CamperBob2 2y ago
You're seriously still going to invoke the Chinese Room argument after what we've seen lately? Wow. The computer understands Chinese better than Searle (or anyone else) understood the nature and functionality of language.
- ryandv 2y ago
  You're seriously going to invoke this braindead reddit-tier of "argumentation," or rather lack thereof, by claiming bewilderment and offering zero substantive points? Wow.
  - CamperBob2 2y ago
    Yes, because the Chinese Room was a weak test the day it was proposed, and it's a heap of smoldering rhetorical wreckage now. It's Searle who failed to offer any substantive points. How do you know you're not arguing with an LLM at the moment? You don't... any more than I do.
    - ryandv 2y ago
      > How do you know you're not arguing with an LLM at the moment? You don't. I wish I was right now. It would probably provide at least the semblance of greater insight into these topics. > the Chinese Room was a weak test the day it was proposed Why?

psychoslave 2y ago

> To make this point stick, you would also have to show why it's not an illusion when humans "appear" to have a theory. That burden of proof is on you, since you are presumably human and you are challenging the need of humans to have more than a mere appearance of having a theory when they claim to have one. Note that even when the only theoretical assumption we go with is that we will have a good laugh watching other people going crazy after random bullshits thrown at them, we still have a theory.

dcre 2y ago

I agree. Of course you can learn and use a theory without having developed it yourself!

jimbokun 2y ago

He doesn't prove the claim. But he does make a strong argument for why it's very unlikely that an LLM would have a theory of a program similar to what a human author of a program would have: > Theories are developed by doing the work and LLMs do not do the work. They ingest the output of work. And this is certainly a true statement about how LLMs are constructed. Maybe this latently induces in the LLM something very similar to what humans do when writing programs. But another possibility is that it's similar to the Brain Teasers that were popular for a long time in programming interviews. The idea was that if the interviewee could use logic to solve riddles, they were probably also likely to be good at writing programs. In reality, it was mostly a test of whether the interviewee had reviewed all the popular riddles commonly asked in these interviews. If they had, they could also produce a realistic chain of logic to simulate the process of solving the riddle from first principles. But if that same interviewee was given a riddle not similar to one they had previously reviewed, they probably wouldn't do nearly as well in solving it. It's very likely that LLMs are like those interviewees who crammed a lot of examples, again due to how LLMs are trained. They can reproduce programs similar to ones in their training set. They can even produce explanations for their "reasoning" based on examples they've seen of explanations of why a program was written in one way instead of another. But that is a very different kind of model than the one a person builds up writing a program from scratch over a long period of time. Having said all this, I'm not sure what experiments you would run to determine if the LLM is using one approach vs another.