3 ms·

> "they can only produce code that's almost identical to what's in their training data" Again, you're misinterpreting in a way that seems like you are reacting

> "they can only produce code that's almost identical to what's in their training data" Again, you're misinterpreting in a way that seems like you are reacting to the perception that someone attacked some of your core beliefs rather than considering what I am saying and conversing about that. I never even used the words "exact same thing" or "almost identical". Not even synonyms. I just said overfitting and quoted from an OpenAI/Anthropic paper that said "predict plausible changes to code from examples of changes" Think about that. Don't react, think. Why do you equate overfitting and plausibility prediction with "exact" and "identical". It very obviously is not what I said. What I am getting at is that a cannon will kill the mosquito. But drawing a fly swatter in the cannonball and saying the plastic ones are obsolete now would be in bad faith. No need to say to someone pointing that out that they are claiming that the cannon can only fire on mosquitoes that have been swatted before.

simonw 2y ago
I don't think I understood your point then. I matched it with the common "LLMs can only produce code that's similar to what they've seen before" argument. Reading back, you said: > I often see people wondering if the some coding task is performed well or not because of availability of code examples in the training data. It's way worse than that. It's overfitting to diffs it was trained on. I'll be honest: I don't understand what you mean by "overfitting to diffs it was trained on" there. Maybe I don't understand what "overfitting" means in this context? (I'm afraid I didn't understand your cannon / fly swatter analogy either.)
- namaria 2y ago
  It's overkill. The models do not capture knowledge about coding. They overfit to the dataset. When one distills data into a useful model the model can be used to predict future behavior of the system. That is the premise of LLM-as-AI. By training these models on enough data, knowledge of the world is purported as having been captured, creating something useful that can be leveraged to process new input and get a prediction of the trajectory of the system in some phase space. But this, I argue, is not the case. The models merely overfit to the training data. Hence the variable results perceived by people. When their intentions and prompt fit to the data in the training, the model appears to give good output. But the situation and prompt do not, the models do no "reason" about it and "infer" anything. It fails. It gives you gibberish or go in circles, or worse if there is some "agentic" arrangement if fails to terminate and burns tokens until you intervene. It's overkill. And I am pointing out it is overkill. It's not a clever system for creating code for any given situation. It overfits to training data set. And your response is to claim that my argument is something else, not that it's overkill but that it can only kill dead things. I never said that. I see it's more than capable of spitting out useful code even if that exact same code is not in the training dataset. But it is just automating the process of going through google, docs and stack overflow and assembling something for you. You might be good at searching and lucky and it is just what you need. You might not be so used to using the right keywords or just be using some uncommon language, or in a domain that happens to not be well represented and then it feels less useful. But instead of just coming up short as search, the model overkills and wastes your time and god knows how much subsidized energy and compute. Lucky you if you're not burning tokens on some agentic monstosity.
  - simonw 2y ago
    If that's the case, it turns out that what I want is a system that's "overfitted to the dataset" on code, since I'm getting incredibly useful results for code out of it. (I'm not personally interested in the whole AGI thing.)
    - namaria 2y ago
      Good man I never said anything about AGI. Why do you keep responding to things I never said? This whole exchange was you having knee-jerk reactions to things you imagined I said. It has been incredibly frustrating. And at the end you shrug and say "eh it's useful to me"?? I am talking about this because of deceitfulness, resource efficiency, societal implications of technology.
      - simonw 2y ago
        "That is the premise of LLM-as-AI" - I assumed that was an AGI reference. My definition of AGI is pretty much "hyped AI". What did you mean by "LLM-as-AI"? In my own writing I don't even use the term "AI" very often because its meaning is so vague. You're right to call me out on this: I did, in this earlier comment - https://news.ycombinator.com/item?id=43644662#43647037 https://news.ycombinator.com/item?id=43644662#43647037 - commit the sin of responding to something you hadn't actually said. (Worse than that, I said "... is uninformed in my opinion" which was rude because I was saying that about a strawman argument.) I did that thing where I saw an excuse to bang on one of my pet peeves (people saying "LLMs can't create new code if it's not already in their training data") and jumped at the opportunity. I've tried to continue the rest of the conversation in good faith though. I'm sorry if it didn't come across that way.
        mdp2021 2y ago
        > My definition of AGI is pretty much Simon, intelligence exists (and unintelligence exists). When you write «I'm not claiming LLMs can invent new computer science», you imply intelligence exists. We can implement it. And it is somehow urgent, because intelligence is very desirable wealth - there is definite scarcity. It is even more urgent after the recent hype has made some people perversely confused about the idea of intelligence. We can and must go well beyond the current state.

simonw 2y ago

I don't think I understood your point then. I matched it with the common "LLMs can only produce code that's similar to what they've seen before" argument. Reading back, you said: > I often see people wondering if the some coding task is performed well or not because of availability of code examples in the training data. It's way worse than that. It's overfitting to diffs it was trained on. I'll be honest: I don't understand what you mean by "overfitting to diffs it was trained on" there. Maybe I don't understand what "overfitting" means in this context? (I'm afraid I didn't understand your cannon / fly swatter analogy either.)

namaria 2y ago
It's overkill. The models do not capture knowledge about coding. They overfit to the dataset. When one distills data into a useful model the model can be used to predict future behavior of the system. That is the premise of LLM-as-AI. By training these models on enough data, knowledge of the world is purported as having been captured, creating something useful that can be leveraged to process new input and get a prediction of the trajectory of the system in some phase space. But this, I argue, is not the case. The models merely overfit to the training data. Hence the variable results perceived by people. When their intentions and prompt fit to the data in the training, the model appears to give good output. But the situation and prompt do not, the models do no "reason" about it and "infer" anything. It fails. It gives you gibberish or go in circles, or worse if there is some "agentic" arrangement if fails to terminate and burns tokens until you intervene. It's overkill. And I am pointing out it is overkill. It's not a clever system for creating code for any given situation. It overfits to training data set. And your response is to claim that my argument is something else, not that it's overkill but that it can only kill dead things. I never said that. I see it's more than capable of spitting out useful code even if that exact same code is not in the training dataset. But it is just automating the process of going through google, docs and stack overflow and assembling something for you. You might be good at searching and lucky and it is just what you need. You might not be so used to using the right keywords or just be using some uncommon language, or in a domain that happens to not be well represented and then it feels less useful. But instead of just coming up short as search, the model overkills and wastes your time and god knows how much subsidized energy and compute. Lucky you if you're not burning tokens on some agentic monstosity.
- simonw 2y ago
  If that's the case, it turns out that what I want is a system that's "overfitted to the dataset" on code, since I'm getting incredibly useful results for code out of it. (I'm not personally interested in the whole AGI thing.)
  - namaria 2y ago
    Good man I never said anything about AGI. Why do you keep responding to things I never said? This whole exchange was you having knee-jerk reactions to things you imagined I said. It has been incredibly frustrating. And at the end you shrug and say "eh it's useful to me"?? I am talking about this because of deceitfulness, resource efficiency, societal implications of technology.
    - simonw 2y ago
      "That is the premise of LLM-as-AI" - I assumed that was an AGI reference. My definition of AGI is pretty much "hyped AI". What did you mean by "LLM-as-AI"? In my own writing I don't even use the term "AI" very often because its meaning is so vague. You're right to call me out on this: I did, in this earlier comment - https://news.ycombinator.com/item?id=43644662#43647037 https://news.ycombinator.com/item?id=43644662#43647037 - commit the sin of responding to something you hadn't actually said. (Worse than that, I said "... is uninformed in my opinion" which was rude because I was saying that about a strawman argument.) I did that thing where I saw an excuse to bang on one of my pet peeves (people saying "LLMs can't create new code if it's not already in their training data") and jumped at the opportunity. I've tried to continue the rest of the conversation in good faith though. I'm sorry if it didn't come across that way.
      - mdp2021 2y ago
        > My definition of AGI is pretty much Simon, intelligence exists (and unintelligence exists). When you write «I'm not claiming LLMs can invent new computer science», you imply intelligence exists. We can implement it. And it is somehow urgent, because intelligence is very desirable wealth - there is definite scarcity. It is even more urgent after the recent hype has made some people perversely confused about the idea of intelligence. We can and must go well beyond the current state.