3 ms·

> ... which explains why some models are better at code than others. No. It explains why models seem better at code in given situations. When your prompt mappe

> ... which explains why some models are better at code than others. No. It explains why models seem better at code in given situations. When your prompt mapped to diffs in the training data that are useful to you they seem great.

simonw 2y ago
I've been writing code with LLM assistance for over two years now and I've had plenty of situations where I am 100% confident the thing I am doing has never been done by anyone else before. I've tried things like searching all of the public code on GitHub for every possible keyword relevant to my problem. ... or I'm writing code against libraries which didn't exist when the models were trained. The idea that models can only write code if they've seen code that does the exact same thing in the past is uninformed in my opinion.
- namaria 2y ago
  > The idea that models can only write code if they've seen code that does the exact same thing in the past is deeply uninformed in my opinion. This is a conceited interpretation of what I said.
  - xboxnolifes 2y ago
    If this isn't what you meant, then what did you mean? To me, it's exactly how I read what you said.
    - namaria 2y ago
      I am sorry but that's nonsense. I quoted the paper "Evolution through Large Models" written in collaboration between OpenAI and Anthropic researchers "In other words, the model learns to predict plausible changes to code from examples of changes made to code by human programmers." https://arxiv.org/pdf/2206.08896 https://arxiv.org/pdf/2206.08896 > The idea that models can only write code if they've seen code that does the exact same thing in the past How do you get "code that does the exact same thing" from "predicting plausible changes?"
      - simonw 2y ago
        That paper describes an experimental diff-focused approach from 2022. It's not clear to me how relevant it is to the way models like Claude 3.7 Sonnet (thinking) and o3-mini work today.
        namaria 2y ago
        If do not you think past research by OpenAI and Anthropic on how to use LLMs to generate code is relevant to how Anthropic LLMs generate code 3 years later I really don't think it is possible to have a reasonable conversation about this topic with you.
        simonw 2y ago
        Can we be sure that research became part of their mainline model development process as opposed to being an interesting side-quest? Are Gemini and DeepSeek and Llama and other strong coding models using the same ideas? Llama and DeepSeek are at least slightly more open about their training processes so there might be clues in their papers (that's a lot of stuff to crunch through though).
- fergal_reid 2y ago
  Strongly agree. This seems to be very hard for people to accept, per the other comments here. Until recently I was willing to accept an argument that perhaps LLMs had mostly learned the patterns; e.g. to maybe believe 'well there aren't that many really different leetcode questions'. But with recent models (eg sonnet-3.7-thinking) they are operating well on such large and novel chunks of code that the idea they've seen everything in the training set, or even, like, a close structural match, is becoming ridiculous.
  - namaria 2y ago
    All due respect to Simon but I would love to see some of that groundbreaking code that the LLMs are coming up with. I am sure that the functionalities implemented are novel but do you really think the training data cannot possibly have had the patterns being used to deliver these features, really? How is it that in the past few months or years people suddenly found the opportunity and motivation to write code that cannot possibly be in any way shape or form represented by patterns in the diffs that have been pushed in the past 30 years?
    - simonw 2y ago
      When I said "the thing I am doing has never been done by anyone else before" I didn't necessarily mean groundbreaking pushes-the-edge-of-computer-science stuff - I meant more pedestrian things like "nobody has ever published Python code to condense and uncondense JSON using this new format I just invented today": https://github.com/simonw/condense-json https://github.com/simonw/condense-json I'm not claiming LLMs can invent new computer science. I'm saying it's not accurate to say "they can only produce code that's almost identical to what's in their training data".
      - namaria 2y ago
        > "they can only produce code that's almost identical to what's in their training data" Again, you're misinterpreting in a way that seems like you are reacting to the perception that someone attacked some of your core beliefs rather than considering what I am saying and conversing about that. I never even used the words "exact same thing" or "almost identical". Not even synonyms. I just said overfitting and quoted from an OpenAI/Anthropic paper that said "predict plausible changes to code from examples of changes" Think about that. Don't react, think. Why do you equate overfitting and plausibility prediction with "exact" and "identical". It very obviously is not what I said. What I am getting at is that a cannon will kill the mosquito. But drawing a fly swatter in the cannonball and saying the plastic ones are obsolete now would be in bad faith. No need to say to someone pointing that out that they are claiming that the cannon can only fire on mosquitoes that have been swatted before.

simonw 2y ago

I've been writing code with LLM assistance for over two years now and I've had plenty of situations where I am 100% confident the thing I am doing has never been done by anyone else before. I've tried things like searching all of the public code on GitHub for every possible keyword relevant to my problem. ... or I'm writing code against libraries which didn't exist when the models were trained. The idea that models can only write code if they've seen code that does the exact same thing in the past is uninformed in my opinion.

namaria 2y ago
> The idea that models can only write code if they've seen code that does the exact same thing in the past is deeply uninformed in my opinion. This is a conceited interpretation of what I said.
- xboxnolifes 2y ago
  If this isn't what you meant, then what did you mean? To me, it's exactly how I read what you said.
  - namaria 2y ago
    I am sorry but that's nonsense. I quoted the paper "Evolution through Large Models" written in collaboration between OpenAI and Anthropic researchers "In other words, the model learns to predict plausible changes to code from examples of changes made to code by human programmers." https://arxiv.org/pdf/2206.08896 https://arxiv.org/pdf/2206.08896 > The idea that models can only write code if they've seen code that does the exact same thing in the past How do you get "code that does the exact same thing" from "predicting plausible changes?"
    - simonw 2y ago
      That paper describes an experimental diff-focused approach from 2022. It's not clear to me how relevant it is to the way models like Claude 3.7 Sonnet (thinking) and o3-mini work today.
      - namaria 2y ago
        If do not you think past research by OpenAI and Anthropic on how to use LLMs to generate code is relevant to how Anthropic LLMs generate code 3 years later I really don't think it is possible to have a reasonable conversation about this topic with you.
        simonw 2y ago
        Can we be sure that research became part of their mainline model development process as opposed to being an interesting side-quest? Are Gemini and DeepSeek and Llama and other strong coding models using the same ideas? Llama and DeepSeek are at least slightly more open about their training processes so there might be clues in their papers (that's a lot of stuff to crunch through though).
fergal_reid 2y ago
Strongly agree. This seems to be very hard for people to accept, per the other comments here. Until recently I was willing to accept an argument that perhaps LLMs had mostly learned the patterns; e.g. to maybe believe 'well there aren't that many really different leetcode questions'. But with recent models (eg sonnet-3.7-thinking) they are operating well on such large and novel chunks of code that the idea they've seen everything in the training set, or even, like, a close structural match, is becoming ridiculous.
- namaria 2y ago
  All due respect to Simon but I would love to see some of that groundbreaking code that the LLMs are coming up with. I am sure that the functionalities implemented are novel but do you really think the training data cannot possibly have had the patterns being used to deliver these features, really? How is it that in the past few months or years people suddenly found the opportunity and motivation to write code that cannot possibly be in any way shape or form represented by patterns in the diffs that have been pushed in the past 30 years?
  - simonw 2y ago
    When I said "the thing I am doing has never been done by anyone else before" I didn't necessarily mean groundbreaking pushes-the-edge-of-computer-science stuff - I meant more pedestrian things like "nobody has ever published Python code to condense and uncondense JSON using this new format I just invented today": https://github.com/simonw/condense-json https://github.com/simonw/condense-json I'm not claiming LLMs can invent new computer science. I'm saying it's not accurate to say "they can only produce code that's almost identical to what's in their training data".
    - namaria 2y ago
      > "they can only produce code that's almost identical to what's in their training data" Again, you're misinterpreting in a way that seems like you are reacting to the perception that someone attacked some of your core beliefs rather than considering what I am saying and conversing about that. I never even used the words "exact same thing" or "almost identical". Not even synonyms. I just said overfitting and quoted from an OpenAI/Anthropic paper that said "predict plausible changes to code from examples of changes" Think about that. Don't react, think. Why do you equate overfitting and plausibility prediction with "exact" and "identical". It very obviously is not what I said. What I am getting at is that a cannon will kill the mosquito. But drawing a fly swatter in the cannonball and saying the plastic ones are obsolete now would be in bad faith. No need to say to someone pointing that out that they are claiming that the cannon can only fire on mosquitoes that have been swatted before.