3 ms·

If you read that much code you could code in your sleep. So far that seems to be a good intuition for models based on LLMs - how well can you code without most

by tveita 3y ago

If you read that much code you could code in your sleep. So far that seems to be a good intuition for models based on LLMs - how well can you code without most of your higher reasoning facilities, just glancing at the previous text and typing whatever your gut feeling tells you? The programs that beat humans at chess and go have added structure to be able to plan ahead; they use a Monte Carlo search to play out the moves that "intuitively" look better, with another "intuition" check to see how good the position looks in the end. Similarly, AlphaCode [1] generates a large set of potential solutions and uses additional logic to verify that the code compiles, runs, and passes tests. [1] https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a...

TeMPOraL 3y ago
Exactly this. I too find this to be the best intuition for LLMs right now: they're not comparable to an entire combined human mind - they're comparable to subconscious, or inner voice (as in, the part of your subconscious that interfaces with your conscious using language - aka. "the voice in your head", if you have one). So, as you say, if we had as much training as those LLMs, we'd be similarly good at coding by gut feel, with barely a conscious thought - and that's across pretty much any domain and technology that existed today. Compare with generic LLMs: a typical adult will be quite adept at saying somewhat coherent things on autopilot when prompted (!), which is reasonable given nearly two decades of constant exposure to natural language as written and spoken - but that same adult will be nowhere as good at this as GPT-4, and definitely not across so many different domains.
- shanebellone 3y ago
  "Exactly this. I too find this to be the best intuition for LLMs right now: they're not comparable to an entire combined human mind - they're comparable to subconscious, or inner voice" Strongly disagree. LLM traps you inside an intellectual bell curve.
  - TeMPOraL 3y ago
    What is your take then? And please don't say "stochastic parrot" or "hype train".
    - shanebellone 3y ago
      I view it as long-form autocomplete.
      - pizza 3y ago
        The devil’s in the details, or, in this case, the joint distribution between what a person would produce and what the model produces. If you came up with a way to train monkeys to write Hamlet on a typewriter, it’s still Hamlet. We’re not there yet - to the point where they consistently expand human potential for thought - but we could be, someday.
      - TeMPOraL 3y ago
        > I view it as long-form autocomplete. My wife sometimes views me as long-form autocomplete, and sometimes as a spell and grammar checker. Hell, my reply to your comment here is indistinguishable from a "long-form autocomplete". Point being, that autocomplete has to work in some way. Our LLM autocompletes have been getting better and better at zero-shot completion to arbitrary long-form text, including arbitrary simulated conversations with a simulated human, without commensurate increase in complexity or resource utilization. This means they're getting better and better at compressing their training data - but in the limit, what is the difference between compression and understanding? I can't prove it formally, but I rather strongly believe they are, fundamentally, the same thing. Also: if it walks like a duck, quacks like a duck, swims like a duck, ducks like a duck, and is indistinguishable from a duck on any possible test you can think of or apply to it, then maybe your artificial faux-duck effectively turned into a real duck?
        mrtranscendence 3y ago
        > what is the difference between compression and understanding? I can't prove it formally, but I rather strongly believe they are, fundamentally, the same thing. I'm not sure this is true in general. I feel as if I understand something when I grasp it in its entirety, not when I've been able to summarize it concisely. And conceptually I can compress something without understanding it by manually implementing compression algorithms and following their instructions by rote. I think understanding and compression are plausibly related; one test of whether I understand something is whether I can explain it to a layperson. But I don't see how they're equivalent even asymptotically. > then maybe your artificial faux-duck effectively turned into a real duck? I can't really get behind this sentiment. If a language model behaves like a duck in every readily observable particular then we can substitute language models for ducks, sure. But that does not imply that a language model is a duck, and whether it even could be a duck remains an interesting and important question. I'm sympathetic to the argument that it doesn't really matter in day-to-day practice, but that shouldn't stop us from raising the question.
- joquarky 3y ago
  I have been thinking along the same lines. The chain of thoughts that arise during meditation remind me a lot of language generators.
  - TeMPOraL 3y ago
    I've had that thought for over a decade now. I felt that my inner voice is a bit of a Markov chain generator at the border between my conscious and unconscious, randomly stringing some thoughts in form of sentences (often mixed-language, to boot), and conscious-level thinking involves evaluating those thought streams - cutting some off completely, letting others continue or mixing them and "feeding back" to the generator, so it iterates more on those. Markov chains (and a lot of caching) were a good high-level working model, but quite inadequate in power when inspected in detail. Deep language models I initially ignored, as they felt more like doubling down on caching alone and building convoluted lookup tables. But, to my surprise, LLMs turned not only to be a better high-level analogy - the way they work in practice feels so close to my experience with my own "inner voice", that I can't believe this is just a coincidence. What I mean here is, in short: whenever I read articles and comments about strengths and weaknesses of current LLMs (especially GPT-4), I find that they might just as well be talking about my own "inner voice" / gut-level, intuition-driven thinking - it has the same strengths and the same failure modes.
kohlerm 3y ago
chatGPT==alphazero(or lc0) without search You can manually try to mitigate it. People call it AutoGPT ;-)
mdaniel 3y ago
> If you read that much code you could code in your sleep I would offer the counter to that: had anyone written that much code, they may be able to code in their sleep, but my 5 years of college tell me that just reading calculus books does nothing in the world for making me able to use calculus I don't recall the 10,000 hour hypothesis being that one watches 10,000 hours of tv on a subject, rather that one has practiced something for 10,000 hours

by tveita 3y ago

TeMPOraL 3y ago

Exactly this. I too find this to be the best intuition for LLMs right now: they're not comparable to an entire combined human mind - they're comparable to subconscious, or inner voice (as in, the part of your subconscious that interfaces with your conscious using language - aka. "the voice in your head", if you have one). So, as you say, if we had as much training as those LLMs, we'd be similarly good at coding by gut feel, with barely a conscious thought - and that's across pretty much any domain and technology that existed today. Compare with generic LLMs: a typical adult will be quite adept at saying somewhat coherent things on autopilot when prompted (!), which is reasonable given nearly two decades of constant exposure to natural language as written and spoken - but that same adult will be nowhere as good at this as GPT-4, and definitely not across so many different domains.

shanebellone 3y ago
"Exactly this. I too find this to be the best intuition for LLMs right now: they're not comparable to an entire combined human mind - they're comparable to subconscious, or inner voice" Strongly disagree. LLM traps you inside an intellectual bell curve.
- TeMPOraL 3y ago
  What is your take then? And please don't say "stochastic parrot" or "hype train".
  - shanebellone 3y ago
    I view it as long-form autocomplete.
    - pizza 3y ago
      The devil’s in the details, or, in this case, the joint distribution between what a person would produce and what the model produces. If you came up with a way to train monkeys to write Hamlet on a typewriter, it’s still Hamlet. We’re not there yet - to the point where they consistently expand human potential for thought - but we could be, someday.
    - TeMPOraL 3y ago
      > I view it as long-form autocomplete. My wife sometimes views me as long-form autocomplete, and sometimes as a spell and grammar checker. Hell, my reply to your comment here is indistinguishable from a "long-form autocomplete". Point being, that autocomplete has to work in some way. Our LLM autocompletes have been getting better and better at zero-shot completion to arbitrary long-form text, including arbitrary simulated conversations with a simulated human, without commensurate increase in complexity or resource utilization. This means they're getting better and better at compressing their training data - but in the limit, what is the difference between compression and understanding? I can't prove it formally, but I rather strongly believe they are, fundamentally, the same thing. Also: if it walks like a duck, quacks like a duck, swims like a duck, ducks like a duck, and is indistinguishable from a duck on any possible test you can think of or apply to it, then maybe your artificial faux-duck effectively turned into a real duck?
      - mrtranscendence 3y ago
        > what is the difference between compression and understanding? I can't prove it formally, but I rather strongly believe they are, fundamentally, the same thing. I'm not sure this is true in general. I feel as if I understand something when I grasp it in its entirety, not when I've been able to summarize it concisely. And conceptually I can compress something without understanding it by manually implementing compression algorithms and following their instructions by rote. I think understanding and compression are plausibly related; one test of whether I understand something is whether I can explain it to a layperson. But I don't see how they're equivalent even asymptotically. > then maybe your artificial faux-duck effectively turned into a real duck? I can't really get behind this sentiment. If a language model behaves like a duck in every readily observable particular then we can substitute language models for ducks, sure. But that does not imply that a language model is a duck, and whether it even could be a duck remains an interesting and important question. I'm sympathetic to the argument that it doesn't really matter in day-to-day practice, but that shouldn't stop us from raising the question.
joquarky 3y ago
I have been thinking along the same lines. The chain of thoughts that arise during meditation remind me a lot of language generators.
- TeMPOraL 3y ago
  I've had that thought for over a decade now. I felt that my inner voice is a bit of a Markov chain generator at the border between my conscious and unconscious, randomly stringing some thoughts in form of sentences (often mixed-language, to boot), and conscious-level thinking involves evaluating those thought streams - cutting some off completely, letting others continue or mixing them and "feeding back" to the generator, so it iterates more on those. Markov chains (and a lot of caching) were a good high-level working model, but quite inadequate in power when inspected in detail. Deep language models I initially ignored, as they felt more like doubling down on caching alone and building convoluted lookup tables. But, to my surprise, LLMs turned not only to be a better high-level analogy - the way they work in practice feels so close to my experience with my own "inner voice", that I can't believe this is just a coincidence. What I mean here is, in short: whenever I read articles and comments about strengths and weaknesses of current LLMs (especially GPT-4), I find that they might just as well be talking about my own "inner voice" / gut-level, intuition-driven thinking - it has the same strengths and the same failure modes.

kohlerm 3y ago

chatGPT==alphazero(or lc0) without search You can manually try to mitigate it. People call it AutoGPT ;-)

mdaniel 3y ago

> If you read that much code you could code in your sleep I would offer the counter to that: had anyone written that much code, they may be able to code in their sleep, but my 5 years of college tell me that just reading calculus books does nothing in the world for making me able to use calculus I don't recall the 10,000 hour hypothesis being that one watches 10,000 hours of tv on a subject, rather that one has practiced something for 10,000 hours