9 ms·

I've been using GPT4 to code and these explanations are somewhat unsatisfactory. I have seen it seemingly come up with novel solutions in a way that I can't des

I've been using GPT4 to code and these explanations are somewhat unsatisfactory. I have seen it seemingly come up with novel solutions in a way that I can't describe in any other way than it is thinking. It's really difficult for me to imagine how such a seemingly simple predictive algorithm could lead to such complex solutions. I'm not sure even the people building these models really grasp it either.

lm28469 4y ago
Care to post a full example ?
- simonw 4y ago
  I used GPT-4 to build this tool https://image-to-jpeg.vercel.app https://image-to-jpeg.vercel.app using a few prompts the other day - my ChatGPT transcript for that is here: https://gist.github.com/simonw/66918b6cde1f87bf4fc883c67735195d https://gist.github.com/simonw/66918b6cde1f87bf4fc883c677351...
  - camillomiller 4y ago
    Love how you didn’t care about styling this like at all, Lol. Btw, if you ask gpt to make it presentable by using bootstrap 5 for example it can style it for you
    - capableweb 4y ago
      One mans "presentable" is another mans bloat. It looks perfectly fine to me, simple, useful and self-explanatory, doesn't need more flash than so.
      - camillomiller 4y ago
        Sure, but presentation and UX basics are not "bloat".
        capableweb 4y ago
        What "basic UX" principles are being violated here exactly? And how would adding Bootstrap solve those?
  - lm28469 4y ago
    See my problem with virtually every single example is that we talk about "I can't describe in any other way than it is thinking", "such complex solutions" but in the end we get a 50 lines "app" that you'd see in a computer science 101 class It's very nice, it's very impressive, it will help people, but it doesn't align with the "you're just about to lose your job" "Skynet comes in the next 6 months" &c. If these basic samples are a bottleneck in your day to day life as a developer I'm worried about the state of the industry
    - simonw 4y ago
      This took me 3 minutes to build. Without ChatGPT it would have taken me 30-60 minutes, if not longer thanks to the research I would have needed to do into the various browser APIs. If it had taken me longer than 3 minutes I wouldn't have bothered - it's not a tool I needed enough to put the work in. That's the thing I find so interesting about this stuff: it's causing me to be much more ambitious in what I chose to build: https://simonwillison.net/2023/Mar/27/ai-enhanced-development/ https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen...
    - int_19h 4y ago
      The concern is the velocity. GPT-4 can solve tasks today that it couldn't solve one months ago. And even one month ago, the things it could do made GPT-3.5 look like a silly toy. Then there's the question of how much this can be scaled further simply by throwing more hardware at it to run larger models. We're not anywhere near the limit of that yet.
  - ZeroGravitas 4y ago
    I'm assuming the bits that say > // Rest of the code remains the same Are exactly as generated by GPT-4, i.e. it knew it didn't need to repeat the bits that hadn't changed, and knew to leave a comment like this to indicate that to the user. It gets confusing when something can fake a human so well.
    - int_19h 4y ago
      Yes, it will do that routinely. For example, you can ask it to generate HTML/JS/SVG in a single file to render some animated scene, and then iterate on that by telling it what looks wrong or what behaviors you like to change - and it will answer by saying things like, "replace the contents of the <script> element with the following".
simonw 4y ago
I've started to suspect that generating code is actually one of the easier things for a predictive text completion model to achieve. Programming languages are a whole lot more structured and predictable than human language. In JavaScript the only token that ever comes after "if " is "(" for example.
- camillomiller 4y ago
  This!
- capableweb 4y ago
  > In JavaScript the only token that ever comes after "if " is "(" for example. I'm pretty sure " " (whitespace) is a token as well, which could come after a `if` as well. I think overall your point is a pretty good one though.
- exitb 4y ago
  On the other hand, if you want to use an external library on the line 80, you need to import it at the top. I once asked it for a short example code of something, no longer than 15 lines and it said "here's a code that's 12 lines long" and then added the code. Did it have the specific code "in mind" already? Or was it just a reasonably-sounding length and it then just came up with code that matched that self-imposed constraint?
  - matjet 4y ago
    The latter option is closest, but neither is quite right. It would have ~known~ that the problem asked, combined with a phrase for a 15 line limit has associations with a length of 12 lines (perhaps most strongly 12, but depending on temp it could have given other answers). From there it is constrained to (complete) solutions that lead to 12 lines, from the several (partial) solutions that already exist in the weights.
- twobitshifter 4y ago
  I loved your example. I think that may be an obvious advantage to LLM, humans are poor at learning new languages after adolescence but a LLM can continue to learn and build new connections. Studies show that multilingual people have an easier time making connections and producing new ideas, In the case of programming, we may build something that knows all programming languages and all design patterns and can merge this knowledge to come up with better solutions than the ordinary programmer.
m3kw9 4y ago
What’s novel to you could be just trained material
ben_w 4y ago
To be deliberately unfair, imagine a huge if-else block — like, a few billion entries big — and each branch played out a carefully chosen and well-written string of text. It would convince a lot of people with the breadth, despite not really having much depth. The real GPT model is much deeper than that, of course, but my toy example should at least give a vibe for why even a simple thing might still feel extraordinary.
- myrmidon 4y ago
  This is absolutely not viable because exponential growth absolutely kills the concept. Such a system would already struggle with multiple-word inputs and it would be completely impossible to make it scale to even a paragraph of text, even if you had ALL of the observable universe at your disposal for encoding the entries. Consider: If you just have simple sentences consisting of 3 words (subject, object, verb, with 1000 options each-- very conservative assumptions), then 9 sentences already give more options than you have atoms (!!) in the observable universe (~10^80)
  - ben_w 4y ago
    α: most of those sentences are meaningless so they won't come up in normal use β: if statements can grab patterns just fine in most languages, they're not limited to pure equality γ: it's a thought experiment about how easy it can be to create illusions without real depth, and specifically not about making an AGI that stands up to scrutiny
    - myrmidon 4y ago
      > most of those sentences are meaningless so they won't come up in normal use Feel free to come up with a better entropy model then. Stackoverflow gives me confidence that it will be between 5 and 11 bits per word anyway [https://linguistics.stackexchange.com/questions/8480/what-is-the-entropy-per-word-of-random-yet-grammatical-text https://linguistics.stackexchange.com/questions/8480/what-is...]. > if statements can grab patterns just fine in most languages, they're not limited to pure equality This does not help you one bit. If you want to produce 9 sentences of output per query then regular expressions, pattern matching or even general intelligence inside your if statements will NOT be able to save the concept.
LeSaucy 4y ago
I have only seen gpt generate imperative algorithms. Does it have the ability to work with concurrency and asynchrony?
- z3c0 4y ago
  I've attempted to pose a concurrency problem to GPT4. The output was invalid code, though likely would have looked correct to the untrained eye. It was only after I spelled out the limitations that it could account for them.
  - k__ 4y ago
    I tried point free solutions, which threw it off.
EForEndeavour 4y ago
The advanced capabilities of scaled up transformer models fed oodles of training data has burdened me with pseudo-philosophical questions about the nature of cognition that I am not well equipped to articulate, and make me wish I'd studied more neuroscience, philosophy, and comp sci earlier in life. A possibly off-topic thought dump: - What is thinking, exactly? - Does human (or superhuman) thinking require consciousness? - What even is consciousness? Why is it that when you take a bunch of molecular physical laws and scale them up into a human brain, a signal pattern emerges that feels things like emotions, continuity between moments, desires, contemplation of itself and the surrounding universe, and so on? - Why and how does a string predictor on steroids turn out to do things that seem so close to a practical definition of thinking? What are the best evidence-based arguments supporting and opposing the statement "GPT4 thinks"? How do people without OpenAI's level of model access try to answer this question? (And yes, it's occurred to me that I could try asking GPT4 to help me make these questions more complete)
- kingkongjaffa 4y ago
  I think since the mechanisms are different we should arrive at a distinction between: organic thinking (I.e. the process our squishy human brains do) and mechanical thinking ( the computational and stochastic processes that computers do ).
  - TuringTest 4y ago
    I don't think the substrate defines the nature of the thinking, but the form of the process does. It is entirely possible to build mechanical thinking in organic material (think Turing machines built on growing tissue), and it could also be possible to build complex self-referential processes simulated on electronic hardware, of the kind high-level brains do, with their rhythms of alfa and beta waves.
- Workaccount2 4y ago
  > has burdened me with pseudo-philosophical questions about the nature of cognition that I am not well equipped to articulate, and make me wish I'd studied more neuroscience, philosophy, and comp sci earlier in life Welcome to the club. There pretty much are no answers, just theories primarily played out as thought experiments. Its on of those areas where you can pick out who knows less (or is being disingenuous) by seeing who most confidently speaks about having answers. We don't know what consciousness is, and we don't know what it means to "think". There, I saved you a decade of reading. Edit: My choice theory is panpsychism, https://plato.stanford.edu/entries/panpsychism/ https://plato.stanford.edu/entries/panpsychism/ but again, we don't yet know how to verify any of this (or any other theory).
agentultra 4y ago
It's not thinking, plain and simple. Anything it generates means nothing to the algorithm. When you read it and interpret what was generated you're experiencing something like the Barnum-Forer effect. It's sort of like reading a horoscope and believing it predicted your future.
- fnordpiglet 4y ago
  Except for when as an expert in a field you ask it questions about that are subtle and it answers in a cogent and insightful way, and as an expert you are fully aware of that. It’s not reasonable to call that a Barnum-Forer effect. It’s perhaps not thinking (but perhaps we need to more clearly define thinking), but its not a self-deception either.
- myrmidon 4y ago
  What gives you any confidence that the way GPT4 comes up with answers is qualitatively different from humans? Why should the emulation of human though, a result of unguided evolution, require anything more than properly wired silicon?
  - agentultra 4y ago
    That's highly reductive of our capacities. We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference model. That's it. It's not going to wake up one day, decide it prefers eggs benny and has had enough of your idle chatter because of that sarcastic remark you made last week. Could we simulate a plausibly realistic human brain on silicon someday? I don't know, maybe? But that's not what GPT is and we're no where near being able to do that. You can scale up the tokens an LLM can manage and all you get is a more accurate model with more weights and transformers. It's not going to wake up one day, have feelings, religion, decide things for itself, look in a mirror and reflect on its predicament, lament the poor response it gave a user, and decide it doesn't want to live with regret and correct its mistakes.
    - myrmidon 4y ago
robotresearcher 4y ago
Perhaps it’s more productive to go the other direction and consider how the concept of ‘thinking’ could be reconsidered. It’s not like we all agree on what thinking is. We never have. It may not even be one thing.
deleted 4y ago
[deleted]
cgearhart 4y ago
It’s a fallacy to describe what the machine does as “thinking” because that’s only process you know for achieving the same outcome. When you initiate the model with some input where you expect some particular correct output, that means there exists some completed sequence of tokens that is correct—if that weren’t true then you either wouldn’t ask or else you wouldn’t blame the model for being wrong. Now imagine a machine that takes in your input and in one step produces the entire output of that correct answer. In all nontrivial cases there are many more _incorrect_ possible outputs than correct ones, so this appears to be a difficult task. But would you say such a machine is “thinking”? Would you still consider it thinking if we could describe the process mathematically as drawing a sample from the output space; that it draws the correct sample implies it has an accurate probability model of the output space conditioned on your input. Does this require “thought”? GPT is just like this machine except that instead of one-step, the inference process is autoregressive so each token comes out one at a time instead of all at once. (Note that BERT-style transformers _do_ spit out the whole answer at once.) It’s possible that this is all that humans do. Perhaps we are mistaken about “thinking” altogether—perhaps the machine thinks (like a human), or perhaps humans do not think (like the machine). In either case I do feel confident that human and machine are not applying the same mechanism; jury is still out whether we’re applying the same process.
- int_19h 4y ago
  Now consider the case when you tell GPT to "think it out loud" before giving you the answer - which, coincidentally, is a well-known trick that tends to significantly improve its ability to produce good results. Is that thinking?
  - cgearhart 4y ago
    Maybe. Mechanically we might also describe it as causing the model to condition more explicitly on specific tokens derived from the training data rather than the implicit conditioning happening in the raw model parameters. This would tend to more tightly constrain the output space—making a smaller haystack to look for a needle. And leveraging the fact that “next token prediction” implies some consistency with preceding tokens. It could be thinking, but I don’t think that’s strong evidence that it is thinking.
HarHarVeryFunny 4y ago
Any overly simple "it's just predicting next word" explanation is really missing the point. It seems more accurate to regard that just as the way they are trained, rather than characterizing what they are learning and therefore what they are doing when they are generating. There are two ways of looking at this. 1) In order to predict next word probabilities correctly, you need to learn something about the input, and the better you want to get, the more you need to learn. For example, if you just learned part-of-speech categories for words (noun vs verb vs adverb, etc), and what usually follows what, then you would be doing better than chance.. If you want to do better than that they you need to learn the grammar of the underlying language(s).. If you want to do better than that then you start to need to learn the meaning of what is being discussed, etc, etc. If you want to correctly predict what comes next after "with a board position of ..., Magnus Carlson might play", then you better have learned a whole lot about the meaning of the input! The "predict next word" training objective and feedback provided doesn't itself limit what can be learned - that's up to the power of the model that is being trained, and evidentially large multi-layer transformers are exceptionally capable. Calling these huge transformers "LLMs" (large language models) is deceptive since beyond a certain scale they are certainly learning a whole lot more than language/grammar. 2) In the words of one of the OpenAI developers (Sutskever), what these models have really learnt is some type of "world model" modelling the underlying generative processes that produced the training data. So, they are not just using surface level statistics to "predict next word", but rather are using the (often very lengthy/detailed) input prompt to "get into the head" of what generated that, and are predicting on that basis.
samstave 4y ago
What is the time-spent for delta btwn fixing GPT code to writing it all yourself? Is it a reasonable scaffold that will grow over time?

lm28469 4y ago

Care to post a full example ?

simonw 4y ago
I used GPT-4 to build this tool https://image-to-jpeg.vercel.app https://image-to-jpeg.vercel.app using a few prompts the other day - my ChatGPT transcript for that is here: https://gist.github.com/simonw/66918b6cde1f87bf4fc883c67735195d https://gist.github.com/simonw/66918b6cde1f87bf4fc883c677351...
- camillomiller 4y ago
  Love how you didn’t care about styling this like at all, Lol. Btw, if you ask gpt to make it presentable by using bootstrap 5 for example it can style it for you
  - capableweb 4y ago
    One mans "presentable" is another mans bloat. It looks perfectly fine to me, simple, useful and self-explanatory, doesn't need more flash than so.
    - camillomiller 4y ago
      Sure, but presentation and UX basics are not "bloat".
      - capableweb 4y ago
        What "basic UX" principles are being violated here exactly? And how would adding Bootstrap solve those?
- lm28469 4y ago
  See my problem with virtually every single example is that we talk about "I can't describe in any other way than it is thinking", "such complex solutions" but in the end we get a 50 lines "app" that you'd see in a computer science 101 class It's very nice, it's very impressive, it will help people, but it doesn't align with the "you're just about to lose your job" "Skynet comes in the next 6 months" &c. If these basic samples are a bottleneck in your day to day life as a developer I'm worried about the state of the industry
  - simonw 4y ago
    This took me 3 minutes to build. Without ChatGPT it would have taken me 30-60 minutes, if not longer thanks to the research I would have needed to do into the various browser APIs. If it had taken me longer than 3 minutes I wouldn't have bothered - it's not a tool I needed enough to put the work in. That's the thing I find so interesting about this stuff: it's causing me to be much more ambitious in what I chose to build: https://simonwillison.net/2023/Mar/27/ai-enhanced-development/ https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen...
  - int_19h 4y ago
    The concern is the velocity. GPT-4 can solve tasks today that it couldn't solve one months ago. And even one month ago, the things it could do made GPT-3.5 look like a silly toy. Then there's the question of how much this can be scaled further simply by throwing more hardware at it to run larger models. We're not anywhere near the limit of that yet.
- ZeroGravitas 4y ago
  I'm assuming the bits that say > // Rest of the code remains the same Are exactly as generated by GPT-4, i.e. it knew it didn't need to repeat the bits that hadn't changed, and knew to leave a comment like this to indicate that to the user. It gets confusing when something can fake a human so well.
  - int_19h 4y ago
    Yes, it will do that routinely. For example, you can ask it to generate HTML/JS/SVG in a single file to render some animated scene, and then iterate on that by telling it what looks wrong or what behaviors you like to change - and it will answer by saying things like, "replace the contents of the <script> element with the following".

simonw 4y ago

I've started to suspect that generating code is actually one of the easier things for a predictive text completion model to achieve. Programming languages are a whole lot more structured and predictable than human language. In JavaScript the only token that ever comes after "if " is "(" for example.

camillomiller 4y ago
This!
capableweb 4y ago
> In JavaScript the only token that ever comes after "if " is "(" for example. I'm pretty sure " " (whitespace) is a token as well, which could come after a `if` as well. I think overall your point is a pretty good one though.
exitb 4y ago
On the other hand, if you want to use an external library on the line 80, you need to import it at the top. I once asked it for a short example code of something, no longer than 15 lines and it said "here's a code that's 12 lines long" and then added the code. Did it have the specific code "in mind" already? Or was it just a reasonably-sounding length and it then just came up with code that matched that self-imposed constraint?
- matjet 4y ago
  The latter option is closest, but neither is quite right. It would have ~known~ that the problem asked, combined with a phrase for a 15 line limit has associations with a length of 12 lines (perhaps most strongly 12, but depending on temp it could have given other answers). From there it is constrained to (complete) solutions that lead to 12 lines, from the several (partial) solutions that already exist in the weights.
twobitshifter 4y ago
I loved your example. I think that may be an obvious advantage to LLM, humans are poor at learning new languages after adolescence but a LLM can continue to learn and build new connections. Studies show that multilingual people have an easier time making connections and producing new ideas, In the case of programming, we may build something that knows all programming languages and all design patterns and can merge this knowledge to come up with better solutions than the ordinary programmer.

m3kw9 4y ago

What’s novel to you could be just trained material

ben_w 4y ago

To be deliberately unfair, imagine a huge if-else block — like, a few billion entries big — and each branch played out a carefully chosen and well-written string of text. It would convince a lot of people with the breadth, despite not really having much depth. The real GPT model is much deeper than that, of course, but my toy example should at least give a vibe for why even a simple thing might still feel extraordinary.

myrmidon 4y ago
This is absolutely not viable because exponential growth absolutely kills the concept. Such a system would already struggle with multiple-word inputs and it would be completely impossible to make it scale to even a paragraph of text, even if you had ALL of the observable universe at your disposal for encoding the entries. Consider: If you just have simple sentences consisting of 3 words (subject, object, verb, with 1000 options each-- very conservative assumptions), then 9 sentences already give more options than you have atoms (!!) in the observable universe (~10^80)
- ben_w 4y ago
  α: most of those sentences are meaningless so they won't come up in normal use β: if statements can grab patterns just fine in most languages, they're not limited to pure equality γ: it's a thought experiment about how easy it can be to create illusions without real depth, and specifically not about making an AGI that stands up to scrutiny
  - myrmidon 4y ago
    > most of those sentences are meaningless so they won't come up in normal use Feel free to come up with a better entropy model then. Stackoverflow gives me confidence that it will be between 5 and 11 bits per word anyway [https://linguistics.stackexchange.com/questions/8480/what-is-the-entropy-per-word-of-random-yet-grammatical-text https://linguistics.stackexchange.com/questions/8480/what-is...]. > if statements can grab patterns just fine in most languages, they're not limited to pure equality This does not help you one bit. If you want to produce 9 sentences of output per query then regular expressions, pattern matching or even general intelligence inside your if statements will NOT be able to save the concept.

LeSaucy 4y ago

I have only seen gpt generate imperative algorithms. Does it have the ability to work with concurrency and asynchrony?

z3c0 4y ago
I've attempted to pose a concurrency problem to GPT4. The output was invalid code, though likely would have looked correct to the untrained eye. It was only after I spelled out the limitations that it could account for them.
- k__ 4y ago
  I tried point free solutions, which threw it off.

EForEndeavour 4y ago

The advanced capabilities of scaled up transformer models fed oodles of training data has burdened me with pseudo-philosophical questions about the nature of cognition that I am not well equipped to articulate, and make me wish I'd studied more neuroscience, philosophy, and comp sci earlier in life. A possibly off-topic thought dump: - What is thinking, exactly? - Does human (or superhuman) thinking require consciousness? - What even is consciousness? Why is it that when you take a bunch of molecular physical laws and scale them up into a human brain, a signal pattern emerges that feels things like emotions, continuity between moments, desires, contemplation of itself and the surrounding universe, and so on? - Why and how does a string predictor on steroids turn out to do things that seem so close to a practical definition of thinking? What are the best evidence-based arguments supporting and opposing the statement "GPT4 thinks"? How do people without OpenAI's level of model access try to answer this question? (And yes, it's occurred to me that I could try asking GPT4 to help me make these questions more complete)

kingkongjaffa 4y ago
I think since the mechanisms are different we should arrive at a distinction between: organic thinking (I.e. the process our squishy human brains do) and mechanical thinking ( the computational and stochastic processes that computers do ).
- TuringTest 4y ago
  I don't think the substrate defines the nature of the thinking, but the form of the process does. It is entirely possible to build mechanical thinking in organic material (think Turing machines built on growing tissue), and it could also be possible to build complex self-referential processes simulated on electronic hardware, of the kind high-level brains do, with their rhythms of alfa and beta waves.
Workaccount2 4y ago
> has burdened me with pseudo-philosophical questions about the nature of cognition that I am not well equipped to articulate, and make me wish I'd studied more neuroscience, philosophy, and comp sci earlier in life Welcome to the club. There pretty much are no answers, just theories primarily played out as thought experiments. Its on of those areas where you can pick out who knows less (or is being disingenuous) by seeing who most confidently speaks about having answers. We don't know what consciousness is, and we don't know what it means to "think". There, I saved you a decade of reading. Edit: My choice theory is panpsychism, https://plato.stanford.edu/entries/panpsychism/ https://plato.stanford.edu/entries/panpsychism/ but again, we don't yet know how to verify any of this (or any other theory).

agentultra 4y ago

It's not thinking, plain and simple. Anything it generates means nothing to the algorithm. When you read it and interpret what was generated you're experiencing something like the Barnum-Forer effect. It's sort of like reading a horoscope and believing it predicted your future.

fnordpiglet 4y ago
Except for when as an expert in a field you ask it questions about that are subtle and it answers in a cogent and insightful way, and as an expert you are fully aware of that. It’s not reasonable to call that a Barnum-Forer effect. It’s perhaps not thinking (but perhaps we need to more clearly define thinking), but its not a self-deception either.
myrmidon 4y ago
What gives you any confidence that the way GPT4 comes up with answers is qualitatively different from humans? Why should the emulation of human though, a result of unguided evolution, require anything more than properly wired silicon?
- agentultra 4y ago
  That's highly reductive of our capacities. We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference model. That's it. It's not going to wake up one day, decide it prefers eggs benny and has had enough of your idle chatter because of that sarcastic remark you made last week. Could we simulate a plausibly realistic human brain on silicon someday? I don't know, maybe? But that's not what GPT is and we're no where near being able to do that. You can scale up the tokens an LLM can manage and all you get is a more accurate model with more weights and transformers. It's not going to wake up one day, have feelings, religion, decide things for itself, look in a mirror and reflect on its predicament, lament the poor response it gave a user, and decide it doesn't want to live with regret and correct its mistakes.
  - myrmidon 4y ago

robotresearcher 4y ago

Perhaps it’s more productive to go the other direction and consider how the concept of ‘thinking’ could be reconsidered. It’s not like we all agree on what thinking is. We never have. It may not even be one thing.

deleted 4y ago

[deleted]

cgearhart 4y ago

It’s a fallacy to describe what the machine does as “thinking” because that’s only process you know for achieving the same outcome. When you initiate the model with some input where you expect some particular correct output, that means there exists some completed sequence of tokens that is correct—if that weren’t true then you either wouldn’t ask or else you wouldn’t blame the model for being wrong. Now imagine a machine that takes in your input and in one step produces the entire output of that correct answer. In all nontrivial cases there are many more _incorrect_ possible outputs than correct ones, so this appears to be a difficult task. But would you say such a machine is “thinking”? Would you still consider it thinking if we could describe the process mathematically as drawing a sample from the output space; that it draws the correct sample implies it has an accurate probability model of the output space conditioned on your input. Does this require “thought”? GPT is just like this machine except that instead of one-step, the inference process is autoregressive so each token comes out one at a time instead of all at once. (Note that BERT-style transformers _do_ spit out the whole answer at once.) It’s possible that this is all that humans do. Perhaps we are mistaken about “thinking” altogether—perhaps the machine thinks (like a human), or perhaps humans do not think (like the machine). In either case I do feel confident that human and machine are not applying the same mechanism; jury is still out whether we’re applying the same process.

int_19h 4y ago
Now consider the case when you tell GPT to "think it out loud" before giving you the answer - which, coincidentally, is a well-known trick that tends to significantly improve its ability to produce good results. Is that thinking?
- cgearhart 4y ago
  Maybe. Mechanically we might also describe it as causing the model to condition more explicitly on specific tokens derived from the training data rather than the implicit conditioning happening in the raw model parameters. This would tend to more tightly constrain the output space—making a smaller haystack to look for a needle. And leveraging the fact that “next token prediction” implies some consistency with preceding tokens. It could be thinking, but I don’t think that’s strong evidence that it is thinking.

HarHarVeryFunny 4y ago

Any overly simple "it's just predicting next word" explanation is really missing the point. It seems more accurate to regard that just as the way they are trained, rather than characterizing what they are learning and therefore what they are doing when they are generating. There are two ways of looking at this. 1) In order to predict next word probabilities correctly, you need to learn something about the input, and the better you want to get, the more you need to learn. For example, if you just learned part-of-speech categories for words (noun vs verb vs adverb, etc), and what usually follows what, then you would be doing better than chance.. If you want to do better than that they you need to learn the grammar of the underlying language(s).. If you want to do better than that then you start to need to learn the meaning of what is being discussed, etc, etc. If you want to correctly predict what comes next after "with a board position of ..., Magnus Carlson might play", then you better have learned a whole lot about the meaning of the input! The "predict next word" training objective and feedback provided doesn't itself limit what can be learned - that's up to the power of the model that is being trained, and evidentially large multi-layer transformers are exceptionally capable. Calling these huge transformers "LLMs" (large language models) is deceptive since beyond a certain scale they are certainly learning a whole lot more than language/grammar. 2) In the words of one of the OpenAI developers (Sutskever), what these models have really learnt is some type of "world model" modelling the underlying generative processes that produced the training data. So, they are not just using surface level statistics to "predict next word", but rather are using the (often very lengthy/detailed) input prompt to "get into the head" of what generated that, and are predicting on that basis.

samstave 4y ago

What is the time-spent for delta btwn fixing GPT code to writing it all yourself? Is it a reasonable scaffold that will grow over time?