4 ms·

Last paragraph: I see what’s happening now as a historic moment. For well over half a century the statistical and symbolic approaches to what we might call “AI

Last paragraph: I see what’s happening now as a historic moment. For well over half a century the statistical and symbolic approaches to what we might call “AI” evolved largely separately. But now, in ChatGPT + Wolfram they’re being brought together. And while we’re still just at the beginning with this, I think we can reasonably expect tremendous power in the combination—and in a sense a new paradigm for “AI-like computation”, made possible by the arrival of ChatGPT, and now by its combination with Wolfram|Alpha and Wolfram Language in ChatGPT + Wolfram.

wolfi008 4y ago
Yeah it was clear that he'd not pass on an opportunity to praise himself.
bottlepalm 4y ago
Between this and his last post, he continues to dismiss ChatGPT as fancy next word statistics.
- int_19h 4y ago
  That's not the takeaway I got from the "how it works" post. He does describe the word statistics part in detail - which is, of course, pretty fundamental to how it works - but he doesn't claim that it's merely a "stochastic parrot", like many do. On the contrary (highlighting mine): "... this means that ChatGPT—even with its ultimately straightforward neural net structure—is successfully able to “capture the essence” of human language and the thinking behind it. And moreover, in its training, ChatGPT has somehow “implicitly discovered” whatever regularities in language (and thinking) make this possible."
- SideQuark 4y ago
  That's exactly what ChatGPT is. It's specifically trained as a next word (well, token) predictor. It has no long term plan, only a local window of context. And inference is completely predict the next token, with a large scales stats prediction model.
  - bottlepalm 4y ago
    The long term plan is what generates the next word.
    - SideQuark 4y ago
      There is a sliding local, not global, window. There is no long term plan. It does not include context past the window. The GPT4 page states the max context is 32768 tokens. Thus it is local. It does not see anything past this. https://platform.openai.com/docs/models/overview https://platform.openai.com/docs/models/overview
      - bottlepalm 4y ago
        The long term plan stays internal to the network, only the next word is outputted.
        SideQuark 4y ago
        If you want to call a local plan long term go ahead. People will continue to correct you. It has no "plan" other than local context. I posted their own statements on it. I don't know how much simpler it clearer I can make it. A human can make a long term plan, and write unlimited words about it, and still leverage any part of unlimited context. This is a long term plan. A human can reference 1 token back, 10000 tokens back, 1 million tokens back, any number of tokens back. No LLM can do this, since they have limited context. The site I just posted from GPT4 themselves gave you the context length. Have you ever coded a LLM? Ever read one of their papers and understood it? Do you understand what the word "context" means? We're done.
        bottlepalm 4y ago
        Tokens in GPT are basically short term memory, just like people have short term memory. You can't reference a million tokens back, it's been compressed into your long term memory. For GPT the long term memory is the trained network itself, of which it contains vastly more knowledge than any of us alone. I will say that writing perfectly working application from a prompt requires a plan, even if the output is the next word, there is a higher level plan you're not seeing in the output. This isnt sophisticated markov chains, it's understanding within the network that generates output.
egman_ekki 4y ago
I think he might actually be onto something here. If you saw Wolfram use WolframAlpha, he’s very good at it, but you kinda need to know what it knows and how to ask for it to be smooth and useful. If this initial hurdle gets removed, this power could enable the usage of WolframAlpha by many more people. And honestly it’s a very powerful combo.

wolfi008 4y ago

Yeah it was clear that he'd not pass on an opportunity to praise himself.

bottlepalm 4y ago

Between this and his last post, he continues to dismiss ChatGPT as fancy next word statistics.

int_19h 4y ago
That's not the takeaway I got from the "how it works" post. He does describe the word statistics part in detail - which is, of course, pretty fundamental to how it works - but he doesn't claim that it's merely a "stochastic parrot", like many do. On the contrary (highlighting mine): "... this means that ChatGPT—even with its ultimately straightforward neural net structure—is successfully able to “capture the essence” of human language and the thinking behind it. And moreover, in its training, ChatGPT has somehow “implicitly discovered” whatever regularities in language (and thinking) make this possible."
SideQuark 4y ago
That's exactly what ChatGPT is. It's specifically trained as a next word (well, token) predictor. It has no long term plan, only a local window of context. And inference is completely predict the next token, with a large scales stats prediction model.
- bottlepalm 4y ago
  The long term plan is what generates the next word.
  - SideQuark 4y ago
    There is a sliding local, not global, window. There is no long term plan. It does not include context past the window. The GPT4 page states the max context is 32768 tokens. Thus it is local. It does not see anything past this. https://platform.openai.com/docs/models/overview https://platform.openai.com/docs/models/overview
    - bottlepalm 4y ago
      The long term plan stays internal to the network, only the next word is outputted.
      - SideQuark 4y ago
        If you want to call a local plan long term go ahead. People will continue to correct you. It has no "plan" other than local context. I posted their own statements on it. I don't know how much simpler it clearer I can make it. A human can make a long term plan, and write unlimited words about it, and still leverage any part of unlimited context. This is a long term plan. A human can reference 1 token back, 10000 tokens back, 1 million tokens back, any number of tokens back. No LLM can do this, since they have limited context. The site I just posted from GPT4 themselves gave you the context length. Have you ever coded a LLM? Ever read one of their papers and understood it? Do you understand what the word "context" means? We're done.
        bottlepalm 4y ago
        Tokens in GPT are basically short term memory, just like people have short term memory. You can't reference a million tokens back, it's been compressed into your long term memory. For GPT the long term memory is the trained network itself, of which it contains vastly more knowledge than any of us alone. I will say that writing perfectly working application from a prompt requires a plan, even if the output is the next word, there is a higher level plan you're not seeing in the output. This isnt sophisticated markov chains, it's understanding within the network that generates output.

egman_ekki 4y ago

I think he might actually be onto something here. If you saw Wolfram use WolframAlpha, he’s very good at it, but you kinda need to know what it knows and how to ask for it to be smooth and useful. If this initial hurdle gets removed, this power could enable the usage of WolframAlpha by many more people. And honestly it’s a very powerful combo.