5 ms·

ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actua

ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the 80's who thinks the answer to an essay question is to copy it from an encyclopedia, only the "encyclopedia" is very large and contains multiple sources. So, the big take away needs to be that there is absolutely no understanding, no cognizance of any kind, no language comprehension going on. The answers look good because they contain all the same words as the most popular answers people have already written which the system scanned. So ChatGPT turns out to great for parsing and summarizing documents, if that's something you need. But, since it doesn't know fact from fiction, it cannot apply logic or math, and it cannot perform reasoning or analysis, it's not good for finding out facts or discerning truth. Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same.

phillipcarter 4y ago
I don't think this is an accurate depiction and I think there's a lot more going on there than you're saying there is. The models also involve Codex, which is able to effectively do pattern matching with more context you give it so that it can emit novel code. I've seen this happen numerous times with Copilot, where it works very well the more open documents you have, especially if those documents are related. True, it doesn't have a full semantic understanding of your codebase, but it's good enough to usually generate the right code once it's been "seeded" with something like an updated type definition that you then need to plumb through various parts of a codebase.
- dtagames 4y ago
  Yes, I use Copilot, too for that very reason. But we need to be very careful about words like "semantic" and "understanding" as the method is neither. We'll get the best use out of these technologies if we don't ascribe to them magical qualities they don't have. Poke around. You'll find out it just statistical math with tokens (letters and punctuation). No meaning is ascribed to anything.
  - phillipcarter 4y ago
    I...did say that it doesn't have a full semantic understanding of your code. Copilot uses tree-sitter under the covers to be able to figure out several things that you can get by analyzing syntax alone, which is actually quite far. It lets you identify what kinds of declarations (e.g., a type) and what kind of expressions (e.g., pattern matching on that type) exist, provided there's a grammar for it, and then lets that influence the suggested code. This isn't perfect because indeed, you need a full language service to actually understand things, especially in resolving ambiguities in symbols (like shadowing in some languages), but, like I said...it's good enough.
    - dtagames 4y ago
      Yes, Copilot has the advantage that programming language syntax is highly constrained and regularized, allowing it to slot-in your own variable names and functions into new code as you write it. This is the best feature of Copilot. It's not part of ChatGPT because that's not meant for writing code. And I wouldn't call it "understanding." But, it is useful.
krackers 4y ago
> you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! This is not sufficient to explain how LLMs are able to synthesize novel, coherent poems or song lyrics. What you're describing seems closer to a markov model. So far I have yet to see a good explanatin of _why_ transformer models seem to have this emergent behavior as you scale it up. It's easy to see by construction that the stacked attention layers will work well to predict the masked token in "Capital of France is [MASK]", or how you can throw an additional head on top for sentiment analysis (which I recall was implemented by literally adding a dummy token at the start into which information about the entire sentence gets embedded). But it's not obvious (to me at least) that this would somehow generalize into being able to do the things chatGPT does.
- dtagames 4y ago
  Ah, but in fact it does. Code and poetry seem different to people but tokens are tokens to the computer, and it knows not one from the other. The reason you get a poetry-style answer when you ask for one is that the mix of words and styles can be taken from different parts of the corpus. This is how you can have it write (bad) poetry about something for which only prose was scanned.
  - nullish_signal 4y ago
    I think that the levels of Associativity between Linguistic Terms goes beyond a mere "mix of words and styles". When pulling on one part of the Prompt causes portions of the Output to shift or vanish, or change genre or word choice or composition or focal point or focal distance - - - then, that's the very opposite of "mix" -That's "Stable Diffusion" in text2img, or the "Reversal of Entropy" in Physics terms.. I'm still blown away that computers have become inage "Imaginers", decades after Humans first started doing 3D through Computers. It's really all quite unbelievable.
  - maxbond 4y ago
    I'd propose that you come up with a prompt to test your hypothesis and try it out. I think seeing is believing here, I don't know how ChatGPT works but after experimenting with it I am sure it's not simply looking for runs of similar words to the prompt in it's training set. Here's an example to consider: https://news.ycombinator.com/item?id=33874110 https://news.ycombinator.com/item?id=33874110
nullish_signal 4y ago
OP question mentions specific network types, while reply uses a vague "million-book library" metaphor, but at least mentions LLM? Also arbitrarily sorts human { teacher, journalist, writer } as "people who do custom work with each client" which makes a bit of sense for a Teacher in a small-class environment. But Writers and Journalists write at SCALE, and while each may speak with gusto to individuals just as well as they do their job, tailoring custom content to individuals is not their job. Not to mention that individual-tailored model files are very handy with AI, share-able as those diles may be. I call ChatGPT on you!!! And also do not know anything about your question, OP...
- dtagames 4y ago
  I wouldn't deign to have ChatGPT write my comments. They're all me! Obviously, an author's audience can include more than one person. But every good piece of human writing is tailored for some audience. That was my point. You've heard the expression, "Know your audience?" ChatGPT cannot.
  - nullish_signal 4y ago
    ChatGPT's "audience" can range from "hate speech for anti-bias training" to "Teenage Mutant Ninja Turtles Deliver Impassioned Speech to The World" I would also like to use the No True Scotsman fallacy, and assert that no Human can ever really "know" or "write for" anybody's perfect preferences.
    - dtagames 4y ago
      I wasn't asking for perfect preferences. I was asking for minimal cognizance, which the tool cannot provide. ChatGPT is more like a parrot than tech dreamers like to admit.
      - nullish_signal 4y ago
        I'll grant you that ChatGPT is more Parrot than All-Seeing Oracle, but no Parrot (or even Google) will take nginx errors for input, and give me back chmod and chown commands to fix the permission issues. AND! the commands it gave me were tailored to match my user and any directories/websites i told it I was working on ! !
the_gipsy 4y ago
> So, the big take away needs to be that there is absolutely no understanding, no cognizance of any kind, no language comprehension going on. Okay, but what is understanding, cognizance, comprehension? And so on for further precision of definitions, until you reach that vague nebulous area that is consciousness. We simply haven't figured that out, period. So we cannot really say that this AI does not have those qualities, unless its output is obviously showing it. Which ChatGPT does not, most of the time.
- deleted 4y ago
  [deleted]
6nf 4y ago
> it cannot apply logic Have you tried?
- dtagames 4y ago
  The tool can parrot back to you "logical" arguments which it has already scanned. It cannot figure out the logic behind them, nor seen when it's wrong, nor correct any errors, nor draw new logical conclusions.
  - 6nf 4y ago
    Are you sure about that?
pushedx 4y ago
Ask ChatGPT to write a song lyric for your favorite band in the style of your favorite poet, with a specific unusual topic. Then claim that the model is “really very simple”
- dtagames 4y ago
  At the end of the day, all computing is very simple. What's impressing you here is the size of the dataset and the speed of the retrieval, brought about by advances in hardware. Also, we like the well-formed writing which is created by filtering and massaging the parroted text into a new set of sentences. It is impressive, and people who don't know how it really works will think it's capable of all kinds of things it's not. Of course, this has been true forever. Just ask anyone who writes code about what non-devs think software can do.
  - pushedx 4y ago
    Have you used ChatGPT? Your understanding of GPTs and neural nets in general is consistently flawed. You are describing a Markov model or at best an SVM model. Would you make the same types of claims about Stable Diffusion, that it is “piecing together pieces of existing images?” That is not how these models work. Similar to the temporal memory that ChatGPT will appear to have during a conversation, “inpainting” with Stable Diffusion produces entirely novel output with context. Have you tried it out?
    - dtagames 4y ago
      I have and I wrote extensively about it after seeing the unpopularity of my original comment. There's a link to my long Medium story about it in another comment. Would I say that DALL-E, etc. also piece together existing images? Yes, of course, since that is the only way that technically it could work. All of computing is "input-process-output." There is no "create" step in any of computing. The pieces that DALL-E pieces together are pixels of a certain color and brightness and placement relative to other pixels. Together, we perceive this as an image. If they weren't arranged that way, we'd just see noise or gray. The same is true with ChatGPT. Unless they're put in a particular order, words just make word salad. It is the order that gives them their perceived meaning in a sentence. ChatGPT crafts "new" sentences by rearranging words from similar sentences it has already seen, just like DALL-E creates new paintings by rearranging pixels from paintings it has already seen. A recent video from Noam Chomsky explains this from his perspectives on learning and language: https://www.youtube.com/watch?v=PBdZi_JtV4c https://www.youtube.com/watch?v=PBdZi_JtV4c
stevenhuang 4y ago
> Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same. ChatGPT can do this if you just ask. E.g., explain as if I'm 5 and it will use simple words and phrases. Explain like I'm an expert and it will use technical jargon and expect a deeper shared knowledge with which it can draw from. You shouldn't write so authoritatively when clearly you haven't explored its capabilities.
- rakejake 4y ago
  This is true but does not refute OP's point. There is an entire subreddit for ELI5 that is probably part of its training data. So if you ask for ELI5, that pattern IS part of the training corpus.
  - weird-eye-issue 4y ago
    It is more complex than that, it isn't just scraping /r/ELI5 lol Ask it a question about JavaScript and tell it to respond as a 1920s gangster and it will happily oblige and do a great job. And just to make sure we are on the same page, JavaScript was not available in the 20s
    - dtagames 4y ago
      It is just scraping and reorganizing words while including words from your prompt in the equation. I already mentioned that the style (gangster) and vocabulary (JavaScript) can come from different scanned documents. Cool? Yes. Is it learning, or understanding? No.
      - weird-eye-issue 4y ago
        If it's not learning then how can you tell it abstractly why it's code is wrong then have it fix it? Without telling it specifically "change this to this"
dtagames 4y ago
I wrote this[0] to better explain my wildly unpopular answer above. Everything is absolutely correct here, but it's not a popular view right now. Often very smart people like those on HN can be fooled by the "magic" of tech that might not really exist in the way they think it does. [0] https://medium.com/gitconnected/behind-the-curtain-understanding-the-magic-of-chatgpt-3bbd23f0fbb3 https://medium.com/gitconnected/behind-the-curtain-understan...

phillipcarter 4y ago

I don't think this is an accurate depiction and I think there's a lot more going on there than you're saying there is. The models also involve Codex, which is able to effectively do pattern matching with more context you give it so that it can emit novel code. I've seen this happen numerous times with Copilot, where it works very well the more open documents you have, especially if those documents are related. True, it doesn't have a full semantic understanding of your codebase, but it's good enough to usually generate the right code once it's been "seeded" with something like an updated type definition that you then need to plumb through various parts of a codebase.

dtagames 4y ago
Yes, I use Copilot, too for that very reason. But we need to be very careful about words like "semantic" and "understanding" as the method is neither. We'll get the best use out of these technologies if we don't ascribe to them magical qualities they don't have. Poke around. You'll find out it just statistical math with tokens (letters and punctuation). No meaning is ascribed to anything.
- phillipcarter 4y ago
  I...did say that it doesn't have a full semantic understanding of your code. Copilot uses tree-sitter under the covers to be able to figure out several things that you can get by analyzing syntax alone, which is actually quite far. It lets you identify what kinds of declarations (e.g., a type) and what kind of expressions (e.g., pattern matching on that type) exist, provided there's a grammar for it, and then lets that influence the suggested code. This isn't perfect because indeed, you need a full language service to actually understand things, especially in resolving ambiguities in symbols (like shadowing in some languages), but, like I said...it's good enough.
  - dtagames 4y ago
    Yes, Copilot has the advantage that programming language syntax is highly constrained and regularized, allowing it to slot-in your own variable names and functions into new code as you write it. This is the best feature of Copilot. It's not part of ChatGPT because that's not meant for writing code. And I wouldn't call it "understanding." But, it is useful.

krackers 4y ago

> you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! This is not sufficient to explain how LLMs are able to synthesize novel, coherent poems or song lyrics. What you're describing seems closer to a markov model. So far I have yet to see a good explanatin of _why_ transformer models seem to have this emergent behavior as you scale it up. It's easy to see by construction that the stacked attention layers will work well to predict the masked token in "Capital of France is [MASK]", or how you can throw an additional head on top for sentiment analysis (which I recall was implemented by literally adding a dummy token at the start into which information about the entire sentence gets embedded). But it's not obvious (to me at least) that this would somehow generalize into being able to do the things chatGPT does.

dtagames 4y ago
Ah, but in fact it does. Code and poetry seem different to people but tokens are tokens to the computer, and it knows not one from the other. The reason you get a poetry-style answer when you ask for one is that the mix of words and styles can be taken from different parts of the corpus. This is how you can have it write (bad) poetry about something for which only prose was scanned.
- nullish_signal 4y ago
  I think that the levels of Associativity between Linguistic Terms goes beyond a mere "mix of words and styles". When pulling on one part of the Prompt causes portions of the Output to shift or vanish, or change genre or word choice or composition or focal point or focal distance - - - then, that's the very opposite of "mix" -That's "Stable Diffusion" in text2img, or the "Reversal of Entropy" in Physics terms.. I'm still blown away that computers have become inage "Imaginers", decades after Humans first started doing 3D through Computers. It's really all quite unbelievable.
- maxbond 4y ago
  I'd propose that you come up with a prompt to test your hypothesis and try it out. I think seeing is believing here, I don't know how ChatGPT works but after experimenting with it I am sure it's not simply looking for runs of similar words to the prompt in it's training set. Here's an example to consider: https://news.ycombinator.com/item?id=33874110 https://news.ycombinator.com/item?id=33874110

nullish_signal 4y ago

OP question mentions specific network types, while reply uses a vague "million-book library" metaphor, but at least mentions LLM? Also arbitrarily sorts human { teacher, journalist, writer } as "people who do custom work with each client" which makes a bit of sense for a Teacher in a small-class environment. But Writers and Journalists write at SCALE, and while each may speak with gusto to individuals just as well as they do their job, tailoring custom content to individuals is not their job. Not to mention that individual-tailored model files are very handy with AI, share-able as those diles may be. I call ChatGPT on you!!! And also do not know anything about your question, OP...

dtagames 4y ago
I wouldn't deign to have ChatGPT write my comments. They're all me! Obviously, an author's audience can include more than one person. But every good piece of human writing is tailored for some audience. That was my point. You've heard the expression, "Know your audience?" ChatGPT cannot.
- nullish_signal 4y ago
  ChatGPT's "audience" can range from "hate speech for anti-bias training" to "Teenage Mutant Ninja Turtles Deliver Impassioned Speech to The World" I would also like to use the No True Scotsman fallacy, and assert that no Human can ever really "know" or "write for" anybody's perfect preferences.
  - dtagames 4y ago
    I wasn't asking for perfect preferences. I was asking for minimal cognizance, which the tool cannot provide. ChatGPT is more like a parrot than tech dreamers like to admit.
    - nullish_signal 4y ago
      I'll grant you that ChatGPT is more Parrot than All-Seeing Oracle, but no Parrot (or even Google) will take nginx errors for input, and give me back chmod and chown commands to fix the permission issues. AND! the commands it gave me were tailored to match my user and any directories/websites i told it I was working on ! !

the_gipsy 4y ago

> So, the big take away needs to be that there is absolutely no understanding, no cognizance of any kind, no language comprehension going on. Okay, but what is understanding, cognizance, comprehension? And so on for further precision of definitions, until you reach that vague nebulous area that is consciousness. We simply haven't figured that out, period. So we cannot really say that this AI does not have those qualities, unless its output is obviously showing it. Which ChatGPT does not, most of the time.

deleted 4y ago
[deleted]

6nf 4y ago

> it cannot apply logic Have you tried?

dtagames 4y ago
The tool can parrot back to you "logical" arguments which it has already scanned. It cannot figure out the logic behind them, nor seen when it's wrong, nor correct any errors, nor draw new logical conclusions.
- 6nf 4y ago
  Are you sure about that?

pushedx 4y ago

Ask ChatGPT to write a song lyric for your favorite band in the style of your favorite poet, with a specific unusual topic. Then claim that the model is “really very simple”

dtagames 4y ago
At the end of the day, all computing is very simple. What's impressing you here is the size of the dataset and the speed of the retrieval, brought about by advances in hardware. Also, we like the well-formed writing which is created by filtering and massaging the parroted text into a new set of sentences. It is impressive, and people who don't know how it really works will think it's capable of all kinds of things it's not. Of course, this has been true forever. Just ask anyone who writes code about what non-devs think software can do.
- pushedx 4y ago
  Have you used ChatGPT? Your understanding of GPTs and neural nets in general is consistently flawed. You are describing a Markov model or at best an SVM model. Would you make the same types of claims about Stable Diffusion, that it is “piecing together pieces of existing images?” That is not how these models work. Similar to the temporal memory that ChatGPT will appear to have during a conversation, “inpainting” with Stable Diffusion produces entirely novel output with context. Have you tried it out?
  - dtagames 4y ago
    I have and I wrote extensively about it after seeing the unpopularity of my original comment. There's a link to my long Medium story about it in another comment. Would I say that DALL-E, etc. also piece together existing images? Yes, of course, since that is the only way that technically it could work. All of computing is "input-process-output." There is no "create" step in any of computing. The pieces that DALL-E pieces together are pixels of a certain color and brightness and placement relative to other pixels. Together, we perceive this as an image. If they weren't arranged that way, we'd just see noise or gray. The same is true with ChatGPT. Unless they're put in a particular order, words just make word salad. It is the order that gives them their perceived meaning in a sentence. ChatGPT crafts "new" sentences by rearranging words from similar sentences it has already seen, just like DALL-E creates new paintings by rearranging pixels from paintings it has already seen. A recent video from Noam Chomsky explains this from his perspectives on learning and language: https://www.youtube.com/watch?v=PBdZi_JtV4c https://www.youtube.com/watch?v=PBdZi_JtV4c

stevenhuang 4y ago

> Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same. ChatGPT can do this if you just ask. E.g., explain as if I'm 5 and it will use simple words and phrases. Explain like I'm an expert and it will use technical jargon and expect a deeper shared knowledge with which it can draw from. You shouldn't write so authoritatively when clearly you haven't explored its capabilities.

rakejake 4y ago
This is true but does not refute OP's point. There is an entire subreddit for ELI5 that is probably part of its training data. So if you ask for ELI5, that pattern IS part of the training corpus.
- weird-eye-issue 4y ago
  It is more complex than that, it isn't just scraping /r/ELI5 lol Ask it a question about JavaScript and tell it to respond as a 1920s gangster and it will happily oblige and do a great job. And just to make sure we are on the same page, JavaScript was not available in the 20s
  - dtagames 4y ago
    It is just scraping and reorganizing words while including words from your prompt in the equation. I already mentioned that the style (gangster) and vocabulary (JavaScript) can come from different scanned documents. Cool? Yes. Is it learning, or understanding? No.
    - weird-eye-issue 4y ago
      If it's not learning then how can you tell it abstractly why it's code is wrong then have it fix it? Without telling it specifically "change this to this"

dtagames 4y ago

I wrote this[0] to better explain my wildly unpopular answer above. Everything is absolutely correct here, but it's not a popular view right now. Often very smart people like those on HN can be fooled by the "magic" of tech that might not really exist in the way they think it does. [0] https://medium.com/gitconnected/behind-the-curtain-understanding-the-magic-of-chatgpt-3bbd23f0fbb3 https://medium.com/gitconnected/behind-the-curtain-understan...