4 ms·

The one thing I want everyone to understand about ChatGPT: ChatGPT interfaces with semantics, and not logic. -- That means that any emergent behavior that ap

The one thing I want everyone to understand about ChatGPT: ChatGPT interfaces with semantics, and not logic. -- That means that any emergent behavior that appears logically sound is only an artifact of the logical soundness of its training data. It can only echo reason. The trouble is, it can't choose which reason to echo! The entire purpose of ChatGPT is to disambiguate, but it will always do so by choosing the most semantically popular result. It just so happens that the overwhelming majority of semantic relationships also happen to be logical relationships. That's an emergent effect of the fact that we are usually using words to express logic. So if you mimic human speech well enough to look semantically interesting, you are guaranteed to also appear logically sound. -- I don't see any way to seed such a system to always produce logically correct results. You could feed it every correct statement about every subject, but as soon as you merge two subjects, you are right back to gambling semantics as logic. I also don't see a scalable way to filter the output to be logically sound every time, because that would be like brute-forcing a hash table. OP considers something in the middle, but that's still pretty messy. They essentially want a dialogue between ChatGPT and WolphramAlpha, but that depends entirely on how logically sound the questions generated by ChatGPT are, before they are sent to WolphramAlpha. It also depends on how capable WolphramAlpha was at parsing them. But we already know that ChatGPT is prone to semantic off-by-one errors, so we already know that ChatGPT is incapable of generating logically sound questions. -- As I see it, there is clearly no way to advance ChatGPT into anything more than it is today. Impressive as it is, the curtain is wide open for all to see, and the art can be viewed plainly as what it truly is: magic, and nothing more.

Winsaucerer 4y ago
> As I see it, there is clearly no way to advance ChatGPT into anything more than it is today. Impressive as it is, the curtain is wide open for all to see, and the art can be viewed plainly as what it truly is: magic, and nothing more. (I only RTFA after writing this comment, and I now see that the below is what they're doing) I'm an outsider to this field. My unexpert thought was that perhaps this model could be used to identify the maths components of that question and then we (programmatically) feed that into a different system that gives the answer. That answer then could be provided as context when asking the real question, so the model has access to the mathematical facts. E.g., I just put this question into the playground: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?" Identify the mathematical question in the preceding question and write it as an equation. The answer I got back was: 234241.24211 + 58352342.52544 = ? That could then be fed into a different system that is designed to do mathematical calculations (or logic). In short, not by doing more of what has been done so far, but instead combining these models with different systems. We take the result of that sum from a separate system, and redo the same question, providing the mathematical details as context. E.g., now asking: 234241.24211 + 58352342.52544 = 58586583.76755 Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? With the response: The result is 58586583.76755. Thus getting mathematically accurate answers. Note that if I don't include this context with the calculation completed, I get back the answer: 6,179,563.76754
- amelius 4y ago
  You are asking ChatGPT to interface with a calculator. This is sort of what Wolfram proposes too: interface ChatGPT with a math language. It could work for simple problems. But extracting a mathematical description from a piece of text can require logic too, I suppose, which is where the difficulties start.
- thomastjeffery 4y ago
  You would be trading the problem for another instance of the same problem. When you ask ChatGPT to construct a mathematical question, it will do so the same way it does everything else: by semantic popularity. And that is the problem we are trying to avoid. The semantically popular guess might be logically sound, but it might not. It's a gamble no matter when or where it is done. -- All it takes is what I call a "semantic off-by-one error". That might look like our first problem: > Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? The problem is that a semantically close response is nothing like "234241.24211 + 58352342.52544". It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. That might be the correct number, is more likely to be an incorrect number, and is even likely to be a wordy response. -- So if we follow your thought process to interrupt the guesswork, it would involve restructuring the prompt. > Please restructure the following prompt into a mathematical statement: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?" The output you are hoping for >> 234241.24211 + 58352342.52544 Another completely valid and possible output: >> 234241.24211 / 58352342.52544 Another: >> 2424332.34434 + 535858932.5358 -- There is no way to guarantee the reformulated question is logically equivalent to its original. That's the problem, and the problem cannot be moved. With every step, ChatGPT is guessing. ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is. The only place you can pause an interaction with ChatGPT to do some logic is instead.
  - Winsaucerer 4y ago
    > It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. My understanding of machine learning in general is that this is not how it works, and rather it uses a neural network for a lot of what it does (which isn't merely picking the closest arbitrary text in its training set), though I don't know details about NLP specifically. I am aware that some activities involve ranking the similarity of particular texts, and picking the closest matches (that's how embeddings get used for providing context to prompts), but I didn't think that's how the underlying models operate when, say, asked the question I asked it. Regarding the rest of your remarks, the aim isn't necessarily perfection. You listed some other possible outputs, but the fact is that it didn't offer these options, and instead it offered the correct one, and it seems that a system that uses ChatGPT + something else can provide an improvement over ChatGPT on its own (with the downside being that it requires more resources to provide those answers). Is the other "completely valid and possible output" you provided actually likely? Even a human could possibly make the mistakes you list, but possible doesn't mean probable, and being possible doesn't take away from the results of such a hybrid system being an improvement over ChatGPT on its own. I didn't think guarantees are the game here. Achieving something closer to what humans are (and even some humans struggle with logic and mathematics), and even exceeding it in some places, would be of immense value. > ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is Is this different to what humans do? There are accounts of the brain where it takes sensory input, updates an internal model, and makes predictions (guesses) about what inputs it expects to receive. Getting better at guessing might be the right game to play. i.e., it's not a criticism to say it's guessing, because that's what these models are trying to do. If it can guess reasonably well when a question is mathematical or logical, and can guess reasonably well about how to structure a mathematical or logical equation or statement based on that text, then that could be fed into another system to produce a more accurate answer.
px43 4y ago
Seems trivial to me to bolt on some critical thinking filters. Right now ChatGPT acts like pure Amygdala. You poke it, and it spits out something that looks right based on what it was trained on. This is fine, it's just a demo, and this gives a really deep understanding of the direct limitations of these sorts of LLMs. Ideally a more mature chat bot would just do what I do in situations where some critical thinking or research is called for. I do a quick fact check. I think up a few search terms for Google, Wikipedia, Wolfram Alpha, whatever, assume those results are more logically accurate, and present that data instead of the typical vomit my brain usually spits out. ChatGPT is actually really good at coming up with good search terms for popular search engines, and incorporating results into responses. It just doesn't have anything that reaches out to the web at the moment, which is probably so a certain subsection of the population doesn't have a total panic attack. It seems like fact check functionality would take less than 100 lines of relatively straight-forward python. I bet you could even get ChatGPT to write it.
- owlruns2468 4y ago
  Thought the same thing — we need ways to figure out if what ChatGPT wrote is correct or not. What do you think about an extension like this for fact checking? https://twitter.com/skippperD/status/1613000965929709569 https://twitter.com/skippperD/status/1613000965929709569
- thomastjeffery 4y ago
  Here's what I suspect filters would look like: function critical_filter(response) => Maybe Text { if logic_parse_check(response) == LogicallySound then { return Some response } else { return Nothing } } do { static seed = 0; seed += 1; response = critical_filter(GPT::generate_response(prompt, seed)) } while response != Nothing We are effectively brute-forcing hashes here. Take as many shots in the light as you need to hit, but take no effort to aim. The only solution is a replacement.
CrushItNerds519 4y ago
This comment and many others speculate on the limits of ChatGPT based on assumptions about what ChatGPT does that are not quite accurate. In particular, ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF. As for the speculation itself, e.g., “as soon as you merge two subjects, you are right back to gambling semantics as logic”, it may be suggestive to consider that such models have learned to translate between English and French after having only trained on the two languages separately.
- gavanwilhite 4y ago
  Thank you for this comment! I haven't spent much time learning about how ChatGPT works, but: 'ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF.' was exactly the explanation I needed!
- thomastjeffery 4y ago
  Much like ChatGPT, you seem to have comprehended the words I said, but not their meaning.
  - CrushItNerds519 4y ago
    Nope. If you were to consider carefully enough the language example that I gave, you would see that it already refutes your speculation. Another way would be to apply your argument to humans, who have managed to be useful despite having been trained only on (a broader class of) semantics (i.e., the sum of all qualia) and the logical imperfections that this entails (cf. your comments). Happy new year!
deleted 4y ago
[deleted]
cj 4y ago
This is the best description of the state of ChatGPT that I’ve read thus far. …Unless this whole comment was generated by ChatGPT. (I strongly dislike that I’m starting to second guess whether comments are written by humans or not)
- thomastjeffery 4y ago
  That's because everything written by ChatGPT is a transformation of other stuff that was written by humans. If we trained it on gibberish and nonsense, we would get that, and no one would care. The tricky part is that we don't have much gibberish or nonsense to train with: because of that pesky property of human expression we call logic.
sinuhe69 4y ago
Yes, and this schematic popularity reflects very clearly in mathematical problems: in maths, the schematic probability of plus and minus operation are equal, thus we should see very often ChatGPT first writes a quite convincing reasoning sentence but then produces a nonsensical equation and thus ends up all wrong.
wvoch235 4y ago
I would wager that it will eventually be able to do math and logic, and may already be able to with the right prompting. It can follow instructions quite well, and multi step problems can be solved by breaking the problem into small digestible steps, and having it write out each step, just the same as humans do. If it can't do this yet, it's likely just that it doesn't have enough parameters yet. But I don't see why this couldn't be achieved with fine tuning. But ultimately I think it's just too computationally heavy to do math, remember obscure facts, or track rapidly changing data, with an LLM, and it's far more efficient to pass it off to a piece of specialized software, just the same as humans do.
- anthonypasq 4y ago
  if by "do" math and logic, you mean spit out what seems most reasonable then yes. ChatGPT doesn't "know" anything, its just a prediction engine. if you ask it what 2+2 is, it will tell you 4, but it didnt actually do math. it only said 4 because everyone else said it. there is no generalizable reasoning going on there.
  - wvoch235 4y ago
    See https://news.ycombinator.com/item?id=34340079 https://news.ycombinator.com/item?id=34340079, I would venture to say that this isn't any different to the reason why you know that 2+2 is 4.
- thomastjeffery 4y ago
  An infinite amount of "fine tuning" would solve any problem, yes... ...but what we just described isn't a language processor anymore: it's just a database. There is no need to use a neural network if you already have a known response to every possible input! The entire point of ChatGPT in the first place is to generate a response that doesn't exist in the training data. If all we needed was to look up the answer we wrote down earlier, we already have SQL. What we are talking about here is being able to provide a logically correct response without just pulling it out of a lookup table. Either you can do that, or you can't, and ChatGPT can't.

Winsaucerer 4y ago

> As I see it, there is clearly no way to advance ChatGPT into anything more than it is today. Impressive as it is, the curtain is wide open for all to see, and the art can be viewed plainly as what it truly is: magic, and nothing more. (I only RTFA after writing this comment, and I now see that the below is what they're doing) I'm an outsider to this field. My unexpert thought was that perhaps this model could be used to identify the maths components of that question and then we (programmatically) feed that into a different system that gives the answer. That answer then could be provided as context when asking the real question, so the model has access to the mathematical facts. E.g., I just put this question into the playground: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?" Identify the mathematical question in the preceding question and write it as an equation. The answer I got back was: 234241.24211 + 58352342.52544 = ? That could then be fed into a different system that is designed to do mathematical calculations (or logic). In short, not by doing more of what has been done so far, but instead combining these models with different systems. We take the result of that sum from a separate system, and redo the same question, providing the mathematical details as context. E.g., now asking: 234241.24211 + 58352342.52544 = 58586583.76755 Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? With the response: The result is 58586583.76755. Thus getting mathematically accurate answers. Note that if I don't include this context with the calculation completed, I get back the answer: 6,179,563.76754

amelius 4y ago
You are asking ChatGPT to interface with a calculator. This is sort of what Wolfram proposes too: interface ChatGPT with a math language. It could work for simple problems. But extracting a mathematical description from a piece of text can require logic too, I suppose, which is where the difficulties start.
thomastjeffery 4y ago
You would be trading the problem for another instance of the same problem. When you ask ChatGPT to construct a mathematical question, it will do so the same way it does everything else: by semantic popularity. And that is the problem we are trying to avoid. The semantically popular guess might be logically sound, but it might not. It's a gamble no matter when or where it is done. -- All it takes is what I call a "semantic off-by-one error". That might look like our first problem: > Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result? The problem is that a semantically close response is nothing like "234241.24211 + 58352342.52544". It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. That might be the correct number, is more likely to be an incorrect number, and is even likely to be a wordy response. -- So if we follow your thought process to interrupt the guesswork, it would involve restructuring the prompt. > Please restructure the following prompt into a mathematical statement: "Bob was asked to add 234241.24211 and 58352342.52544, and he wanted to know the result. What is the result?" The output you are hoping for >> 234241.24211 + 58352342.52544 Another completely valid and possible output: >> 234241.24211 / 58352342.52544 Another: >> 2424332.34434 + 535858932.5358 -- There is no way to guarantee the reformulated question is logically equivalent to its original. That's the problem, and the problem cannot be moved. With every step, ChatGPT is guessing. ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is. The only place you can pause an interaction with ChatGPT to do some logic is instead.
- Winsaucerer 4y ago
  > It's just going to be whatever arbitrary text that already exists in the training dataset is closest to the semantic phrasing of the question. My understanding of machine learning in general is that this is not how it works, and rather it uses a neural network for a lot of what it does (which isn't merely picking the closest arbitrary text in its training set), though I don't know details about NLP specifically. I am aware that some activities involve ranking the similarity of particular texts, and picking the closest matches (that's how embeddings get used for providing context to prompts), but I didn't think that's how the underlying models operate when, say, asked the question I asked it. Regarding the rest of your remarks, the aim isn't necessarily perfection. You listed some other possible outputs, but the fact is that it didn't offer these options, and instead it offered the correct one, and it seems that a system that uses ChatGPT + something else can provide an improvement over ChatGPT on its own (with the downside being that it requires more resources to provide those answers). Is the other "completely valid and possible output" you provided actually likely? Even a human could possibly make the mistakes you list, but possible doesn't mean probable, and being possible doesn't take away from the results of such a hybrid system being an improvement over ChatGPT on its own. I didn't think guarantees are the game here. Achieving something closer to what humans are (and even some humans struggle with logic and mathematics), and even exceeding it in some places, would be of immense value. > ChatGPT cannot do anything at all without making a guess, because "guess" is everything that ChatGPT is Is this different to what humans do? There are accounts of the brain where it takes sensory input, updates an internal model, and makes predictions (guesses) about what inputs it expects to receive. Getting better at guessing might be the right game to play. i.e., it's not a criticism to say it's guessing, because that's what these models are trying to do. If it can guess reasonably well when a question is mathematical or logical, and can guess reasonably well about how to structure a mathematical or logical equation or statement based on that text, then that could be fed into another system to produce a more accurate answer.

px43 4y ago

Seems trivial to me to bolt on some critical thinking filters. Right now ChatGPT acts like pure Amygdala. You poke it, and it spits out something that looks right based on what it was trained on. This is fine, it's just a demo, and this gives a really deep understanding of the direct limitations of these sorts of LLMs. Ideally a more mature chat bot would just do what I do in situations where some critical thinking or research is called for. I do a quick fact check. I think up a few search terms for Google, Wikipedia, Wolfram Alpha, whatever, assume those results are more logically accurate, and present that data instead of the typical vomit my brain usually spits out. ChatGPT is actually really good at coming up with good search terms for popular search engines, and incorporating results into responses. It just doesn't have anything that reaches out to the web at the moment, which is probably so a certain subsection of the population doesn't have a total panic attack. It seems like fact check functionality would take less than 100 lines of relatively straight-forward python. I bet you could even get ChatGPT to write it.

owlruns2468 4y ago
Thought the same thing — we need ways to figure out if what ChatGPT wrote is correct or not. What do you think about an extension like this for fact checking? https://twitter.com/skippperD/status/1613000965929709569 https://twitter.com/skippperD/status/1613000965929709569
thomastjeffery 4y ago
Here's what I suspect filters would look like: function critical_filter(response) => Maybe Text { if logic_parse_check(response) == LogicallySound then { return Some response } else { return Nothing } } do { static seed = 0; seed += 1; response = critical_filter(GPT::generate_response(prompt, seed)) } while response != Nothing We are effectively brute-forcing hashes here. Take as many shots in the light as you need to hit, but take no effort to aim. The only solution is a replacement.

CrushItNerds519 4y ago

This comment and many others speculate on the limits of ChatGPT based on assumptions about what ChatGPT does that are not quite accurate. In particular, ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF. As for the speculation itself, e.g., “as soon as you merge two subjects, you are right back to gambling semantics as logic”, it may be suggestive to consider that such models have learned to translate between English and French after having only trained on the two languages separately.

gavanwilhite 4y ago
Thank you for this comment! I haven't spent much time learning about how ChatGPT works, but: 'ChatGPT does not simply output the “most semantically popular result”. That description applies only to the base model, before instruction tuning and RLHF.' was exactly the explanation I needed!
thomastjeffery 4y ago
Much like ChatGPT, you seem to have comprehended the words I said, but not their meaning.
- CrushItNerds519 4y ago
  Nope. If you were to consider carefully enough the language example that I gave, you would see that it already refutes your speculation. Another way would be to apply your argument to humans, who have managed to be useful despite having been trained only on (a broader class of) semantics (i.e., the sum of all qualia) and the logical imperfections that this entails (cf. your comments). Happy new year!

deleted 4y ago

[deleted]

cj 4y ago

This is the best description of the state of ChatGPT that I’ve read thus far. …Unless this whole comment was generated by ChatGPT. (I strongly dislike that I’m starting to second guess whether comments are written by humans or not)

thomastjeffery 4y ago
That's because everything written by ChatGPT is a transformation of other stuff that was written by humans. If we trained it on gibberish and nonsense, we would get that, and no one would care. The tricky part is that we don't have much gibberish or nonsense to train with: because of that pesky property of human expression we call logic.

sinuhe69 4y ago

Yes, and this schematic popularity reflects very clearly in mathematical problems: in maths, the schematic probability of plus and minus operation are equal, thus we should see very often ChatGPT first writes a quite convincing reasoning sentence but then produces a nonsensical equation and thus ends up all wrong.

wvoch235 4y ago

I would wager that it will eventually be able to do math and logic, and may already be able to with the right prompting. It can follow instructions quite well, and multi step problems can be solved by breaking the problem into small digestible steps, and having it write out each step, just the same as humans do. If it can't do this yet, it's likely just that it doesn't have enough parameters yet. But I don't see why this couldn't be achieved with fine tuning. But ultimately I think it's just too computationally heavy to do math, remember obscure facts, or track rapidly changing data, with an LLM, and it's far more efficient to pass it off to a piece of specialized software, just the same as humans do.

anthonypasq 4y ago
if by "do" math and logic, you mean spit out what seems most reasonable then yes. ChatGPT doesn't "know" anything, its just a prediction engine. if you ask it what 2+2 is, it will tell you 4, but it didnt actually do math. it only said 4 because everyone else said it. there is no generalizable reasoning going on there.
- wvoch235 4y ago
  See https://news.ycombinator.com/item?id=34340079 https://news.ycombinator.com/item?id=34340079, I would venture to say that this isn't any different to the reason why you know that 2+2 is 4.
thomastjeffery 4y ago
An infinite amount of "fine tuning" would solve any problem, yes... ...but what we just described isn't a language processor anymore: it's just a database. There is no need to use a neural network if you already have a known response to every possible input! The entire point of ChatGPT in the first place is to generate a response that doesn't exist in the training data. If all we needed was to look up the answer we wrote down earlier, we already have SQL. What we are talking about here is being able to provide a logically correct response without just pulling it out of a lookup table. Either you can do that, or you can't, and ChatGPT can't.