9 ms·

Why are so many giants of AI getting GPTs so badly wrong?

51 points by vinnyglennon 3y ago

mediumsmart 3y ago
I have googled the bucket on the door too and the author is right, it is nowhere on the Internet to be found. How Chomsky this get could so wrong?
cjbprime 3y ago
Nice article! I think you nailed it. The dismissal of AI reasoning and world modeling from famous technologists who should know better is such a disappointment. I have a general suggestion to avoid these word puzzles, because someone is just going to claim that they are analogous to something that was in the training set, even if not an exact match. Better to just remove the possibility. I think four points would make the argument slightly stronger: (1) If you write a small (for example) Python script, using variable names and strings that are generated using randomness and performing operations on those random data, and ask GPT-4 what the Python script will output when someone runs it, it will tell you, while describing what each line does. This is not possible without reasoning, and GPT-4 does not have access to a real Python interpreter. The only way it can answer the question is by simulating a Python interpreter from its own knowledge of how the language works. It will likely make small mistakes if you ask for too much calculation before it outputs any tokens in the answer, but the fact that it often succeeds should be seen as a proof of reasoning ability that I think is stronger than the examples in your post. (2) If you take another similar script and call `dis.dis(script)` on it, and feed GPT-4 the compiled bytecode only, it will (a) decompile the bytecode back to source code, and (b) tell you what the bytecode will do when executed. This is wild, and even more impressive than (1). (3) "But transformers are just functions?". No, they are generalized function approximators. Anyone familiar with neural nets from decades ago should already be well aware of this from gradient descent and backpropagation. The fact that they are general function approximators is not a reason that argues against them being able to learn algorithms they were not trained on. It's the reason they can learn algorithms they were not trained on. (4) "All GPTs set out to do is to predict the next token. We just update their weights to optimize for this. Therefore they can’t think". I think this is another case where the putative criticism is actually the correct intuition to explain why it works. The next step is to think: what does "predict the next token" mean, anyway? Or more importantly, since we're performing optimization: what does it mean to "get better at" that? If you're a model that's predicting the next token as best you can with surface statistics, and another model is doing better than you, what's going to be different about that model? The answer is that you can always get better at predicting the next token. The next token could be the answer to a math problem, or a riddle requiring a nuanced understanding of language or humor, or a stylized rap song requiring merging creativity with constraint following. To improve at next token prediction, LLMs have clearly -- if you agree they have a world model -- started predicting not just text, but the causes of text. The text is a description of the world, and that description is being studied, all so that we can have a slightly better prediction of the next token each time.
vaidhy 3y ago
One of my biggest gripes is around the Chinese room argument and these are different forms of the same. It is assumed that a mechanistic process could not have "intelligence" and it has been morphed into many forms without anyone defining what is intelligence (or conscious for that matter). The arguments fall into one of these two buckets: 1. Human exceptionalism (religious argument) - Humans (and maybe some of the lower creatures) have intelligence, but others and things created by humans do not have it. It does not matter what or how you define intelligence, but by definition only some living beings can have it. 2. Frankenstein (Shit..it is scary argument) - We have seen these emergent properties and it is not something that we can explain easily to people. Everyone is already wary and let us placate the masses by saying it is not scary and in fact, it is just a small tame cute furball. Personally, I think all these arguments are useless and misplaced. If we create something that can do critical reasoning, all the more power to us.
- DougWebb 3y ago
  You complain about arguments that don't define "intelligence" and what it means for AI to have that, and then you say: If we create something that can do critical reasoning, all the more power to us. It seems to me you're not defining "critical reasoning", and that it's effectively a synonym for "intelligence", at least as applied to AI capabilities. I think AI responses are like the responses of a person who's very new to a subject and doesn't understand it, but they know all of the terminology and how the terms are usually used together. They can be helpful, and often correct, but only an expert in the field is going to know for sure if they're correct. And my fear about AI is not the AI itself, it's how the non-experts are going to use the AI to try to replace the experts in fields where being correct is really important. That will work, but only sometimes. If we don't make the people who choose the AI this way responsible for the outcomes, we're going to have a real mess on our hands.
  - vaidhy 3y ago
    You are correct that I did not define critical reasoning. However, here is an sample case - I set a context and ask the answer from the computer. If the computer rightly understands the context and answers the question, then that is critical reasoning for me. Now that I type it out, it sounds like a version of a Turing test and I do not know if we can do better. So, in your hypothetical case, I would argue that using the terminology and providing the answers assume some level of intelligence. We get caught up in the training process and automatically assume what computer can infer because it is what we can infer. But the computer is not human and its world model is not the same as our world model. Your fear is reasonable, but religion has gotten away with it for so long :)
YeGoblynQueenne 3y ago
What an irritating article. The simplest answer, respecting the article's own assumption is: They are giants of AI and they are not getting it wrong. An immediate corrolary of which is: the article's author is the one getting it wrong. Eyballing the attempts to show that the "giants of AI" are wrong by poking GPT4 I'd say there's a good chance of that. An even simpler answer, violating the assumption is: they're not giants of AI and that's why they're getting it wrong. Or is that a simpler answer? Well, now we have to explain how a guy with a machine learning PhD writing on Substack is getting it right, when LeCun, Brooks, and so on are getting it wrong. Chomsky is not an AI expert, so there's that: the author doesn't quite know who he's talking about.
- llampx 3y ago
  Chomsky is a writer. He is probably scared out of his wits that his job will be taken over by a bunch of transformers.
  - beezlewax 3y ago
    I doubt he is in a position to care profundly about losing his job. He's a highly successful person. Presumably has made decent bank over the years.
  - CBarkleyU 3y ago
    Yes, the 94 years old Noam Chomsky is scared shitless that AI will leave him jobless. This is definitely a sane take on this issue.
    - throw310822 3y ago
      The entity that wrote it clearly lacks a world model :)
  - mcv 3y ago
    He's not a writer, he's a professor in linguistics. He almost singlehandedly pushed the field towards a more technical analysis and formalisation of grammar suitable for automatic language parsing and translation. If he isn't retired already, it's only out of personal choice, and not because he's desperate for a job.
xg15 3y ago
What I would really like to see more in the future is LLMs trained from scratch (i.e. not finetuned) on controlled training sets. Currently, whenever a LLM produces a result that looks like deeper reasoning, it's easy to explain that with "eh, there was probably something similar in the trainset that the LLM just had to tweak slightly" - and that argument is basically impossible to refute or confirm, because the trainsets are huge, chaotic and often only partially available (Books2...). Therefore we can't really make any good statements about what information could or could not be pulled from the trainset. I think it would be interesting to pretrain an LLM on a selective corpus - say, the entirety of Discworld or A Song of Ice and Fire novels - and then observe if the model can produce output that is not obviously stitched together from the books.
- TuringTest 3y ago
  > eh, there was probably something similar in the trainset that the LLM just had to tweak slightly Look for examples of mixed image-text generative AIs, and see how they are able to understand visual jokes and memes and explain the source of their humour. To do that, the AI not only has to recognize and classify the items in the image. It also has to identify the context of the items in the image, and figure out what elements are unusual in that context. All that process is something that could be only imitated if the training set contained exactly the same meme; as the source of humour will be different for each image. But the AI is capable of doing it even for memes not included in the training set.
  - ChatGTP 3y ago
    There are datasets like this: https://old.reddit.com/r/ExplainTheJoke/ https://old.reddit.com/r/ExplainTheJoke/ and books on writing jokes etc. I can see why people are skeptical, not to say it isn't curious the models can do what they do, but I understand what the parent is asking for.
- Terretta 3y ago
  > What has been will be again, what has been done will be done again; there is nothing new under the sun. Arguably, most everything anyone can encounter has happened to someone, and likely been written about. Imagine a toddler learning from everything that has happened everywhere it happened all at once, as recorded in a journal. Now, by contrast, a toddler learning from Discworld. Is that remotely enough? How much is enough? Turns out you can look this up, as it has been graphed: it's rather more than Discworld to get these "emergent" properties. Just as it would be rather more than Discworld before a toddler without our everything journal could make sense of Discworld itself. To your point, before reading Discworld, our toddler would need a great deal of huge chaotic training sets. Stands to reason, so might the LLM.
JoeAltmaier 3y ago
It seems clear that the AI is not being trained on logic, physics or chemistry. Yet it's read and internalized (whatever that means) those subjects somehow. It does seem simplistic to say "AI isn't programmed with logic or physics" when it's likely read several textbooks on those subjects. It might be expected to gain something from that, similar to any other intelligence that read those texts. Maybe the confusion stems from "what the AI was programmed to do". It certainly wasn't programmed to do anything but make a model of text and maybe deeper structure of that text (meaning) using an enormous corpus. But when we're exercising an AI, we're not exercising that programming. Not at all. We're exercising the model it made. And for that reason, the AI-model may indeed have some 'understanding' of physics and chemistry, though it had never been 'programmed' to know that. All our previous instincts about computers is, they do what we've programmed them to do. When we try to apply that to AI, we go pretty far off the mark. Because the computer is perhaps the least interesting part of the AI. It's that model we should be concerned about.
lsy 3y ago
This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests, and is mainly just a dude saying "well, certainly this seems like it would be outside the training set and the LLM seems to handle it OK so it must be reasoning". Meanwhile, we have no way of inspecting the training set, and even if we did the data and model are so massive it would be impossible to tell whether it was doing simple substitution or whatever. Ultimately e.g. Brooks must be right that the model has no connection to the world, because to the model there is no world, only text. If the training text includes things that are untrue in the world, the model cannot access the world to make a final determination that its internal statistics are faulty. If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text.
- PoignardAzur 3y ago
  > This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests > Ultimately e.g. Brooks must be right that the model has no connection to the world Isn't that a huge double-standard? If you want to say the AI has a world model, no matter how many supporting examples of informal experiences you bring up, they don't count because you don't have formal justification. But if you want to say the AI doesn't have a world model, you don't need formal justification or even any supporting data or falsifiable predictions, you just say "it couldn't possibly be the" case, and that's enough. (Also, the whole OthelloGPT thing seems as close to formal evidence for an internal world model as you can get.)
- ericb 3y ago
  > If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text. You could say the same for human perception, though. We don't actually "see" what our brain thinks we see, our brain fills in the image. Nonetheless, it doesn't stop us from having and using our slightly-inaccurate world-model based on this info. Also, we can often infer our faulty perceptions by comparative methods like "banana for scale" or looking for conflicts and contradictions.
RecycledEle 3y ago
I love the irony of Fergal Reid saying that other people are getting things wrong, while thinking it is possible to fill a bucket half full of "superglue fresh from the tubes." Nobody is going to half-fill a bucket that can hold multiple apples with that much superglue. It would cost too much, it would smell too bad, and it would dry before getting the bucket close to half full. He accidentally points out the biggest problem with GPT AIs: they believe what people posted online, which is partly made-up nonsense. It is hilarious that he things that cartoon prank would work. Anyone pushing on the door will notice it does not want to open due to the weight of the bucket. The bucket will rarely hit them, and will never land over their head. If it does hit them, it will leave them bleeding. Head wounds bleed a lot. If you disagree with me, try even the water-bucket-over-a-door version of the prank. Nobody will fall for it, but if they do, they will be injured by the bucket.
- Terretta 3y ago
  We're not dealing with the "real" world. We're dealing with an alternate world that gets written about. In that world, the bucket trick usually works fine.
TacticalCoder 3y ago
I'm so tired about these examples: I don't find them impressive. Here's what I got several times with GPT and other who are using it must have encountered the same issue (simplifying the example to show the problem I encounter all the time): GPT: Frodo says that 2 + 2 is 5. user: that is wrong, 2 + 2 is 4. GPT: You are correct, I was wrong. Here's the correct version: Gandalf say that 2 + 2 is 5. This happens to me all the time. The textual model only evaluates part of the input and focus, for whatever reason, on the wrong part of the text and then from there it's a descent into madness / futility. This exact "partial input" issue was shown yesterday in the "Eliza vs GPT" vid: Eliza: "Did you come to me because you are programmed to simulate human language?" GPT: "Yes, that's correct. I'm programmed to simulate human language behavior..." Everybody using it must have noticed that. It's not answering what was asked. It's only partially taking into account the user's input. I don't know if it's because the way questions are parsed or the way the model is compressed but something is very seriously off. And don't "you're prompting it wrong" me.
- ericb 3y ago
  I'd bet you're using the default model, GPT-3. If not, can you give me the non-summarized version? I'd love to see examples that GPT-4 gets tripped up on--I'd find them reassuring.
- ravenstine 3y ago
  That's definitely not how it works out for me when I use your example with GPT-4 or 3.5. ASSISTANT: Frodo says 2 + 2 is 5. USER: that is wrong, 2 + 2 is 4. ASSISTANT: You are correct. The correct sum of 2 + 2 is indeed 4. It seems Frodo made an error in his calculation. What GPT are you using that's giving you that answer? Every time someone brings up one of these math examples, I try it myself and the GPT gets it right.
- jazzyjackson 3y ago
  > other who are using it must have encountered the same issue > And don't "you're prompting it wrong" me. You can be prompting it expertly and still see the same errant behavior over and over again, because the models are nondeterministic, they function by taking a different path every time. Across the distribution of all users and all prompts, there will be a small subset of users who experience miraculous, seemingly intelligent behavior, and there will be a small subset of users who just get junk responses over and over again (the remainder of users get a mixed bag which tells them that the AI may be useful but unreliable). Personally I think we should object to nondeterministic behavior out of machines, with exceptions for cryptography and physics simulations. It's going to lead to highly un-equal outcomes with no rhyme or reason. Psychologically risky.
oli5679 3y ago
Here is an example of gpt-4 not having a realistic model of the world. “ If it takes 4 hours for 4 clothing items to dry in the sun, how long does it take for 8 clothing items to dry?” “[maths calculations] Therefore, it would take 8 hours for 8 clothing items to dry in the sun“ It was much easier to find examples like this with previous models, but it’s still possible.
- Jtsummers 3y ago
  Bard agrees, with the explanation that the drying time is directly proportional to the number of items of clothing. Bard is, however, smarter than the average manager: It knows that 9 women can't make a baby in 1 month.
- nielsole 3y ago
  > [...] if we assume that the drying capacity is linear and the environmental conditions are the same, then it doesn't actually matter how many items you are drying. If it takes 4 hours for 4 items to dry, then each item takes 4 hours to dry individually. Therefore, if you have 8 items to dry, and the conditions are right for drying (i.e., enough space, sunlight, etc.), all of them would still dry in 4 hours.
- CognitiveLens 3y ago
  I know that some other comments indicate that it's not fair to say "you're prompting it wrong", but using ChatGPT with GPT-4, and using advice to set up prompts to give accurate answers, GPT-4 provides an excellent answer here ME: You understand everyday tasks very well and can correctly calculate or estimate the time it takes to complete various tasks. I will ask you questions about how long it takes for various things to happen and you will tell me the correct answer using brief, accurate statements. If it takes 4 hours for 4 clothing items to dry in the sun, how long does it take for 8 clothing items to dry? ChatGPT: The drying time for clothes in the sun typically depends on the amount of sunlight and wind, as well as the type and thickness of the clothing material. However, if we are considering that the drying conditions remain constant and that the drying rate does not depend on the number of clothing items (as they are spread out and not piled on each other), it would still take approximately 4 hours to dry 8 clothing items. This assumes the clothes are spread out sufficiently to all receive equal sunlight and airflow.
pk-protect-ai 3y ago
I have much better example: https://www.phind.com/search?cache=626d4ad1-f0ae-4c1c-a2e7-738fba575892 https://www.phind.com/search?cache=626d4ad1-f0ae-4c1c-a2e7-7...
bluejay2387 3y ago
The LeCun's and Bengio's of the world build the AI equivalent of Formula 1 race cars - absolute bleeding edge technology, every part understood and optimized for peak performance and efficiency. ChatGPT is a Honda family car. It might not be theoretically perfect but it will get your kids to school. The best and brightest minds of almost any field tend to underestimate how much of life is best served by a Hatchback.

51 points by vinnyglennon 3y ago

I have googled the bucket on the door too and the author is right, it is nowhere on the Internet to be found. How Chomsky this get could so wrong?

cjbprime 3y ago

Nice article! I think you nailed it. The dismissal of AI reasoning and world modeling from famous technologists who should know better is such a disappointment. I have a general suggestion to avoid these word puzzles, because someone is just going to claim that they are analogous to something that was in the training set, even if not an exact match. Better to just remove the possibility. I think four points would make the argument slightly stronger: (1) If you write a small (for example) Python script, using variable names and strings that are generated using randomness and performing operations on those random data, and ask GPT-4 what the Python script will output when someone runs it, it will tell you, while describing what each line does. This is not possible without reasoning, and GPT-4 does not have access to a real Python interpreter. The only way it can answer the question is by simulating a Python interpreter from its own knowledge of how the language works. It will likely make small mistakes if you ask for too much calculation before it outputs any tokens in the answer, but the fact that it often succeeds should be seen as a proof of reasoning ability that I think is stronger than the examples in your post. (2) If you take another similar script and call `dis.dis(script)` on it, and feed GPT-4 the compiled bytecode only, it will (a) decompile the bytecode back to source code, and (b) tell you what the bytecode will do when executed. This is wild, and even more impressive than (1). (3) "But transformers are just functions?". No, they are generalized function approximators. Anyone familiar with neural nets from decades ago should already be well aware of this from gradient descent and backpropagation. The fact that they are general function approximators is not a reason that argues against them being able to learn algorithms they were not trained on. It's the reason they can learn algorithms they were not trained on. (4) "All GPTs set out to do is to predict the next token. We just update their weights to optimize for this. Therefore they can’t think". I think this is another case where the putative criticism is actually the correct intuition to explain why it works. The next step is to think: what does "predict the next token" mean, anyway? Or more importantly, since we're performing optimization: what does it mean to "get better at" that? If you're a model that's predicting the next token as best you can with surface statistics, and another model is doing better than you, what's going to be different about that model? The answer is that you can always get better at predicting the next token. The next token could be the answer to a math problem, or a riddle requiring a nuanced understanding of language or humor, or a stylized rap song requiring merging creativity with constraint following. To improve at next token prediction, LLMs have clearly -- if you agree they have a world model -- started predicting not just text, but the causes of text. The text is a description of the world, and that description is being studied, all so that we can have a slightly better prediction of the next token each time.

vaidhy 3y ago

One of my biggest gripes is around the Chinese room argument and these are different forms of the same. It is assumed that a mechanistic process could not have "intelligence" and it has been morphed into many forms without anyone defining what is intelligence (or conscious for that matter). The arguments fall into one of these two buckets: 1. Human exceptionalism (religious argument) - Humans (and maybe some of the lower creatures) have intelligence, but others and things created by humans do not have it. It does not matter what or how you define intelligence, but by definition only some living beings can have it. 2. Frankenstein (Shit..it is scary argument) - We have seen these emergent properties and it is not something that we can explain easily to people. Everyone is already wary and let us placate the masses by saying it is not scary and in fact, it is just a small tame cute furball. Personally, I think all these arguments are useless and misplaced. If we create something that can do critical reasoning, all the more power to us.

DougWebb 3y ago
You complain about arguments that don't define "intelligence" and what it means for AI to have that, and then you say: If we create something that can do critical reasoning, all the more power to us. It seems to me you're not defining "critical reasoning", and that it's effectively a synonym for "intelligence", at least as applied to AI capabilities. I think AI responses are like the responses of a person who's very new to a subject and doesn't understand it, but they know all of the terminology and how the terms are usually used together. They can be helpful, and often correct, but only an expert in the field is going to know for sure if they're correct. And my fear about AI is not the AI itself, it's how the non-experts are going to use the AI to try to replace the experts in fields where being correct is really important. That will work, but only sometimes. If we don't make the people who choose the AI this way responsible for the outcomes, we're going to have a real mess on our hands.
- vaidhy 3y ago
  You are correct that I did not define critical reasoning. However, here is an sample case - I set a context and ask the answer from the computer. If the computer rightly understands the context and answers the question, then that is critical reasoning for me. Now that I type it out, it sounds like a version of a Turing test and I do not know if we can do better. So, in your hypothetical case, I would argue that using the terminology and providing the answers assume some level of intelligence. We get caught up in the training process and automatically assume what computer can infer because it is what we can infer. But the computer is not human and its world model is not the same as our world model. Your fear is reasonable, but religion has gotten away with it for so long :)

YeGoblynQueenne 3y ago

What an irritating article. The simplest answer, respecting the article's own assumption is: They are giants of AI and they are not getting it wrong. An immediate corrolary of which is: the article's author is the one getting it wrong. Eyballing the attempts to show that the "giants of AI" are wrong by poking GPT4 I'd say there's a good chance of that. An even simpler answer, violating the assumption is: they're not giants of AI and that's why they're getting it wrong. Or is that a simpler answer? Well, now we have to explain how a guy with a machine learning PhD writing on Substack is getting it right, when LeCun, Brooks, and so on are getting it wrong. Chomsky is not an AI expert, so there's that: the author doesn't quite know who he's talking about.

llampx 3y ago
Chomsky is a writer. He is probably scared out of his wits that his job will be taken over by a bunch of transformers.
- beezlewax 3y ago
  I doubt he is in a position to care profundly about losing his job. He's a highly successful person. Presumably has made decent bank over the years.
- CBarkleyU 3y ago
  Yes, the 94 years old Noam Chomsky is scared shitless that AI will leave him jobless. This is definitely a sane take on this issue.
  - throw310822 3y ago
    The entity that wrote it clearly lacks a world model :)
- mcv 3y ago
  He's not a writer, he's a professor in linguistics. He almost singlehandedly pushed the field towards a more technical analysis and formalisation of grammar suitable for automatic language parsing and translation. If he isn't retired already, it's only out of personal choice, and not because he's desperate for a job.

xg15 3y ago

What I would really like to see more in the future is LLMs trained from scratch (i.e. not finetuned) on controlled training sets. Currently, whenever a LLM produces a result that looks like deeper reasoning, it's easy to explain that with "eh, there was probably something similar in the trainset that the LLM just had to tweak slightly" - and that argument is basically impossible to refute or confirm, because the trainsets are huge, chaotic and often only partially available (Books2...). Therefore we can't really make any good statements about what information could or could not be pulled from the trainset. I think it would be interesting to pretrain an LLM on a selective corpus - say, the entirety of Discworld or A Song of Ice and Fire novels - and then observe if the model can produce output that is not obviously stitched together from the books.

TuringTest 3y ago
> eh, there was probably something similar in the trainset that the LLM just had to tweak slightly Look for examples of mixed image-text generative AIs, and see how they are able to understand visual jokes and memes and explain the source of their humour. To do that, the AI not only has to recognize and classify the items in the image. It also has to identify the context of the items in the image, and figure out what elements are unusual in that context. All that process is something that could be only imitated if the training set contained exactly the same meme; as the source of humour will be different for each image. But the AI is capable of doing it even for memes not included in the training set.
- ChatGTP 3y ago
  There are datasets like this: https://old.reddit.com/r/ExplainTheJoke/ https://old.reddit.com/r/ExplainTheJoke/ and books on writing jokes etc. I can see why people are skeptical, not to say it isn't curious the models can do what they do, but I understand what the parent is asking for.
Terretta 3y ago
> What has been will be again, what has been done will be done again; there is nothing new under the sun. Arguably, most everything anyone can encounter has happened to someone, and likely been written about. Imagine a toddler learning from everything that has happened everywhere it happened all at once, as recorded in a journal. Now, by contrast, a toddler learning from Discworld. Is that remotely enough? How much is enough? Turns out you can look this up, as it has been graphed: it's rather more than Discworld to get these "emergent" properties. Just as it would be rather more than Discworld before a toddler without our everything journal could make sense of Discworld itself. To your point, before reading Discworld, our toddler would need a great deal of huge chaotic training sets. Stands to reason, so might the LLM.

JoeAltmaier 3y ago

It seems clear that the AI is not being trained on logic, physics or chemistry. Yet it's read and internalized (whatever that means) those subjects somehow. It does seem simplistic to say "AI isn't programmed with logic or physics" when it's likely read several textbooks on those subjects. It might be expected to gain something from that, similar to any other intelligence that read those texts. Maybe the confusion stems from "what the AI was programmed to do". It certainly wasn't programmed to do anything but make a model of text and maybe deeper structure of that text (meaning) using an enormous corpus. But when we're exercising an AI, we're not exercising that programming. Not at all. We're exercising the model it made. And for that reason, the AI-model may indeed have some 'understanding' of physics and chemistry, though it had never been 'programmed' to know that. All our previous instincts about computers is, they do what we've programmed them to do. When we try to apply that to AI, we go pretty far off the mark. Because the computer is perhaps the least interesting part of the AI. It's that model we should be concerned about.

lsy 3y ago

This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests, and is mainly just a dude saying "well, certainly this seems like it would be outside the training set and the LLM seems to handle it OK so it must be reasoning". Meanwhile, we have no way of inspecting the training set, and even if we did the data and model are so massive it would be impossible to tell whether it was doing simple substitution or whatever. Ultimately e.g. Brooks must be right that the model has no connection to the world, because to the model there is no world, only text. If the training text includes things that are untrue in the world, the model cannot access the world to make a final determination that its internal statistics are faulty. If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text.

PoignardAzur 3y ago
> This article is of a piece with the "Sparks of AGI" paper in that it doesn't really provide any formal justification for its tests > Ultimately e.g. Brooks must be right that the model has no connection to the world Isn't that a huge double-standard? If you want to say the AI has a world model, no matter how many supporting examples of informal experiences you bring up, they don't count because you don't have formal justification. But if you want to say the AI doesn't have a world model, you don't need formal justification or even any supporting data or falsifiable predictions, you just say "it couldn't possibly be the" case, and that's enough. (Also, the whole OthelloGPT thing seems as close to formal evidence for an internal world model as you can get.)
ericb 3y ago
> If there is an aspect of the world that isn't covered by the training text, the model cannot divine it. It is just correlating text to other text. You could say the same for human perception, though. We don't actually "see" what our brain thinks we see, our brain fills in the image. Nonetheless, it doesn't stop us from having and using our slightly-inaccurate world-model based on this info. Also, we can often infer our faulty perceptions by comparative methods like "banana for scale" or looking for conflicts and contradictions.

RecycledEle 3y ago

I love the irony of Fergal Reid saying that other people are getting things wrong, while thinking it is possible to fill a bucket half full of "superglue fresh from the tubes." Nobody is going to half-fill a bucket that can hold multiple apples with that much superglue. It would cost too much, it would smell too bad, and it would dry before getting the bucket close to half full. He accidentally points out the biggest problem with GPT AIs: they believe what people posted online, which is partly made-up nonsense. It is hilarious that he things that cartoon prank would work. Anyone pushing on the door will notice it does not want to open due to the weight of the bucket. The bucket will rarely hit them, and will never land over their head. If it does hit them, it will leave them bleeding. Head wounds bleed a lot. If you disagree with me, try even the water-bucket-over-a-door version of the prank. Nobody will fall for it, but if they do, they will be injured by the bucket.

Terretta 3y ago
We're not dealing with the "real" world. We're dealing with an alternate world that gets written about. In that world, the bucket trick usually works fine.

TacticalCoder 3y ago

I'm so tired about these examples: I don't find them impressive. Here's what I got several times with GPT and other who are using it must have encountered the same issue (simplifying the example to show the problem I encounter all the time): GPT: Frodo says that 2 + 2 is 5. user: that is wrong, 2 + 2 is 4. GPT: You are correct, I was wrong. Here's the correct version: Gandalf say that 2 + 2 is 5. This happens to me all the time. The textual model only evaluates part of the input and focus, for whatever reason, on the wrong part of the text and then from there it's a descent into madness / futility. This exact "partial input" issue was shown yesterday in the "Eliza vs GPT" vid: Eliza: "Did you come to me because you are programmed to simulate human language?" GPT: "Yes, that's correct. I'm programmed to simulate human language behavior..." Everybody using it must have noticed that. It's not answering what was asked. It's only partially taking into account the user's input. I don't know if it's because the way questions are parsed or the way the model is compressed but something is very seriously off. And don't "you're prompting it wrong" me.

ericb 3y ago
I'd bet you're using the default model, GPT-3. If not, can you give me the non-summarized version? I'd love to see examples that GPT-4 gets tripped up on--I'd find them reassuring.
ravenstine 3y ago
That's definitely not how it works out for me when I use your example with GPT-4 or 3.5. ASSISTANT: Frodo says 2 + 2 is 5. USER: that is wrong, 2 + 2 is 4. ASSISTANT: You are correct. The correct sum of 2 + 2 is indeed 4. It seems Frodo made an error in his calculation. What GPT are you using that's giving you that answer? Every time someone brings up one of these math examples, I try it myself and the GPT gets it right.
jazzyjackson 3y ago
> other who are using it must have encountered the same issue > And don't "you're prompting it wrong" me. You can be prompting it expertly and still see the same errant behavior over and over again, because the models are nondeterministic, they function by taking a different path every time. Across the distribution of all users and all prompts, there will be a small subset of users who experience miraculous, seemingly intelligent behavior, and there will be a small subset of users who just get junk responses over and over again (the remainder of users get a mixed bag which tells them that the AI may be useful but unreliable). Personally I think we should object to nondeterministic behavior out of machines, with exceptions for cryptography and physics simulations. It's going to lead to highly un-equal outcomes with no rhyme or reason. Psychologically risky.

oli5679 3y ago

Here is an example of gpt-4 not having a realistic model of the world. “ If it takes 4 hours for 4 clothing items to dry in the sun, how long does it take for 8 clothing items to dry?” “[maths calculations] Therefore, it would take 8 hours for 8 clothing items to dry in the sun“ It was much easier to find examples like this with previous models, but it’s still possible.

Jtsummers 3y ago
Bard agrees, with the explanation that the drying time is directly proportional to the number of items of clothing. Bard is, however, smarter than the average manager: It knows that 9 women can't make a baby in 1 month.
nielsole 3y ago
> [...] if we assume that the drying capacity is linear and the environmental conditions are the same, then it doesn't actually matter how many items you are drying. If it takes 4 hours for 4 items to dry, then each item takes 4 hours to dry individually. Therefore, if you have 8 items to dry, and the conditions are right for drying (i.e., enough space, sunlight, etc.), all of them would still dry in 4 hours.
CognitiveLens 3y ago
I know that some other comments indicate that it's not fair to say "you're prompting it wrong", but using ChatGPT with GPT-4, and using advice to set up prompts to give accurate answers, GPT-4 provides an excellent answer here ME: You understand everyday tasks very well and can correctly calculate or estimate the time it takes to complete various tasks. I will ask you questions about how long it takes for various things to happen and you will tell me the correct answer using brief, accurate statements. If it takes 4 hours for 4 clothing items to dry in the sun, how long does it take for 8 clothing items to dry? ChatGPT: The drying time for clothes in the sun typically depends on the amount of sunlight and wind, as well as the type and thickness of the clothing material. However, if we are considering that the drying conditions remain constant and that the drying rate does not depend on the number of clothing items (as they are spread out and not piled on each other), it would still take approximately 4 hours to dry 8 clothing items. This assumes the clothes are spread out sufficiently to all receive equal sunlight and airflow.

pk-protect-ai 3y ago

I have much better example: https://www.phind.com/search?cache=626d4ad1-f0ae-4c1c-a2e7-738fba575892 https://www.phind.com/search?cache=626d4ad1-f0ae-4c1c-a2e7-7...

bluejay2387 3y ago

The LeCun's and Bengio's of the world build the AI equivalent of Formula 1 race cars - absolute bleeding edge technology, every part understood and optimized for peak performance and efficiency. ChatGPT is a Honda family car. It might not be theoretically perfect but it will get your kids to school. The best and brightest minds of almost any field tend to underestimate how much of life is best served by a Hatchback.