3 ms·

The one thing that has convinced me that chatGPT has built a real world model is asking it how to stack different objects. You can ask it for the safest way to

The one thing that has convinced me that chatGPT has built a real world model is asking it how to stack different objects. You can ask it for the safest way to stack a pillow, an egg, a nail, a book and an action figure. Even get more complicated. GPT-4 will, most of the time, correctly reason through how to stack these items to form a stable structure without breaking. That tells me it knows intimately the physical properties of all of these objects and how they interact with each other.

avereveard 3y ago
Or was that one of the training samples provided to gpt 3.5? It is going to be increasingly harder to divide between what it was in the training set and what is creative thinking going forward. Like, at the very minimum, all the publicly known reasoning samples that were used to probe gpt version 3.5 downward are going to be unreliable for knowing what gpt4 reasoning abilities are
- tornato7 3y ago
  The point of the question is that you can easily come up with combinations of objects that are unlikely, or statistically impossible, to be in their training samples. How about this one? "Tell me how to safely stack the following objects on top of each other without any additional support: A Nikon DSLR, A small succulent, 4 lightbulbs, a pack of gummy worms, and a hardcover copy of Harry Potter and the Halfblood Prince." Hardcover copy of Harry Potter and the Half-Blood Prince: Start with the largest, heaviest, and most stable object. This will serve as your base. Make sure to place it on a flat, secure surface. Pack of gummy worms: This item is fairly light and flexible. It should sit nicely on the book without risking damage to either. You can place it right in the center to maintain balance. Nikon DSLR: This is a heavy object with a fairly stable shape. Put this on top of the pack of gummy worms, as it's sturdy enough to handle the weight and won't be harmed by the soft package. Make sure to place it so that the lens is pointing upward and it sits flat and stable. Don't use any protruding lenses or parts to balance it, as they could easily be damaged. Four Lightbulbs: Since light bulbs are round and fragile, you will need to stack them with care. If the bulbs are in a box, you can place the box on top of the DSLR, making sure it's centered and stable. If the bulbs are loose, it's a bit more tricky. You might want to nestle them into the lens of the DSLR (if the lens is large enough), which should provide some natural containment for them. If not, it might be safer not to include the lightbulbs in your stack. A small succulent: Finally, the succulent can be placed on top. It's likely the lightest object and it is usually quite stable due to its pot. If the lightbulbs were in a box, place the succulent on top of that. If you ended up not including the lightbulbs, place the succulent directly on top of the DSLR.
  - deleted 3y ago
    [deleted]
  - RandomLensman 3y ago
    Reads well if you don't think about it too much... For example: Where does the succulent go if the light bulbs are nestled into the lens of the DLSR? Balanced on the light bulbs? Why would the gummy worm package need to be in the center of the book to maintain balance?
    - tornato7 3y ago
      https://i.imgur.com/feEiiZA.png https://i.imgur.com/feEiiZA.png I tried to stack all of these objects myself and couldn't really. I think GPT-4's approach is actually really good. It correctly points out that the gummy worms make a flexible base for the DSLR (otherwise the protruding buttons/viewfinder make it wobbly on the hard book), and the light bulbs are able to nestle into the front of the lens. If they were smaller light bulbs I could probably use the four of them as a small base on top of the lens to host the succulent.
      - RandomLensman 3y ago
        Might also put the light bulbs as a base (especially if in a box). They are pretty sturdy and can hold a book.
        tornato7 3y ago
        The point is that ChatGPT undeniably built a world model good enough to understand the physical and three-dimensional properties of these items pretty well, and it gives me a somewhat workable way to stack them, despite never having seen that in its training data.
        RandomLensman 3y ago
        You cannot conclude that from the output - the training data will likely contain a lot stacking things. Everyday objects also might have some stacking properties that make these questions easy to answer even with semi-random answers. Plus, some stuff clearly makes no sense or is ignored (like the gummy worms in the center, forgetting about the succulent in some cases). If you want to test world modeling, give it objects it will have never encountered, describe them and then ask to stack etc. For example, a bunch of 7 dimensional objects that can only be stacked a certain way.

avereveard 3y ago

Or was that one of the training samples provided to gpt 3.5? It is going to be increasingly harder to divide between what it was in the training set and what is creative thinking going forward. Like, at the very minimum, all the publicly known reasoning samples that were used to probe gpt version 3.5 downward are going to be unreliable for knowing what gpt4 reasoning abilities are

tornato7 3y ago
The point of the question is that you can easily come up with combinations of objects that are unlikely, or statistically impossible, to be in their training samples. How about this one? "Tell me how to safely stack the following objects on top of each other without any additional support: A Nikon DSLR, A small succulent, 4 lightbulbs, a pack of gummy worms, and a hardcover copy of Harry Potter and the Halfblood Prince." Hardcover copy of Harry Potter and the Half-Blood Prince: Start with the largest, heaviest, and most stable object. This will serve as your base. Make sure to place it on a flat, secure surface. Pack of gummy worms: This item is fairly light and flexible. It should sit nicely on the book without risking damage to either. You can place it right in the center to maintain balance. Nikon DSLR: This is a heavy object with a fairly stable shape. Put this on top of the pack of gummy worms, as it's sturdy enough to handle the weight and won't be harmed by the soft package. Make sure to place it so that the lens is pointing upward and it sits flat and stable. Don't use any protruding lenses or parts to balance it, as they could easily be damaged. Four Lightbulbs: Since light bulbs are round and fragile, you will need to stack them with care. If the bulbs are in a box, you can place the box on top of the DSLR, making sure it's centered and stable. If the bulbs are loose, it's a bit more tricky. You might want to nestle them into the lens of the DSLR (if the lens is large enough), which should provide some natural containment for them. If not, it might be safer not to include the lightbulbs in your stack. A small succulent: Finally, the succulent can be placed on top. It's likely the lightest object and it is usually quite stable due to its pot. If the lightbulbs were in a box, place the succulent on top of that. If you ended up not including the lightbulbs, place the succulent directly on top of the DSLR.
- deleted 3y ago
  [deleted]
- RandomLensman 3y ago
  Reads well if you don't think about it too much... For example: Where does the succulent go if the light bulbs are nestled into the lens of the DLSR? Balanced on the light bulbs? Why would the gummy worm package need to be in the center of the book to maintain balance?
  - tornato7 3y ago
    https://i.imgur.com/feEiiZA.png https://i.imgur.com/feEiiZA.png I tried to stack all of these objects myself and couldn't really. I think GPT-4's approach is actually really good. It correctly points out that the gummy worms make a flexible base for the DSLR (otherwise the protruding buttons/viewfinder make it wobbly on the hard book), and the light bulbs are able to nestle into the front of the lens. If they were smaller light bulbs I could probably use the four of them as a small base on top of the lens to host the succulent.
    - RandomLensman 3y ago
      Might also put the light bulbs as a base (especially if in a box). They are pretty sturdy and can hold a book.
      - tornato7 3y ago
        The point is that ChatGPT undeniably built a world model good enough to understand the physical and three-dimensional properties of these items pretty well, and it gives me a somewhat workable way to stack them, despite never having seen that in its training data.
        RandomLensman 3y ago
        You cannot conclude that from the output - the training data will likely contain a lot stacking things. Everyday objects also might have some stacking properties that make these questions easy to answer even with semi-random answers. Plus, some stuff clearly makes no sense or is ignored (like the gummy worms in the center, forgetting about the succulent in some cases). If you want to test world modeling, give it objects it will have never encountered, describe them and then ask to stack etc. For example, a bunch of 7 dimensional objects that can only be stacked a certain way.