3 ms·

" An LLM has to map the idea of left and right into symbols via text and line breaks." Oh yeah that's i suggested it :) I do wonder though if we give the LLMs

" An LLM has to map the idea of left and right into symbols via text and line breaks." Oh yeah that's i suggested it :) I do wonder though if we give the LLMs enough examples of texts with people describing their relative spatial position to each other and things will it eventually "learn" to work things these out a bit better

famouswaffles 3y ago
>I do wonder though if we give the LLMs enough examples of texts with people describing their relative spatial position to each other and things will it eventually "learn" to work things these out a bit better GPT-4's spatial position understanding is actually really good all things considered. By the end, 4 was able to construct an accurate maze just from feedback about the current position and possible next moves after each move by GPT-4. https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-into-react-agent-and-llm-s-thoughts-b1823fb266ee https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-... I think we just don't write much about moving through space and that is why reasoning about it is more limited.
- astrange 3y ago
  A funny thing GPT-4 is unusually good at is giving driving directions. This shouldn't work, and of course isn't 100% right, but… it's kind of right. Bard can answer questions like this, but I think it actually uses the Maps API. (It certainly says that's what it's doing.) On the other hand, every chatbot including GPT-4 is both unable to do ASCII art and unable to tell it can't do it. (Bard always shows you `cowsay` and tells you it's what you asked for, no matter what it was supposed to be.)
  - bt1a 3y ago
    I tried so hard to make ascii art with GPT-4 api :(
walnutclosefarm 3y ago
Probably. But what seems much more interesting is to have a spatial model pre-seeded in the LLM, so that it "attaches" language to that as part of its training. Ditto for other models of the world we want the language module to be able to draw on and reason with.

famouswaffles 3y ago

>I do wonder though if we give the LLMs enough examples of texts with people describing their relative spatial position to each other and things will it eventually "learn" to work things these out a bit better GPT-4's spatial position understanding is actually really good all things considered. By the end, 4 was able to construct an accurate maze just from feedback about the current position and possible next moves after each move by GPT-4. https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-into-react-agent-and-llm-s-thoughts-b1823fb266ee https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-... I think we just don't write much about moving through space and that is why reasoning about it is more limited.

astrange 3y ago
A funny thing GPT-4 is unusually good at is giving driving directions. This shouldn't work, and of course isn't 100% right, but… it's kind of right. Bard can answer questions like this, but I think it actually uses the Maps API. (It certainly says that's what it's doing.) On the other hand, every chatbot including GPT-4 is both unable to do ASCII art and unable to tell it can't do it. (Bard always shows you `cowsay` and tells you it's what you asked for, no matter what it was supposed to be.)
- bt1a 3y ago
  I tried so hard to make ascii art with GPT-4 api :(

walnutclosefarm 3y ago

Probably. But what seems much more interesting is to have a spatial model pre-seeded in the LLM, so that it "attaches" language to that as part of its training. Ditto for other models of the world we want the language module to be able to draw on and reason with.