4 ms·

> has a solid understanding of space How so? Is it simply better at predicting the answer to spatial questions based on being a more powerful autocomplete than

• famouswaffles 4y ago
  In science results matter much more than vague and ill defined assertions. A system reasons and understands when it demonstrates understanding and reasoning. That's how you asses reasoning in anything/anybody. Evaluation. If you want to tell me there's a special distinction between what an LLMs outputs and "true reasoning TM" then cool but when you can't show me what that distinction is, how to test for it, the qualitative or quantitative differences then I'm going to throw your argument away because it's not a valid one. It's just an arbitrary line drawn on sand. A distinction that can't be tested for is not a distinction.
  • adroniser 4y ago
    The point is as far as I know it hasn't been properly tested for. No one has come up with a comprehensive framework of tests to evaluate spacial reasoning using entirely sentences that aren't isomorphic to training data.
  • bombolo 4y ago
    Try asking chatgpt to solve a non-well-known programming exercise. It will fail miserably and insist on rehashing the same wrong solution when told that it was wrong. A real person might give up, but would not claim that an obviously wrong answer was in fact the answer.
• rcpt 4y ago
  In the Microsoft sparks paper they ask it to "draw a unicorn in TikZ". It does well. Then they remove the horn and ask it to put the horn on the unicorn and it gets that to.