4 ms·

I played asci Tic-Tac-Toe with ChatGPT using new rules that I came up with, and it was able to play using the new rules and even explained it's reasoning for th

by Drakim 3y ago

I played asci Tic-Tac-Toe with ChatGPT using new rules that I came up with, and it was able to play using the new rules and even explained it's reasoning for the moves when I asked. I have a hard time understanding that as not including thinking and planning.

abrichr 3y ago
Sounds interesting! Can you please share the transcript?
- Drakim 3y ago
  I'm looking for it, but unfortunately I can't find it, which is weird because I've never deliberately deleted any of the transcripts. The homemade rules I used were as follows (each for a separate round): 1. The first player to get three in a row loses. 2. The first player to take the center square loses. 3. Players can place their marker in squares already taken. Admittedly it didn't nail it perfectly, a few times it messed up, but most of the time it was able to follow the rules as explained without issue, and could explain it's reasoning when I asked it why it picked that square.
intended 3y ago
This is the call out though - the difference between a proof of concept, and then seeing what happens in production. Production and scale definitively shows that LLMs don't reason. The output is highly unpredictable. Small changes can result in absolutely unrelated outcomes. If you were to classify text using chat gpt - chat gpt switches from classification to text generation if your text is longer than a certain length. LLM based Agents are the place where LLM reasoning died in practice. I really hope that there is a change, maybe larger context windows will allow the generated text to "reason". Although, that would not necessarily be reasoning. Even the article linked, "Indeed, LLMs make it easy to get problem-specific knowledge as long as we are willing to relax correctness requirements of that knowledge." LLMs generate text that approximates what a plan would "sound" like. It doesn't plan.
- deleted 3y ago
  [deleted]
- Drakim 3y ago
  When you poll humans for opinions, your wording can often have a very high impact on the opinions presented by folks.
distant_hat 3y ago
I suspect this appears to be the case because there is a ton of example pages around explaining tic-tac-toe games. My conjecture is that if you tried it with a game that wasn't so well studied as tic-tac-toe the LLM would fall flat on its face.

by Drakim 3y ago

abrichr 3y ago

Sounds interesting! Can you please share the transcript?

Drakim 3y ago
I'm looking for it, but unfortunately I can't find it, which is weird because I've never deliberately deleted any of the transcripts. The homemade rules I used were as follows (each for a separate round): 1. The first player to get three in a row loses. 2. The first player to take the center square loses. 3. Players can place their marker in squares already taken. Admittedly it didn't nail it perfectly, a few times it messed up, but most of the time it was able to follow the rules as explained without issue, and could explain it's reasoning when I asked it why it picked that square.

intended 3y ago

This is the call out though - the difference between a proof of concept, and then seeing what happens in production. Production and scale definitively shows that LLMs don't reason. The output is highly unpredictable. Small changes can result in absolutely unrelated outcomes. If you were to classify text using chat gpt - chat gpt switches from classification to text generation if your text is longer than a certain length. LLM based Agents are the place where LLM reasoning died in practice. I really hope that there is a change, maybe larger context windows will allow the generated text to "reason". Although, that would not necessarily be reasoning. Even the article linked, "Indeed, LLMs make it easy to get problem-specific knowledge as long as we are willing to relax correctness requirements of that knowledge." LLMs generate text that approximates what a plan would "sound" like. It doesn't plan.

deleted 3y ago
[deleted]
Drakim 3y ago
When you poll humans for opinions, your wording can often have a very high impact on the opinions presented by folks.

distant_hat 3y ago

I suspect this appears to be the case because there is a ton of example pages around explaining tic-tac-toe games. My conjecture is that if you tried it with a game that wasn't so well studied as tic-tac-toe the LLM would fall flat on its face.