4 ms·

Wasn't there a humorous law which said "Any headline that ends in a question mark can be answered by the word no" and indeed, what makes you think it works? It

by chx 4y ago

Wasn't there a humorous law which said "Any headline that ends in a question mark can be answered by the word no" and indeed, what makes you think it works? It fools people into thinking it works but have you seen that chess match between Stockfish and ChatGPT? It's the best, most succinct demonstration of what ChatGPT is. https://pastebin.com/X6kBRTa9 https://pastebin.com/X6kBRTa9 https://i.redd.it/fl4puwzxt9ha1.gif https://i.redd.it/fl4puwzxt9ha1.gif https://youtu.be/rSCNW1OCk_M https://youtu.be/rSCNW1OCk_M

jodrellblank 4y ago
Betteridge's law of headlines - https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
the_af 4y ago
In my experience, ChatGPT is unsuitable even for tic tac toe. When I tried it, it hallucinated completely incorrect board positions (e.g. forgot about moves already made or wanted to continue playing after I had won).
- williamcotton 4y ago
  You (or running software you interface with ChatGPT) needs to maintain the state of the game between moves. LLMs aren’t computers but they can interact with computers. LLM(InitialInstructions)->Computer(CodeWrittenByLLM)->LLM(InstructionsOutputByComputer)->LoopUntilWin
  - janalsncm 4y ago
    In my experiments that doesn’t help. ChatGPT failed to play any valid move given a position, let alone a good one. The point is that language is probably too “linear” to represent what’s going on during a game. Pieces on the board have complex relationships (consider a pinned piece, for example) so autoregressive decoding is simply not enough.
    - williamcotton 4y ago
      Try something like: The current board state is: board = [['', '', ''], ['', '', ''], ['', '', '']]; write a javascript function called bestMove(board) that predicts the best tic-tac-toe move to make given a board. use that function to update the board state and return the board state in JSON form. The response will have a bunch of functions like function bestMove(board) { function getEmptySpaces(board) { function predictBestMove(board, player) { function minimax(board, isMaximizing) { function checkWinner(board) { ... These functions should work together to determine the best move to make in a game of Tic-Tac-Toe, using the minimax algorithm to evaluate each possible move and choosing the one with the highest score. Then eval and execute the bestMove function, passing in the initial board state, returning the updated board state. Then the human player makes a move. Then another prompt: Try something like: The current board state is: board = [['X', '', ''], ['', 'O', ''], ['', '', '']]; assume there is a function called bestMove(board) and checkWinner(board) that predicts the best tic-tac-toe move to make given a board. use those functions to update the board state and check the winner and return the board state and current winner in JSON form. etc... Using my little engine, I get this solution: question: "Answering as [rowInt, colInt], writing custom predictBestMove, getEmptySpaces, minimax and checkWinner functions implemented in the thunk, what is the best tic-tac-toe move for player X on this board: [['X', '_', 'X'], ['_', '_', '_'], ['_', '_', '_']]?", answer: [ 0, 1 ], https://gist.github.com/williamcotton/e6bdcca0a96a6e7bf5d2fe422d0611d1 https://gist.github.com/williamcotton/e6bdcca0a96a6e7bf5d2fe...
      - the_af 4y ago
        Interesting. Have you tried playing a full game like this, instead of a single move? In any case, I don't think this is what people expect out of ChatGPT. Your approach is too "programmer centric". I think people expect telling ChatGPT the rules of the game, in almost plain language, and then expect to be able to play a game of Tic Tac Toe interacting with it like one would with a person. This means, not asking it to write functions or remind it of the state of the board at every step. This doesn't work consistently for a well-known game like Tic Tac Toe, much less for an arbitrary game you make up.
  - the_af 4y ago
    That's interesting! But it does show that the hype around ChatGPT is misplaced: as impressive as it is (and I do find it impressive) it doesn't really build a "model" of the conversation; you have to help it all the way or it will go astray, since nonsensical board game moves make as much sense from a conversational point of view. It's also easy to make it hallucinate nonfactual information, which makes it bad at exploring things you're unsure about and about which you could inadvertently write misleading questions (Note this isn't the same as asking misleading questions to a human who will answer confidently out of arrogance; ChatGPT has no "ego" but it will write completely false/nonfactual answers if asked to by mistake. I have examples of this). It's easy to get confused about GPT's limitations because it's a pretty successful parrot, and it writes convincing conversations in a vast number of cases.
- int_19h 4y ago
  I did that experiment at one point, as well. So long as I repeated the board state as an ASCII diagram with ``` fence, it would respect it and make legitimate moves... but they were very dumb moves for sure. And yes, it didn't notice when the game was over.
bobsmooth 4y ago
That only applies if the headline is a yes or no question.

by chx 4y ago

jodrellblank 4y ago

Betteridge's law of headlines - https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

the_af 4y ago

In my experience, ChatGPT is unsuitable even for tic tac toe. When I tried it, it hallucinated completely incorrect board positions (e.g. forgot about moves already made or wanted to continue playing after I had won).

williamcotton 4y ago
You (or running software you interface with ChatGPT) needs to maintain the state of the game between moves. LLMs aren’t computers but they can interact with computers. LLM(InitialInstructions)->Computer(CodeWrittenByLLM)->LLM(InstructionsOutputByComputer)->LoopUntilWin
- janalsncm 4y ago
  In my experiments that doesn’t help. ChatGPT failed to play any valid move given a position, let alone a good one. The point is that language is probably too “linear” to represent what’s going on during a game. Pieces on the board have complex relationships (consider a pinned piece, for example) so autoregressive decoding is simply not enough.
  - williamcotton 4y ago
    Try something like: The current board state is: board = [['', '', ''], ['', '', ''], ['', '', '']]; write a javascript function called bestMove(board) that predicts the best tic-tac-toe move to make given a board. use that function to update the board state and return the board state in JSON form. The response will have a bunch of functions like function bestMove(board) { function getEmptySpaces(board) { function predictBestMove(board, player) { function minimax(board, isMaximizing) { function checkWinner(board) { ... These functions should work together to determine the best move to make in a game of Tic-Tac-Toe, using the minimax algorithm to evaluate each possible move and choosing the one with the highest score. Then eval and execute the bestMove function, passing in the initial board state, returning the updated board state. Then the human player makes a move. Then another prompt: Try something like: The current board state is: board = [['X', '', ''], ['', 'O', ''], ['', '', '']]; assume there is a function called bestMove(board) and checkWinner(board) that predicts the best tic-tac-toe move to make given a board. use those functions to update the board state and check the winner and return the board state and current winner in JSON form. etc... Using my little engine, I get this solution: question: "Answering as [rowInt, colInt], writing custom predictBestMove, getEmptySpaces, minimax and checkWinner functions implemented in the thunk, what is the best tic-tac-toe move for player X on this board: [['X', '_', 'X'], ['_', '_', '_'], ['_', '_', '_']]?", answer: [ 0, 1 ], https://gist.github.com/williamcotton/e6bdcca0a96a6e7bf5d2fe422d0611d1 https://gist.github.com/williamcotton/e6bdcca0a96a6e7bf5d2fe...
    - the_af 4y ago
      Interesting. Have you tried playing a full game like this, instead of a single move? In any case, I don't think this is what people expect out of ChatGPT. Your approach is too "programmer centric". I think people expect telling ChatGPT the rules of the game, in almost plain language, and then expect to be able to play a game of Tic Tac Toe interacting with it like one would with a person. This means, not asking it to write functions or remind it of the state of the board at every step. This doesn't work consistently for a well-known game like Tic Tac Toe, much less for an arbitrary game you make up.
- the_af 4y ago
  That's interesting! But it does show that the hype around ChatGPT is misplaced: as impressive as it is (and I do find it impressive) it doesn't really build a "model" of the conversation; you have to help it all the way or it will go astray, since nonsensical board game moves make as much sense from a conversational point of view. It's also easy to make it hallucinate nonfactual information, which makes it bad at exploring things you're unsure about and about which you could inadvertently write misleading questions (Note this isn't the same as asking misleading questions to a human who will answer confidently out of arrogance; ChatGPT has no "ego" but it will write completely false/nonfactual answers if asked to by mistake. I have examples of this). It's easy to get confused about GPT's limitations because it's a pretty successful parrot, and it writes convincing conversations in a vast number of cases.
int_19h 4y ago
I did that experiment at one point, as well. So long as I repeated the board state as an ASCII diagram with ``` fence, it would respect it and make legitimate moves... but they were very dumb moves for sure. And yes, it didn't notice when the game was over.

bobsmooth 4y ago

That only applies if the headline is a yes or no question.