3 ms·

We may be talking about different logic puzzles? The only model I've seen that didn't need some rather extreme adjustments to eventually solve it was Mistral la

by kromem 2y ago

We may be talking about different logic puzzles? The only model I've seen that didn't need some rather extreme adjustments to eventually solve it was Mistral large.

fragmede 2y ago
what's your puzzle? mines the river crossing puzzle
- kromem 2y ago
  Mine is also the river crossing puzzle. What's your variation?
  - fragmede 2y ago
    the boat is big enough to hold all the items I asked it > lion, goat, wolf riddle, but the boat is big enough to carry all of them and it said it could do it in one step. https://chat.openai.com/share/7b7a5462-7649-403d-a4f0-01c387e7eaf0 https://chat.openai.com/share/7b7a5462-7649-403d-a4f0-01c387... ChatGPT-4 today (April 28th, 2024) still fails at it sometimes: https://chat.openai.com/share/1bec923d-d727-42fe-ba9b-9f92b9780bcb https://chat.openai.com/share/1bec923d-d727-42fe-ba9b-9f92b9... This is ChatGPT-4 getting it wrong, months ago: https://chat.openai.com/share/caa37ad6-b7a8-451d-8f39-8a2c04249f4c https://chat.openai.com/share/caa37ad6-b7a8-451d-8f39-8a2c04... This is ChatGPT-4 getting it right, today (April 28th, 2024): https://chat.openai.com/share/d2d9e63e-819e-4681-9f9f-8f77ea12f282 https://chat.openai.com/share/d2d9e63e-819e-4681-9f9f-8f77ea...
    - kromem 2y ago
      Ah, ok. My variation is it's a vegetarian wolf, a carnivorous goat, and a cabbage. There's a few different hacks that will get it to work, but one of the more interesting is switching the nouns to emojis. But almost none of the models ever get it on the first try, and every major model since GPT-4 can have the prompt tweaked to get it with the exception of Llama-3, which I just can't get to solve it with anything I've tried so far (and I'm not sure if it's because of extra strong associations to the standard form from the extra training run or if it lacks the core competencies, though I am starting to think it's the latter given how it responds as I point out errors). I particularly like this variation because it requires remapping concepts in unintuitive ways based on broad abstractions, like having a goat potentially eat a wolf because of it being carnivorous.
      - fragmede 2y ago
        Oh that's a good one. Interesting that it's unable to get that right, because when I've tried asking it using a different theme (Star Trek on a shuttlecraft instead of a river, or an LOTR theme), and it's able to recognize that it's still the river crossing problem. It's also inconsistent at solving the twist. Here's ChatGPT-4 getting it right and wrong within minutes of each other just today (April 29th, 2024). right: https://chat.openai.com/share/178adc81-e43b-4ecf-b7b7-d7fa5ab6f0e1 https://chat.openai.com/share/178adc81-e43b-4ecf-b7b7-d7fa5a... wrong: https://chat.openai.com/share/9ae78081-c0eb-409d-93c6-650f3754f8e9 https://chat.openai.com/share/9ae78081-c0eb-409d-93c6-650f37... Have you tried

by kromem 2y ago

We may be talking about different logic puzzles? The only model I've seen that didn't need some rather extreme adjustments to eventually solve it was Mistral large.

fragmede 2y ago

what's your puzzle? mines the river crossing puzzle

kromem 2y ago
Mine is also the river crossing puzzle. What's your variation?
- fragmede 2y ago
  the boat is big enough to hold all the items I asked it > lion, goat, wolf riddle, but the boat is big enough to carry all of them and it said it could do it in one step. https://chat.openai.com/share/7b7a5462-7649-403d-a4f0-01c387e7eaf0 https://chat.openai.com/share/7b7a5462-7649-403d-a4f0-01c387... ChatGPT-4 today (April 28th, 2024) still fails at it sometimes: https://chat.openai.com/share/1bec923d-d727-42fe-ba9b-9f92b9780bcb https://chat.openai.com/share/1bec923d-d727-42fe-ba9b-9f92b9... This is ChatGPT-4 getting it wrong, months ago: https://chat.openai.com/share/caa37ad6-b7a8-451d-8f39-8a2c04249f4c https://chat.openai.com/share/caa37ad6-b7a8-451d-8f39-8a2c04... This is ChatGPT-4 getting it right, today (April 28th, 2024): https://chat.openai.com/share/d2d9e63e-819e-4681-9f9f-8f77ea12f282 https://chat.openai.com/share/d2d9e63e-819e-4681-9f9f-8f77ea...
  - kromem 2y ago
    Ah, ok. My variation is it's a vegetarian wolf, a carnivorous goat, and a cabbage. There's a few different hacks that will get it to work, but one of the more interesting is switching the nouns to emojis. But almost none of the models ever get it on the first try, and every major model since GPT-4 can have the prompt tweaked to get it with the exception of Llama-3, which I just can't get to solve it with anything I've tried so far (and I'm not sure if it's because of extra strong associations to the standard form from the extra training run or if it lacks the core competencies, though I am starting to think it's the latter given how it responds as I point out errors). I particularly like this variation because it requires remapping concepts in unintuitive ways based on broad abstractions, like having a goat potentially eat a wolf because of it being carnivorous.
    - fragmede 2y ago
      Oh that's a good one. Interesting that it's unable to get that right, because when I've tried asking it using a different theme (Star Trek on a shuttlecraft instead of a river, or an LOTR theme), and it's able to recognize that it's still the river crossing problem. It's also inconsistent at solving the twist. Here's ChatGPT-4 getting it right and wrong within minutes of each other just today (April 29th, 2024). right: https://chat.openai.com/share/178adc81-e43b-4ecf-b7b7-d7fa5ab6f0e1 https://chat.openai.com/share/178adc81-e43b-4ecf-b7b7-d7fa5a... wrong: https://chat.openai.com/share/9ae78081-c0eb-409d-93c6-650f3754f8e9 https://chat.openai.com/share/9ae78081-c0eb-409d-93c6-650f37... Have you tried