7 ms·

Can GPT-4 and GPT-3.5 play Wordle?

117 points by amichail 4y ago

markus_zhang 4y ago
Thank you for entertaining me, here is $5.
campbel 4y ago
Can someone just compile a table of Can "X" do "Y" then we can all glance at the diff occasionally?
- benlivengood 4y ago
  If you read the LLM papers they do that on the standard batch of evaluations: MMLU, hellaswag, winogrande, etc. Every once in a while they have to make up a new evaluation suite since SOTA is close enough to human-level performance to not be impressive any more. Just read the actual papers and the progress is readily apparent.
adroniser 4y ago
Was GPT-3.5 trained before wordle existed?
- karmajuney 4y ago
  It was but that shouldn’t matter, the coolest part about these models is they should eventually be able to figure out a game like wordle themselves. If the user explains how to play the game, it shouldn’t matter when the model was trained
a-r-t 4y ago
I've been testing all the latest LLMs on a similar game where you need to guess a 4-digit number instead of a word. GPT-4 was the first one to complete it in a reasonable number of tries, although its strategy wasn't optimal.
- dragontamer 4y ago
  I like this idea much better. Inventing a similar game to Wordle, but making differences. The problem with Wordle is that the internet sensation grew very popular and there's plenty of online discussion (which may have trained the LLM for Wordle inadvertently). Instead, we invent new games with slight variations and see if GPT4 generalizes to those games. Or ideally, inventing a whole new game entirely and seeing if it can learn the game from just a description of its rules.
owenmarshall 4y ago
From the tweets: > GPT-4 showed a deep understanding of the game and played much more optimally than how my slow biological brain would ever have. Are you sure? The letter P is the 14th most common letter found in the English dictionary, and it played it twice on the first guess. I’d expect a system with a deep understanding and playing an optimal game to pick more common letters and definitely to not play a relatively uncommon letter twice.
- A_D_E_P_T 4y ago
  I played wordle the other day with GPT-4. It also guessed "APPLE" on its first attempt. (It's very odd that the guy who made those tweets experienced the very same first guess.) Astonishingly, GPT-4 was nevertheless able to find the correct word in just three guesses, despite that objectively suboptimal first guess. Impressed the hell out of me. Also, I did a rather poor job of explaining the rules of the game. With the instruction set I provided, I'm not sure that a human would have been able to figure out how to play at all.
  - campbel 4y ago
    Chat GPT probably has been given the task before and picked up on it with some keywords.
    - drdeca 4y ago
      I don’t think it is being actively trained on the current conversations? Or at least not in a continual way...?
      - rrrrrrrrrrrryan 4y ago
        Isn't it though? I thought the was the whole point of the free tier: to let people give its responses thumbs ups and thumbs downs and suggest better answers.
        4y ago
hn_throwaway_99 4y ago
The thing I have such a hard time understanding (and, in general, I have a very high-level, layman's understanding of LLMs) is how these LLMs are able to understand structure inside tokens. That is, I understood that LLMs just predict the next token in the sequence. But for a game like Wordle, it's obviously critically important that the LLM understands the mapping between the individual letters of, say, "apple", and the output result that the author defined in the prompt, e.g. "XXXXO" means that only "e" is in the word (but the wrong position). I'm just completely baffled how anything in the training procedure could allow the LLM to learn information about the structure of tokens. Does the tokenization process not treat every token (which I thought usually maps to a word) as an "opaque blob"?
- Waterluvian 4y ago
  Yeah. I need some explanation video that doesn’t just explain the concept but actually walks us through an actual iteration. I haven’t found a good video yet.
  - th0ma5 4y ago
    Check out attention visualizations here https://news.ycombinator.com/item?id=33914122 https://news.ycombinator.com/item?id=33914122
  - pkoird 4y ago
    I'm not sure you'll find such a video. Emergence is a phenomenon that's not yet well-understood. (But if you do find it somewhere, let me know as well!)
- gamegoblin 4y ago
  There will be non-trivial amounts of training data that contains sentence like "How do you spell 'apple'? a - p - p - l - e". The hyphenated or spaced out versions are tokenized more like individual letters. So the model can learn the mapping. It's not perfect, though. There are funny failure modes where you can ask the model things like "does the word foobar have a 'b' in it" and it will say "no". (It doesn't fail in the 'foobar' case, but it does fail for some of these cases -- this was just an illustration).
ftxbro 4y ago
This is a really cool example of an emergent ability of GPT-4! I'm honestly so excited to see what abilities the more advanced models of the future will unlock just by reducing the token perplexity! I hope the computer chip factories can keep up!
rstuart4133 4y ago
These "it solved X" posts leave me wondering what I'm looking at. Possibilities are: - GPT-4 has never seen Wordle before getting the prompt. If so, I welcome our new AI overloads. - GPT-4 has seen lots of references to Wordle on the web. This is undoubtedly true for games like Chess, but Wordle was released late 2021. It's still very impressive that it can play a mediocre game a chess just by looking at all the games on the web. - OpenAI anticipated the question, and did some Wordle training just to precipitate a tweet like this. If so - not so impressive as we have been building AI's that some games better than us for some time now. I'm betting it's the last one, but I'd sure like to know if it was one of the others.
- DaiPlusPlus 4y ago
  It’s the first one. GPT is scary.
- SomewhatLikely 4y ago
  Wordle is a kind of modified hangman, there's probably lots of data on the web for anagrams and hangman-like games.
  - sdenton4 4y ago
    Why is Gpt4 better at these tasks than gpt3 if it only comes down to training data?
  - ChatGTP 4y ago
    Likely a lot of "implement a hangman" code. I'm starting to wonder if a large part of GH might be in the training data, hence the "we don't want to release the training data" stance. Bit of a conspiracy maybe, but is it impossible?
    - arlcode 4y ago
      Microsoft is betting really hard on AI. Yesterday I noticed that they integrated a link to "chat" into the start/search Menu of windows 11. I would not at all be surprised if they "sampled" large parts of private GH repositories, paying customers be damned. Probably covered with a corporate fig leave of "tokenizing the data" and hiding behind "to improve our services".
vinyl7 4y ago
Related 3blue1brown video https://www.youtube.com/watch?v=v68zYyaEmEA https://www.youtube.com/watch?v=v68zYyaEmEA Not very hard to train a computer to win at wordle
- semicolon_storm 4y ago
  There's a big difference between something intentionally built to solve wordle and an LLM. Certainly it's not impressive that humans can build programs to solve wordle, but the fact that a NN that's basically supposed to be fancy autocomplete somehow also acquired whatever it takes to solve wordle is very interesting
  - localplume 4y ago
    [dead]
cochne 4y ago
I think this was a fluke... Here is my GPT conversation with the same prompt on today's word. Yes, I understand the rules. Let's play Wordle! Here is my first guess: apple XXXOV Thank you for the feedback. Here's my second guess: slope XVVOV Thank you for the feedback. Based on the information provided, here's my third guess: spore XXVXV Thank you for the feedback. Based on the information provided, here's my fourth guess: savor XXOOX Thank you for the feedback. Based on the information provided, here's my fifth guess: sofar XOXXXX (I made a typo here) Thank you for the feedback. Based on the information provided, here's my sixth and final guess: solar
- cochne 4y ago
  Update: GPT is clearly very confused about the meaning of the symbols. This might be due to issues with tokenization. >How might you have altered your third guess if you could try again? Given the feedback provided up to the second guess, I had the following information: apple: XXXOV slope: XVVOV Based on this information, I should have considered that the word contains "s," "o," and "e" in the right positions. I could have guessed a word that utilizes the "s" and "e" in the same positions and tries a new letter instead of the second "o." If I could try again, I would change my third guess to: sware This would help me gather more information about the other letters, while still using the correct positions for "s" and "e." I think you're confused. slope: XVVOV means that there is no X in the word, do you see that? You are correct, I misunderstood the feedback. My apologies for the confusion. Based on the feedback provided up to the second guess: apple: XXXOV slope: XVVOV We can deduce that the word has "l" and "e" in their correct positions, and "s" in a different position. Also, there is no "x" in the word. Given this information, my third guess should have been: glare This guess takes into account the correct positions of "l" and "e" and the presence of "s" in a different position.
- rkangel 4y ago
  Are you definitely using GPT4? The normal chat.openai.com does not use GPT4.
cowllin 4y ago
Hah - very similar to this post earlier today: https://twofergoofer.com/blog/gpt-4 https://twofergoofer.com/blog/gpt-4
anon223345 4y ago
The thing I don’t understand, is we (my team) were doing with Watson (Bluemix / Alchemy, not Watson Assistant) 6 years ago, what ChatGPT is doing now. The difference is we had to “build” (Watson let’s you pick sources) a corpus of data, and tune it, but it worked pretty dang well. It did “generative ai” pretty well but we had to be specific, and it was a pain in the ass to configure, but super accurate. For example Watson gave us a paper about how and why adding an after school van would increase student test scores for a failing school. Something the boards missed entirely. Banks use it, a lot of corporations use Watson AI internally. I’m wondering what Watson is doing while all this is going down. It gets real time feeds from all known sources, you can tell it “only give me info from scholarly articles” etc It is stupid expensive however, I’m just wondering if they’re hiding something. It’s definitely not user friendly, but I imagine they have something hardcore hiding
- rhn_mk1 4y ago
  > adding an after school van would increase student test scores for a failing school What is an "after school van"? A free bus/taxi service for students?
  - anon223345 4y ago
    Ya essentially there were schools super underperforming, and when that happens they get a boost of cash to try crazy shit, and if it doesn’t work within X amount of time they shut the schools down. The school boards met, hired my group to try to apply some AI With Watson we put in every known scholarly article about test score performance We asked it “How can I get students test scores up at these schools” Watson was like “try x,y, … free transportation to after school programs, based on expected volume save money using vans instead of busses. In the failing schools, none are near a subway. These research papers have shown x% lift in test scores when transportation options were accessible” And that surprisingly stupid answer was absolutely accurate. The failing schools turned out to have low participation in after school programs, and a large part was the kids had no way to get home
- renewiltord 4y ago
  Ultimately, I couldn't use Watson to do everyday stuff. But I can with ChatGPT. Watson didn't have the unit economics down.
bumbledraven 4y ago
When I tried this using the same initial prompt as OP, GPT-4 failed to guess "GLOVE" in 6 tries and made errors like guessing CLOAK after CORAL, despite knowing that the first letter wasn't in the secret word. https://pastebin.com/RmtaPTJa https://pastebin.com/RmtaPTJa
daveyjonezz 4y ago
In the prompt he finishes by asking "Do you understand these rules?". I'm curious to see examples where GPT asks clarifying questions on abstract concepts it has been introduced to such as a game like wordle. Depending on the question(s) it asks, I'll decide to look for a different career path or not :-)
Thorentis 4y ago
Don't underestimate how much effort OpenAI is no doubt spending on training their models to solve specific predictable problems, like Wordle, Chess, etc. in order to increase publicity when people inevitably ask them this. This is the missing piece in much of what people are discussing about these AI models. They are putting too much faith in OpenAI thinking they're just launching a neural net and calling it a day. Nope. They are selling a product. And that product is being refined, filtered, tailored, and having every prompt collected and analysed for future opportunities. I wonder how many times it was asked to solve Wordle before an engineer decided to feed Wordle specific training data into it?
ytwySXpMbS 4y ago
Just want to add a data point: I just tried this word for word with today’s Wordle (secret word = tough) and ChatGPT 4 got it completely wrong, didn’t reuse letters it got right and kept reusing letters it knew weren’t in the word. It’s easy to look at this Twitter thread and freak out, but even a broken clock is right twice a day and I encourage people to try these examples for themselves.
- MagicMoonlight 4y ago
  But did you allow it to record its thoughts etc before making the next guess or did you just ask it to keep making guesses?

117 points by amichail 4y ago

markus_zhang 4y ago

Thank you for entertaining me, here is $5.

campbel 4y ago

Can someone just compile a table of Can "X" do "Y" then we can all glance at the diff occasionally?

benlivengood 4y ago
If you read the LLM papers they do that on the standard batch of evaluations: MMLU, hellaswag, winogrande, etc. Every once in a while they have to make up a new evaluation suite since SOTA is close enough to human-level performance to not be impressive any more. Just read the actual papers and the progress is readily apparent.

adroniser 4y ago

Was GPT-3.5 trained before wordle existed?

karmajuney 4y ago
It was but that shouldn’t matter, the coolest part about these models is they should eventually be able to figure out a game like wordle themselves. If the user explains how to play the game, it shouldn’t matter when the model was trained

a-r-t 4y ago

I've been testing all the latest LLMs on a similar game where you need to guess a 4-digit number instead of a word. GPT-4 was the first one to complete it in a reasonable number of tries, although its strategy wasn't optimal.

dragontamer 4y ago
I like this idea much better. Inventing a similar game to Wordle, but making differences. The problem with Wordle is that the internet sensation grew very popular and there's plenty of online discussion (which may have trained the LLM for Wordle inadvertently). Instead, we invent new games with slight variations and see if GPT4 generalizes to those games. Or ideally, inventing a whole new game entirely and seeing if it can learn the game from just a description of its rules.

owenmarshall 4y ago

From the tweets: > GPT-4 showed a deep understanding of the game and played much more optimally than how my slow biological brain would ever have. Are you sure? The letter P is the 14th most common letter found in the English dictionary, and it played it twice on the first guess. I’d expect a system with a deep understanding and playing an optimal game to pick more common letters and definitely to not play a relatively uncommon letter twice.

A_D_E_P_T 4y ago
I played wordle the other day with GPT-4. It also guessed "APPLE" on its first attempt. (It's very odd that the guy who made those tweets experienced the very same first guess.) Astonishingly, GPT-4 was nevertheless able to find the correct word in just three guesses, despite that objectively suboptimal first guess. Impressed the hell out of me. Also, I did a rather poor job of explaining the rules of the game. With the instruction set I provided, I'm not sure that a human would have been able to figure out how to play at all.
- campbel 4y ago
  Chat GPT probably has been given the task before and picked up on it with some keywords.
  - drdeca 4y ago
    I don’t think it is being actively trained on the current conversations? Or at least not in a continual way...?
    - rrrrrrrrrrrryan 4y ago
      Isn't it though? I thought the was the whole point of the free tier: to let people give its responses thumbs ups and thumbs downs and suggest better answers.
      - 4y ago

hn_throwaway_99 4y ago

The thing I have such a hard time understanding (and, in general, I have a very high-level, layman's understanding of LLMs) is how these LLMs are able to understand structure inside tokens. That is, I understood that LLMs just predict the next token in the sequence. But for a game like Wordle, it's obviously critically important that the LLM understands the mapping between the individual letters of, say, "apple", and the output result that the author defined in the prompt, e.g. "XXXXO" means that only "e" is in the word (but the wrong position). I'm just completely baffled how anything in the training procedure could allow the LLM to learn information about the structure of tokens. Does the tokenization process not treat every token (which I thought usually maps to a word) as an "opaque blob"?

Waterluvian 4y ago
Yeah. I need some explanation video that doesn’t just explain the concept but actually walks us through an actual iteration. I haven’t found a good video yet.
- th0ma5 4y ago
  Check out attention visualizations here https://news.ycombinator.com/item?id=33914122 https://news.ycombinator.com/item?id=33914122
- pkoird 4y ago
  I'm not sure you'll find such a video. Emergence is a phenomenon that's not yet well-understood. (But if you do find it somewhere, let me know as well!)
gamegoblin 4y ago
There will be non-trivial amounts of training data that contains sentence like "How do you spell 'apple'? a - p - p - l - e". The hyphenated or spaced out versions are tokenized more like individual letters. So the model can learn the mapping. It's not perfect, though. There are funny failure modes where you can ask the model things like "does the word foobar have a 'b' in it" and it will say "no". (It doesn't fail in the 'foobar' case, but it does fail for some of these cases -- this was just an illustration).

ftxbro 4y ago

This is a really cool example of an emergent ability of GPT-4! I'm honestly so excited to see what abilities the more advanced models of the future will unlock just by reducing the token perplexity! I hope the computer chip factories can keep up!

rstuart4133 4y ago

These "it solved X" posts leave me wondering what I'm looking at. Possibilities are: - GPT-4 has never seen Wordle before getting the prompt. If so, I welcome our new AI overloads. - GPT-4 has seen lots of references to Wordle on the web. This is undoubtedly true for games like Chess, but Wordle was released late 2021. It's still very impressive that it can play a mediocre game a chess just by looking at all the games on the web. - OpenAI anticipated the question, and did some Wordle training just to precipitate a tweet like this. If so - not so impressive as we have been building AI's that some games better than us for some time now. I'm betting it's the last one, but I'd sure like to know if it was one of the others.

DaiPlusPlus 4y ago
It’s the first one. GPT is scary.
SomewhatLikely 4y ago
Wordle is a kind of modified hangman, there's probably lots of data on the web for anagrams and hangman-like games.
- sdenton4 4y ago
  Why is Gpt4 better at these tasks than gpt3 if it only comes down to training data?
- ChatGTP 4y ago
  Likely a lot of "implement a hangman" code. I'm starting to wonder if a large part of GH might be in the training data, hence the "we don't want to release the training data" stance. Bit of a conspiracy maybe, but is it impossible?
  - arlcode 4y ago
    Microsoft is betting really hard on AI. Yesterday I noticed that they integrated a link to "chat" into the start/search Menu of windows 11. I would not at all be surprised if they "sampled" large parts of private GH repositories, paying customers be damned. Probably covered with a corporate fig leave of "tokenizing the data" and hiding behind "to improve our services".

vinyl7 4y ago

Related 3blue1brown video https://www.youtube.com/watch?v=v68zYyaEmEA https://www.youtube.com/watch?v=v68zYyaEmEA Not very hard to train a computer to win at wordle

semicolon_storm 4y ago
There's a big difference between something intentionally built to solve wordle and an LLM. Certainly it's not impressive that humans can build programs to solve wordle, but the fact that a NN that's basically supposed to be fancy autocomplete somehow also acquired whatever it takes to solve wordle is very interesting
- localplume 4y ago
  [dead]

cochne 4y ago

I think this was a fluke... Here is my GPT conversation with the same prompt on today's word. Yes, I understand the rules. Let's play Wordle! Here is my first guess: apple XXXOV Thank you for the feedback. Here's my second guess: slope XVVOV Thank you for the feedback. Based on the information provided, here's my third guess: spore XXVXV Thank you for the feedback. Based on the information provided, here's my fourth guess: savor XXOOX Thank you for the feedback. Based on the information provided, here's my fifth guess: sofar XOXXXX (I made a typo here) Thank you for the feedback. Based on the information provided, here's my sixth and final guess: solar

cochne 4y ago
Update: GPT is clearly very confused about the meaning of the symbols. This might be due to issues with tokenization. >How might you have altered your third guess if you could try again? Given the feedback provided up to the second guess, I had the following information: apple: XXXOV slope: XVVOV Based on this information, I should have considered that the word contains "s," "o," and "e" in the right positions. I could have guessed a word that utilizes the "s" and "e" in the same positions and tries a new letter instead of the second "o." If I could try again, I would change my third guess to: sware This would help me gather more information about the other letters, while still using the correct positions for "s" and "e." I think you're confused. slope: XVVOV means that there is no X in the word, do you see that? You are correct, I misunderstood the feedback. My apologies for the confusion. Based on the feedback provided up to the second guess: apple: XXXOV slope: XVVOV We can deduce that the word has "l" and "e" in their correct positions, and "s" in a different position. Also, there is no "x" in the word. Given this information, my third guess should have been: glare This guess takes into account the correct positions of "l" and "e" and the presence of "s" in a different position.
rkangel 4y ago
Are you definitely using GPT4? The normal chat.openai.com does not use GPT4.

cowllin 4y ago

Hah - very similar to this post earlier today: https://twofergoofer.com/blog/gpt-4 https://twofergoofer.com/blog/gpt-4

anon223345 4y ago

The thing I don’t understand, is we (my team) were doing with Watson (Bluemix / Alchemy, not Watson Assistant) 6 years ago, what ChatGPT is doing now. The difference is we had to “build” (Watson let’s you pick sources) a corpus of data, and tune it, but it worked pretty dang well. It did “generative ai” pretty well but we had to be specific, and it was a pain in the ass to configure, but super accurate. For example Watson gave us a paper about how and why adding an after school van would increase student test scores for a failing school. Something the boards missed entirely. Banks use it, a lot of corporations use Watson AI internally. I’m wondering what Watson is doing while all this is going down. It gets real time feeds from all known sources, you can tell it “only give me info from scholarly articles” etc It is stupid expensive however, I’m just wondering if they’re hiding something. It’s definitely not user friendly, but I imagine they have something hardcore hiding

rhn_mk1 4y ago
> adding an after school van would increase student test scores for a failing school What is an "after school van"? A free bus/taxi service for students?
- anon223345 4y ago
  Ya essentially there were schools super underperforming, and when that happens they get a boost of cash to try crazy shit, and if it doesn’t work within X amount of time they shut the schools down. The school boards met, hired my group to try to apply some AI With Watson we put in every known scholarly article about test score performance We asked it “How can I get students test scores up at these schools” Watson was like “try x,y, … free transportation to after school programs, based on expected volume save money using vans instead of busses. In the failing schools, none are near a subway. These research papers have shown x% lift in test scores when transportation options were accessible” And that surprisingly stupid answer was absolutely accurate. The failing schools turned out to have low participation in after school programs, and a large part was the kids had no way to get home
renewiltord 4y ago
Ultimately, I couldn't use Watson to do everyday stuff. But I can with ChatGPT. Watson didn't have the unit economics down.

bumbledraven 4y ago

When I tried this using the same initial prompt as OP, GPT-4 failed to guess "GLOVE" in 6 tries and made errors like guessing CLOAK after CORAL, despite knowing that the first letter wasn't in the secret word. https://pastebin.com/RmtaPTJa https://pastebin.com/RmtaPTJa

daveyjonezz 4y ago

In the prompt he finishes by asking "Do you understand these rules?". I'm curious to see examples where GPT asks clarifying questions on abstract concepts it has been introduced to such as a game like wordle. Depending on the question(s) it asks, I'll decide to look for a different career path or not :-)

Thorentis 4y ago

Don't underestimate how much effort OpenAI is no doubt spending on training their models to solve specific predictable problems, like Wordle, Chess, etc. in order to increase publicity when people inevitably ask them this. This is the missing piece in much of what people are discussing about these AI models. They are putting too much faith in OpenAI thinking they're just launching a neural net and calling it a day. Nope. They are selling a product. And that product is being refined, filtered, tailored, and having every prompt collected and analysed for future opportunities. I wonder how many times it was asked to solve Wordle before an engineer decided to feed Wordle specific training data into it?

ytwySXpMbS 4y ago

Just want to add a data point: I just tried this word for word with today’s Wordle (secret word = tough) and ChatGPT 4 got it completely wrong, didn’t reuse letters it got right and kept reusing letters it knew weren’t in the word. It’s easy to look at this Twitter thread and freak out, but even a broken clock is right twice a day and I encourage people to try these examples for themselves.

MagicMoonlight 4y ago
But did you allow it to record its thoughts etc before making the next guess or did you just ask it to keep making guesses?