5 ms·

I really thought the author was going to start writing chess notation and Copilot would actually play a game, that would have been impressive.

AnIdiotOnTheNet 5y ago
Well now I'm curious why they didn't. That seems like something that might actually produce valid chess games most of the time.
__alexs 5y ago
I turned Copilot back on see what it would do... I gave it the input > Let's play chess! I'll go first. > 1.e4 c5 Here's the first 7 turns of the game it generated https://lichess.org/bzaWuFNg https://lichess.org/bzaWuFNg I think this is a normal Sicilian opening? At turn 8 it starts not generating full turns anymore. Update: I tried to play a full game against a Level 1 Stockfish bot vs GitHub Copilot. It needed a bit of help sometimes since it generated invalid moves but here's the whole game https://lichess.org/6asVFqwv https://lichess.org/6asVFqwv It resigned after it got stuck in a long loop of moving it's queen back and forth.
- treis 5y ago
  2) D4 is not any mainline opening 4) C3 is just a terrible move that gives away a knight for nothing.
  - buzzy_hacker 5y ago
    2. d4 is the Morra gambit
- idonov 5y ago
  This is awesome
- BbzzbB 5y ago
  Didn't expect level 1 Stockfish to be that bad, for a few moves I was thinking white was Copilot until I verified.
jcuenod 5y ago
I tried playing against it but by its fourth move it was invalid.
- lostmsu 5y ago
  You can switch to the next suggestion until you find a valid one.
nowahe 5y ago
I don't think it's possible with GPT-3, and that's mostly due to how the text is parsed into token before being fed to the network [1]. It breaks down the text in ~4 words token, which allows to effectively quadruple the max input size, at the cost of loosing fine details on the input data. It leads to issues like not being able to create rhymes, not understanding humor or not being able to parse fine structures. Gwern has a nice article talking about the limitations introduced by it [2]. [1] https://beta.openai.com/docs/introduction/tokens https://beta.openai.com/docs/introduction/tokens [2] https://www.gwern.net/GPT-3#bpes https://www.gwern.net/GPT-3#bpes
- gwern 5y ago
  Interestingly, Codex/Copilot might have even more extreme BPE issues than GPT-3 does. They mention that they manage to expand the window in terms of characters greatly (to 4096?), which I take as implying they recomputed the BPEs on the source corpus to make it much more tailored and relevant to source code (which makes sense, because I would expect source code to be far more verbose and repetitive, in terms of syntax and vocab, than Internet-wide natural language, and so simply using the old GPT-2/GPT-3 BPEs won't work well). Which is fantastic if you're doing Python, and that wide window is part of how they do tricks like being able to put in parts of debugging sessions, but if you are going to use Copilot for non-source-code generation, I wonder what the consequences might be...? BPE issues can be hard to notice even when you are looking for them and have the vocab at hand to check.
BuildTheRobots 5y ago
I've got fond (drunk) memories of training a Bayesian spam filter to play chess 15+ years ago, so I was also expecting Copilot to actually play a game. Thanks to your sibling commenter __alexs for doing so. http://dbacl.sourceforge.net/spam_chess-1.html http://dbacl.sourceforge.net/spam_chess-1.html
BuildTheRobots 5y ago
I've got fond (drunk) memories of training a Bayesian spam filter to play chess 15+ years ago, so I was also expecting Copilot to actually play a game. Thanks to your sibling commenter __alexs for doing so. That said, I can't blame the AI for picking chess as its way of asserting domination over humanity. "No human has beaten a computer in a chess tournament in 15 years." http://dbacl.sourceforge.net/spam_chess-1.html http://dbacl.sourceforge.net/spam_chess-1.html. https://www.newswise.com/articles/ai-chess-engine-sacrifices-mastery-to-mimic-human-play https://www.newswise.com/articles/ai-chess-engine-sacrifices...

AnIdiotOnTheNet 5y ago

Well now I'm curious why they didn't. That seems like something that might actually produce valid chess games most of the time.

__alexs 5y ago

I turned Copilot back on see what it would do... I gave it the input > Let's play chess! I'll go first. > 1.e4 c5 Here's the first 7 turns of the game it generated https://lichess.org/bzaWuFNg https://lichess.org/bzaWuFNg I think this is a normal Sicilian opening? At turn 8 it starts not generating full turns anymore. Update: I tried to play a full game against a Level 1 Stockfish bot vs GitHub Copilot. It needed a bit of help sometimes since it generated invalid moves but here's the whole game https://lichess.org/6asVFqwv https://lichess.org/6asVFqwv It resigned after it got stuck in a long loop of moving it's queen back and forth.

treis 5y ago
2) D4 is not any mainline opening 4) C3 is just a terrible move that gives away a knight for nothing.
- buzzy_hacker 5y ago
  2. d4 is the Morra gambit
idonov 5y ago
This is awesome
BbzzbB 5y ago
Didn't expect level 1 Stockfish to be that bad, for a few moves I was thinking white was Copilot until I verified.

jcuenod 5y ago

I tried playing against it but by its fourth move it was invalid.

lostmsu 5y ago
You can switch to the next suggestion until you find a valid one.

nowahe 5y ago

I don't think it's possible with GPT-3, and that's mostly due to how the text is parsed into token before being fed to the network [1]. It breaks down the text in ~4 words token, which allows to effectively quadruple the max input size, at the cost of loosing fine details on the input data. It leads to issues like not being able to create rhymes, not understanding humor or not being able to parse fine structures. Gwern has a nice article talking about the limitations introduced by it [2]. [1] https://beta.openai.com/docs/introduction/tokens https://beta.openai.com/docs/introduction/tokens [2] https://www.gwern.net/GPT-3#bpes https://www.gwern.net/GPT-3#bpes

gwern 5y ago
Interestingly, Codex/Copilot might have even more extreme BPE issues than GPT-3 does. They mention that they manage to expand the window in terms of characters greatly (to 4096?), which I take as implying they recomputed the BPEs on the source corpus to make it much more tailored and relevant to source code (which makes sense, because I would expect source code to be far more verbose and repetitive, in terms of syntax and vocab, than Internet-wide natural language, and so simply using the old GPT-2/GPT-3 BPEs won't work well). Which is fantastic if you're doing Python, and that wide window is part of how they do tricks like being able to put in parts of debugging sessions, but if you are going to use Copilot for non-source-code generation, I wonder what the consequences might be...? BPE issues can be hard to notice even when you are looking for them and have the vocab at hand to check.

BuildTheRobots 5y ago

I've got fond (drunk) memories of training a Bayesian spam filter to play chess 15+ years ago, so I was also expecting Copilot to actually play a game. Thanks to your sibling commenter __alexs for doing so. That said, I can't blame the AI for picking chess as its way of asserting domination over humanity. "No human has beaten a computer in a chess tournament in 15 years." http://dbacl.sourceforge.net/spam_chess-1.html http://dbacl.sourceforge.net/spam_chess-1.html. https://www.newswise.com/articles/ai-chess-engine-sacrifices-mastery-to-mimic-human-play https://www.newswise.com/articles/ai-chess-engine-sacrifices...