7 ms·
You're "disproving" the article by doing things differently to how the article did. If you're going to disprove that the method given in the article does as wel
by vidarh 4y ago
You're "disproving" the article by doing things differently to how the article did. If you're going to disprove that the method given in the article does as well as the article claims at least use the same method.
- throwwwaway69 4y agoHe literally used the same prompt as the article. Claim: "ChatGPT's Chess Elo is 1400" Reality: ChatGPT gives illegal moves (this happened to article author too), something a 1400 ranked player would never do Result: ChatGPT's rank is not 1400.
- erulabs 4y agoNo, the author of the article specifically says that the entire move sequence should be supplied to chatGPT each time, not simply the next move. Be very careful when "disproving" an experiment with squinted eyes.
- throwwwaway69 4y agoI'm not really sure what to say here. Both the parent commenter and the author of the article had issues with ChatGPT supplying illegal moves. Both methods resulted in this. It sort of doesn't matter how we're trying to establish that it's a 1400 level player, there's no defined correct way to do this. Regardless of method we've disproven it's a 1400 level player due to these illegal moves.
- whimsicalism 4y ago> Regardless of method we've disproven it's a 1400 level player due to these illegal moves. Explain your thought process here further if you don't mind.
- pattrn 4y agoI think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.
- whimsicalism 4y agoThink blindfolded 1400 players, which is what this effectively is, would make illegal moves. But even if it doesn't play like human 1400 players, if it can get to a 1400 elo while resigning games it makes illegal moves on, that seems 1400 level to me. And i bet that some 1400s do occasionally make illegal moves (missing pins) while playing otb
- throwwwaway69 4y agoThis isn't really an apt metaphor. Firstly because higher level blindfolded players, when trained to play with a blindfold, also virtually never make mistakes. Secondly because a computer has permanent concrete state management (compared to humans) and can, without error, keep a perfect representation of a chess if it chooses to do so.
- whimsicalism 4y ago1400 FIDE !=. high level blindfolded player.
- DSMan195276 4y agoPersonally I think the illegal moves are irreverent, the fact that it doesn't play exactly like a typical 1400 doesn't mean it can't have a 1400 rating. Rating is purely determined by wins and losses against opponents, it doesn't matter if you lose a game by checkmate, resignation, or playing an illegal move. That's not to say ChatGPT can play at 1400, just that that playing in an odd way doesn't determine its rating.
- deleted 4y ago[deleted]
- tedsanders 4y agoThe #1 misconception when working with large language models is thinking that a capability is a property of the model, rather than the model + input. It may be simultaneously true that ChatGPT has an elo of 100 when given a conversational message and an elo of 1400 when given an optimized message (e.g., strings that resemble chess games, with many examples present in the conversation). Understanding this concept is crucial for getting good results out of large language models.
- deleted 4y ago[deleted]
- mynameisvlad 4y ago> something a 1400 ranked player would never do The fact that rules and articles exist describing what to do if you or your opponent makes an illegal move indicates this is not the case. Humans are also... human. They make mistakes. It may not happen often at 1400, but to say that it'll never happen is preposterous.
- throwwwaway69 4y agofine, fair, "never" was too much. posting link to this comment to not repeat same discussion twice https://news.ycombinator.com/item?id=35201037 https://news.ycombinator.com/item?id=35201037
- eddsh1994 4y agoI can’t remember the last time I played an illegal move tbf, and I’ve played 7 games of chess this morning already to give you an idea of total games played
- mynameisvlad 4y agoYou have never made an illegal move, ever? The bar isn’t “I didn’t make an illegal move this morning” it’s “something a 1400 ranked player would never do”. My entire point is that it happens. Not often, but also not “never”.
- unyttigfjelltol 4y agoThe author said ChatGPT gives illegal moves. So, a quirky sort of 'grandmaster'. He considered illegal moves to be a resignation. Maybe you need to tell ChatGPT that the alternatives are to win via legal moves, and if it is not possible to do so, to resign? Does that fix it?
- z3c0 4y agoThey are disproving an assertion. Demonstrating that an alternate approach implodes the assertion is a perfectly acceptable route, especially when the original approach was cherry-picking successes and throwing out failures. I wish I could just make bullshit moves and get a higher chess ranking. Sounds nice.
- whimsicalism 4y agoI disagree. If there is a procedure for getting ChatGPT to play chess accurately and you discard that and do some naive approach as a way of disproving the article, doesn't sound to me like you have disproven anything. I dont understand the point of your second sentence, seems to be entirely missing the substance of the conversation.
- z3c0 4y agoThe gymnastics you GPT True Believers go through to make this stuff "work" are really something else. By the way - definitely read the article. But once again - I thought the methodology was bad, and thus the conclusion was bad.
- whimsicalism 4y agoI don’t think this is any crazy level of gymnastics. But not going to keep replying, you engage online in a way that will turn lots of people you talk to away.
- z3c0 4y agoI'll admit to having mistook your reply with another (hence the non-sequitur second half of my comment.) Apologies for my brusque tone.
- vidarh 4y agoIt was not throwing out failures. It was treating even the first illegal move as a forfeit something which is stricter than FIDE rules.
- k2052 4y agoYou are right that my method differed slightly so I did things again. It took me one try to find a sequence of moves that "breaks" what is claimed. You just have to make odd patterns of moves and it clearly has no understanding of the position. Here is the convo: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Alright, I'm ready to play! Please give me the move sequence. me: 1. e3 Nf6 2. f4 d6 3. e4 ChatGPT: My next move as black would be 3... e5 Completely ignoring the hanging pawn.This is not the play of a 1400 elo player. It is the play of something predicting patterns. I ran a bunch of experiments in the past where I played normal moves and ChatGPT does respond extraordinarily well. With the right prompts and sequences you can get it to play like a strong grandmaster. But it is a "trick" you are getting it to perform by choosing good data and prompts. It is impressive but it is not doing what is claimed by the article.
- nwienert 4y agoI'll add in as someone new to chess (~800 ELO): ChatGPT is in no way 1400, or even close to it. The fact this article gets upvoted around here is proof that people aren't thinking clearly about this stuff. It's trivially easy to prove it wrong. Live unbelievably so, I tried the same prompt and within 12 moves it made multiple ridiculous errors I never would, and then an illegal move. Keep in mind a 1400 level player would need to basically make 0 mistakes that bad in a typical game, and further would need to play 30-50 moves in that fashion, with the final moves being some of the most important and hard to do. There's just no way it's even close, my guess would be even if you correct it's many errors, it's something like ~200 ELO. Pure FUD. The author of this article is cashing in the hype and I'm wondering how they even got the results they did.
- babel_ 4y agoThey probably got them. The problem is that it's difficult to repeat, thanks to temperature, meaning users will get a random spread of outcomes. Today, someone got a legal game. Tomorrow, someone might get a grandmaster level game. But then everyone else trying to repeat or leverage this ends up with worse luck and gets illegal moves or, if they're lucky, moves that make sense in a limited context (such as related to specific gambits etc) but have no role in longer-term play.
- PaulHoule 4y agoIt’s super scary how ChatGPT brings out people who are veeeery good at seeing the Emperor’s clothes.
- YeGoblynQueenne 4y agoYou know, I didn't remember the story very well so I checked wikipedia. Here's what it says about the (start of) the plot: >> Two swindlers arrive at the capital city of an emperor who spends lavishly on clothing at the expense of state matters. Posing as weavers, they offer to supply him with magnificent clothes that are invisible to those who are stupid or incompetent. The emperor hires them, and they set up looms and go to work. A succession of officials, and then the emperor himself, visit them to check their progress. Each sees that the looms are empty but pretends otherwise to avoid being thought a fool. So everyone "pretends otherwise to avoid being thought a fool". Huh. I guess that explains it. Good metaphor.