16 ms·

Debunking a Computer Chess Scandal

77 points by worldvoyageur 15y ago

worldvoyageur 15y ago
Simply on the basis of telling a fascinating story, the series is well worth reading. Beyond that, some of what particularly struck me was that: - since about 2005, chess programs that run on a regular PC have consistently outperformed the best human grandmasters, and the skill of the computer chess programs continues to improve. 2005 was a quantum shift in the chess programming world, when chess programs first played "brilliant games with deep, beautiful combinations...[and] routinely produce[d] highly artistic masterpieces of chess while avoiding a great many pointless “computer” moves that for many years had been a source of ridicule among strong human players." - improvement in algorithms is driven by an active community ecosystem, open source contributions and flagrant copying of the brilliant insights of a few. No matter how brilliant the insight, the community is able to improve upon it. Intellectual property protection would have stopped much of this improvement in its tracks. It's four separate articles, so I posted the first. All four are: http://www.chessbase.com/newsdetail.asp?newsid=7791 http://www.chessbase.com/newsdetail.asp?newsid=7791 (part 1) http://www.chessbase.com/newsdetail.asp?newsid=7807 http://www.chessbase.com/newsdetail.asp?newsid=7807 (part 2) http://www.chessbase.com/newsdetail.asp?newsid=7811 http://www.chessbase.com/newsdetail.asp?newsid=7811 (part 3) http://www.chessbase.com/newsdetail.asp?newsid=7813 http://www.chessbase.com/newsdetail.asp?newsid=7813 (part 4)
- wisty 15y ago
  A quote from Rajlich: > Yes, the publication of Fruit 2.1 was huge. Look at how many engines took a massive jump in its wake: Rybka, HIARCS, Fritz, Zappa, Spike, List, and so on. I went through the Fruit 2.1 source code forwards and backwards and took many things. It's not as if he never admitted to studying Fruit. Also, most of the confusion is there because Rajlich didn't have a repo until Rybka 4, so when the reverse engineered his code they formatted it the same way as Fruit, making it look very similar. If a judge missed the bit saying that the Rybka code is "functionally equivalent code", not the actual code, they might be fooled into thinking the orignal codebases looked very similar. The guy doing this wrote something like 30 posts a day on the discussion board about Rybka, and was clearly not a fan. They should have made a 3-way comparison - Rybka's PST, Fruit's PST, and some independent program's PST (in the same format as the other two).
- jules 15y ago
  What is a good resource to learn about the important advances in computer chess?
- deleted 15y ago
  [deleted]
- Bootvis 15y ago
  A response has come from the author and panel members on Chessvibes, see the pdf links in here: http://chessvibes.com/reports/controversy-over-rybkas-disqualification-and-ban http://chessvibes.com/reports/controversy-over-rybkas-disqua...
copper 15y ago
For what it's worth, the WCCC's Rule 2 seems to be a lot like a university honor code (if you do borrow code or ideas, make sure you cite it.) I don't believe that's too hard to follow. Maybe someone should fund a kaggle competition to design the best chess-playing program, with the final requirement of making the code open-source after :) Given that most of what you'd implement a computer chess engine is pretty much available online (at most,behind a paywall), and digging through the source code of GNUChess and Crafty would give a lot of insight into the scoring function, so, yes, the score function would be pretty much similar in most parts unless someone comes up with a radically new way of doing things.
wbhart 15y ago
From the article, "By definition, plagiarism only happens when credit to sources is not given, which was never the case with Rybka." I simply do not agree with this. I cannot copy slabs of open source code, incorporate them into my product and pass if off as my own just because I somewhere credited the great performance of my program to stuff I learned from reading open source software. To me, plagiarism is copying some portion of someone else's work and including it in a work which I (explicitly or implicitly) claim to be my own. For example, a student handing in a wikipedia article for a homework assignment still commits plagiarism even if they credit wikipedia as a source but don't explicitly say the thing is a wikipedia article. They hand it in as their work when they did not write it. So at this point I do not feel I agree with the article. It's as though the argument is, yes he copied, but it's ok because he gave "credit", there were plenty of new ideas and everyone else was doing it too. That's not the same thing as, "he didn't commit plagiarism". The latter means he did not copy the code. Having said that, I am absolutely gobsmacked that the committee did not simply ask for the source code to both programs, check that it in fact compiles to the binaries in question, then do a comparison for legally significant quantities of identical code. It's even more remarkable when you realise that the version of Fruit involved was open source, so they didn't even need to ask for the source code to that! Note: I edited the above in response to the comments below.
- sp332 15y ago
  I disagree. I think if a student turns in an assignment that's clearly and admittedly just a WP article, the student should not get credit for doing the work, but he didn't commit plagiarism.
  - deleted 15y ago
    [deleted]
- SeanLuke 15y ago
  I don't have an opinion on what this guy did and whether he should have been stripped, but you're redefining plagiarism pure and simple. Plagiarism is passing off someone else's ideas or expression of those ideas as if they were your own. And that's it. Plagiarism has nothing to do with whether you benefit from someone else's ideas illegally or unethically. Let's say that there's a competition to write an original novel. I take Moby Dick and change one sentence. Then I submit it with a big warning on the front cover that says: "This novel is Moby Dick with one sentence changed by me." I have violated the rules of the contest perhaps. If I win I maybe benefitted from it. But I have not committed plagiarism -- I did not pass off these ideas as if they were my own. It was made clear whose ideas they were.
losethos 15y ago
False whitness is a commandment. I'm not sure plagarism is mentioned. Charity is not mentioned. Thou shall not covet is mentioned. These things mean more when God exists. God says... C:\LoseThos\www.losethos.com\text\QUIX.TXT nts referring to this famous knight; and this being his persuasion, he did not despair of finding the conclusion of this pleasant history, which, heaven favouring him, he did find in a way that shall be related in the Second Part. CHAPTER IX. IN WHICH IS CONCLUDED AND FINISHED THE TERRIFIC BATTLE BETWEEN THE GALLANT BISCAYAN AND THE VALIANT MANCHEGAN In the First Part of this history we left the valiant Biscayan and the renowned Don Quixote with drawn swords uplifted, ready to deliver t --------------- You have no idea how bassackward your inquisition is. Now, you shall be judged by God. Judge not lest you be judged. By the standard you judge, you shall be judged. Jesus commanded charity, but it didn't make the ten. God says... C:\LoseThos\www.losethos.com\text\BIBLE.TXT D thy God hath heard: wherefore lift up thy prayer for the remnant that is left. 37:5 So the servants of king Hezekiah came to Isaiah. 37:6 And Isaiah said unto them, Thus shall ye say unto your master, Thus saith the LORD, Be not afraid of the words that thou hast heard, wherewith the servants of the king of Assyria have blasphemed me. 37:7 Behold, I will send a blast upon him, and he shall hear a rumour, and return to his own land; and I will cause him to fall by the sword in his own land.
mark_l_watson 15y ago
Good article. I agree that to slam someone's reputation in public, the ICGA should have had much better evidence. My understanding is that the release of Fruit source code inspired many chess programmers.
medusa666 15y ago
It has been discussed (in chess circles) in excruciating and unambiguous detail how Rajlich blatantly ripped off (Google it) open source code, thereby violating tournament rules, open source licenses, etc. This "debunking" series of articles by Chessbase is a pathetic attempt to reinvent the past on behalf of one of their most lucrative products, and perhaps to fool a wider audience who didn't follow the original scandal when it happened. I used to be a big fan of Rybka, Rajlich and Chessbase, but these continued denials just make a bad thing worse. It's like Floyd Landis, Tyler Hamilton, and the third guy.
- wisty 15y ago
  The thing I'd like to see - a three way comparison of the alleged "cheating". Rybka and Fruit have some similar parts, but how about Fruit and any other similar program? It might just be that any chess program will have parts that look similar.
  - jsnell 15y ago
    A document like that was included in the original investigation, with 9 different chess engines. It didn't compare source code though, but on which features of the board each engine would consider when evaluating a position. For example one engine might give an extra boost to a position where your knight is in a central location and capable of affecting a larger part of the board, another might give a boost to situations with interlocking pawn defenses, etc. According to the evidence, two engines with different ancestry would have a feature overlap of about 30-35%, up to 45% in the most extreme cases. With the exception of Fruit and Rybka, with 75% overlap.
- Almaviva 15y ago
  As far as I can tell, the basic facts are: 1. There is damning evidence (probably beyond reasonable doubt) that Rybka contains some copy-pasted code from Fruit (the open source program). 2. The interesting thing about Rybka was that it was dramatically better than any chess engine of its time, due to original improvements. 3. Many people are bitter that Rybka became an extremely successful commercial product, due to being clearly the best chess engine of its time. It also made several formerly viable commercial products non-viable due to being much better. 4. A mass of clones obtained by either disassembly or obtaining source code appeared, which are now equal or slightly superior to Rybka, and this has essentially destroyed its commercial viability. For people who value the letter of the law, #1 is damning, and Rybka cheated, period. This isn't beyond theoretical doubt, but it's beyond a reasonable one. But, given that Rajlich could have easily and legally re-implemented the same algorithms in the mudane parts of Fruit that are in question without exact code reuse, and that what is significant about Rybka is actually the original and innovative part, there's a pretty good argument that he's well within his rights to profit from his creation.
ww520 15y ago
OT: why do websites find it useful to disable the up/down arrow keys? It is not. It's very annoying. Make navigation a pain in the butt. Make people never want to come back to your site.
- jmmcd 15y ago
  Maybe it's the fault of the game-history widgets on the page. They use left and right arrow keys for game navigation, so it's possible they steal up and down arrows also.
mml 15y ago
A fine example of terrible writing.
rwmj 15y ago
For UK/Hungary computer nerds, Dr David Levy was also responsible for the Enterprise 64 computer: https://en.wikipedia.org/wiki/Enterprise_%28computer%29 https://en.wikipedia.org/wiki/Enterprise_%28computer%29
bjornsing 15y ago
Isn't there something slightly ironic about accusing the best pupil in the class of peeking over the shoulders of the other students during tests? Not that it can't be a correct accusation, but if this guy "cruised to victory in four consecutive WCCC tournaments in 2007, 2008, 2009 and 2010" then he must have done something right (other than copy-paste). From a copyright perspective I can certainly see an issue, but that would be between Rajlich and the author of this Fruit I would say.
- dalke 15y ago
  Actually, the argument isn't about cut&paste programming. That would be somewhat easy to determine by simply comparing the binary output (after some searching for the right compiler options from the compiler for that era). Indeed, the accusations clearly say that there were significant changes to the code, which makes a reverse engineered comparison harder to do. Instead, the complaint is that the underlying algorithms were re-implemented, instead of being original algorithms.
janzer 15y ago
Although the articles seem to not mention it, it should probably be noted the author is a moderator on the Rybka forums so is most likely not coming from an exactly unbiased viewpoint.
- worldvoyageur 15y ago
  Though buried as an aside in the fourth part of the piece, the author does mention that he is the moderator on the Rybka forums. "In my capacity as Rybka forum moderator I have access to posting statistics." Indeed, his moderator status leads to one of his core debunking arguments. Some Dr. Hyatt guy, apparently a key individual behind the accusations against Rybka, appeared to be making the attack a full-time job, shifting the arguments as needed to defend a guilty conclusion. Though he doesn't say it, I suspect that noticing this was impetus for the author to seriously tackle a look at the charges from the defense perspective. Not that there is anything wrong with either side picking a conclusion (guilty, not guilty) and then working hard to support it. However, part of the credence given the initial charge is that it resulted from an independent search for truth. This does not appear to have been the case. Instead, the charges were the prosecution half of an adversarial process. In these articles, now we see the defense. Caveat: all I know about this matter is what I read in the four part article posted here.

77 points by worldvoyageur 15y ago

Simply on the basis of telling a fascinating story, the series is well worth reading. Beyond that, some of what particularly struck me was that: - since about 2005, chess programs that run on a regular PC have consistently outperformed the best human grandmasters, and the skill of the computer chess programs continues to improve. 2005 was a quantum shift in the chess programming world, when chess programs first played "brilliant games with deep, beautiful combinations...[and] routinely produce[d] highly artistic masterpieces of chess while avoiding a great many pointless “computer” moves that for many years had been a source of ridicule among strong human players." - improvement in algorithms is driven by an active community ecosystem, open source contributions and flagrant copying of the brilliant insights of a few. No matter how brilliant the insight, the community is able to improve upon it. Intellectual property protection would have stopped much of this improvement in its tracks. It's four separate articles, so I posted the first. All four are: http://www.chessbase.com/newsdetail.asp?newsid=7791 http://www.chessbase.com/newsdetail.asp?newsid=7791 (part 1) http://www.chessbase.com/newsdetail.asp?newsid=7807 http://www.chessbase.com/newsdetail.asp?newsid=7807 (part 2) http://www.chessbase.com/newsdetail.asp?newsid=7811 http://www.chessbase.com/newsdetail.asp?newsid=7811 (part 3) http://www.chessbase.com/newsdetail.asp?newsid=7813 http://www.chessbase.com/newsdetail.asp?newsid=7813 (part 4)

wisty 15y ago
A quote from Rajlich: > Yes, the publication of Fruit 2.1 was huge. Look at how many engines took a massive jump in its wake: Rybka, HIARCS, Fritz, Zappa, Spike, List, and so on. I went through the Fruit 2.1 source code forwards and backwards and took many things. It's not as if he never admitted to studying Fruit. Also, most of the confusion is there because Rajlich didn't have a repo until Rybka 4, so when the reverse engineered his code they formatted it the same way as Fruit, making it look very similar. If a judge missed the bit saying that the Rybka code is "functionally equivalent code", not the actual code, they might be fooled into thinking the orignal codebases looked very similar. The guy doing this wrote something like 30 posts a day on the discussion board about Rybka, and was clearly not a fan. They should have made a 3-way comparison - Rybka's PST, Fruit's PST, and some independent program's PST (in the same format as the other two).
jules 15y ago
What is a good resource to learn about the important advances in computer chess?
deleted 15y ago
[deleted]
Bootvis 15y ago
A response has come from the author and panel members on Chessvibes, see the pdf links in here: http://chessvibes.com/reports/controversy-over-rybkas-disqualification-and-ban http://chessvibes.com/reports/controversy-over-rybkas-disqua...

copper 15y ago

For what it's worth, the WCCC's Rule 2 seems to be a lot like a university honor code (if you do borrow code or ideas, make sure you cite it.) I don't believe that's too hard to follow. Maybe someone should fund a kaggle competition to design the best chess-playing program, with the final requirement of making the code open-source after :) Given that most of what you'd implement a computer chess engine is pretty much available online (at most,behind a paywall), and digging through the source code of GNUChess and Crafty would give a lot of insight into the scoring function, so, yes, the score function would be pretty much similar in most parts unless someone comes up with a radically new way of doing things.

wbhart 15y ago

From the article, "By definition, plagiarism only happens when credit to sources is not given, which was never the case with Rybka." I simply do not agree with this. I cannot copy slabs of open source code, incorporate them into my product and pass if off as my own just because I somewhere credited the great performance of my program to stuff I learned from reading open source software. To me, plagiarism is copying some portion of someone else's work and including it in a work which I (explicitly or implicitly) claim to be my own. For example, a student handing in a wikipedia article for a homework assignment still commits plagiarism even if they credit wikipedia as a source but don't explicitly say the thing is a wikipedia article. They hand it in as their work when they did not write it. So at this point I do not feel I agree with the article. It's as though the argument is, yes he copied, but it's ok because he gave "credit", there were plenty of new ideas and everyone else was doing it too. That's not the same thing as, "he didn't commit plagiarism". The latter means he did not copy the code. Having said that, I am absolutely gobsmacked that the committee did not simply ask for the source code to both programs, check that it in fact compiles to the binaries in question, then do a comparison for legally significant quantities of identical code. It's even more remarkable when you realise that the version of Fruit involved was open source, so they didn't even need to ask for the source code to that! Note: I edited the above in response to the comments below.

sp332 15y ago
I disagree. I think if a student turns in an assignment that's clearly and admittedly just a WP article, the student should not get credit for doing the work, but he didn't commit plagiarism.
- deleted 15y ago
  [deleted]
SeanLuke 15y ago
I don't have an opinion on what this guy did and whether he should have been stripped, but you're redefining plagiarism pure and simple. Plagiarism is passing off someone else's ideas or expression of those ideas as if they were your own. And that's it. Plagiarism has nothing to do with whether you benefit from someone else's ideas illegally or unethically. Let's say that there's a competition to write an original novel. I take Moby Dick and change one sentence. Then I submit it with a big warning on the front cover that says: "This novel is Moby Dick with one sentence changed by me." I have violated the rules of the contest perhaps. If I win I maybe benefitted from it. But I have not committed plagiarism -- I did not pass off these ideas as if they were my own. It was made clear whose ideas they were.

losethos 15y ago

False whitness is a commandment. I'm not sure plagarism is mentioned. Charity is not mentioned. Thou shall not covet is mentioned. These things mean more when God exists. God says... C:\LoseThos\www.losethos.com\text\QUIX.TXT nts referring to this famous knight; and this being his persuasion, he did not despair of finding the conclusion of this pleasant history, which, heaven favouring him, he did find in a way that shall be related in the Second Part. CHAPTER IX. IN WHICH IS CONCLUDED AND FINISHED THE TERRIFIC BATTLE BETWEEN THE GALLANT BISCAYAN AND THE VALIANT MANCHEGAN In the First Part of this history we left the valiant Biscayan and the renowned Don Quixote with drawn swords uplifted, ready to deliver t --------------- You have no idea how bassackward your inquisition is. Now, you shall be judged by God. Judge not lest you be judged. By the standard you judge, you shall be judged. Jesus commanded charity, but it didn't make the ten. God says... C:\LoseThos\www.losethos.com\text\BIBLE.TXT D thy God hath heard: wherefore lift up thy prayer for the remnant that is left. 37:5 So the servants of king Hezekiah came to Isaiah. 37:6 And Isaiah said unto them, Thus shall ye say unto your master, Thus saith the LORD, Be not afraid of the words that thou hast heard, wherewith the servants of the king of Assyria have blasphemed me. 37:7 Behold, I will send a blast upon him, and he shall hear a rumour, and return to his own land; and I will cause him to fall by the sword in his own land.

mark_l_watson 15y ago

Good article. I agree that to slam someone's reputation in public, the ICGA should have had much better evidence. My understanding is that the release of Fruit source code inspired many chess programmers.

medusa666 15y ago

It has been discussed (in chess circles) in excruciating and unambiguous detail how Rajlich blatantly ripped off (Google it) open source code, thereby violating tournament rules, open source licenses, etc. This "debunking" series of articles by Chessbase is a pathetic attempt to reinvent the past on behalf of one of their most lucrative products, and perhaps to fool a wider audience who didn't follow the original scandal when it happened. I used to be a big fan of Rybka, Rajlich and Chessbase, but these continued denials just make a bad thing worse. It's like Floyd Landis, Tyler Hamilton, and the third guy.

wisty 15y ago
The thing I'd like to see - a three way comparison of the alleged "cheating". Rybka and Fruit have some similar parts, but how about Fruit and any other similar program? It might just be that any chess program will have parts that look similar.
- jsnell 15y ago
  A document like that was included in the original investigation, with 9 different chess engines. It didn't compare source code though, but on which features of the board each engine would consider when evaluating a position. For example one engine might give an extra boost to a position where your knight is in a central location and capable of affecting a larger part of the board, another might give a boost to situations with interlocking pawn defenses, etc. According to the evidence, two engines with different ancestry would have a feature overlap of about 30-35%, up to 45% in the most extreme cases. With the exception of Fruit and Rybka, with 75% overlap.
Almaviva 15y ago
As far as I can tell, the basic facts are: 1. There is damning evidence (probably beyond reasonable doubt) that Rybka contains some copy-pasted code from Fruit (the open source program). 2. The interesting thing about Rybka was that it was dramatically better than any chess engine of its time, due to original improvements. 3. Many people are bitter that Rybka became an extremely successful commercial product, due to being clearly the best chess engine of its time. It also made several formerly viable commercial products non-viable due to being much better. 4. A mass of clones obtained by either disassembly or obtaining source code appeared, which are now equal or slightly superior to Rybka, and this has essentially destroyed its commercial viability. For people who value the letter of the law, #1 is damning, and Rybka cheated, period. This isn't beyond theoretical doubt, but it's beyond a reasonable one. But, given that Rajlich could have easily and legally re-implemented the same algorithms in the mudane parts of Fruit that are in question without exact code reuse, and that what is significant about Rybka is actually the original and innovative part, there's a pretty good argument that he's well within his rights to profit from his creation.

ww520 15y ago

OT: why do websites find it useful to disable the up/down arrow keys? It is not. It's very annoying. Make navigation a pain in the butt. Make people never want to come back to your site.

jmmcd 15y ago
Maybe it's the fault of the game-history widgets on the page. They use left and right arrow keys for game navigation, so it's possible they steal up and down arrows also.

mml 15y ago

A fine example of terrible writing.

rwmj 15y ago

For UK/Hungary computer nerds, Dr David Levy was also responsible for the Enterprise 64 computer: https://en.wikipedia.org/wiki/Enterprise_%28computer%29 https://en.wikipedia.org/wiki/Enterprise_%28computer%29

bjornsing 15y ago

Isn't there something slightly ironic about accusing the best pupil in the class of peeking over the shoulders of the other students during tests? Not that it can't be a correct accusation, but if this guy "cruised to victory in four consecutive WCCC tournaments in 2007, 2008, 2009 and 2010" then he must have done something right (other than copy-paste). From a copyright perspective I can certainly see an issue, but that would be between Rajlich and the author of this Fruit I would say.

dalke 15y ago
Actually, the argument isn't about cut&paste programming. That would be somewhat easy to determine by simply comparing the binary output (after some searching for the right compiler options from the compiler for that era). Indeed, the accusations clearly say that there were significant changes to the code, which makes a reverse engineered comparison harder to do. Instead, the complaint is that the underlying algorithms were re-implemented, instead of being original algorithms.

janzer 15y ago

Although the articles seem to not mention it, it should probably be noted the author is a moderator on the Rybka forums so is most likely not coming from an exactly unbiased viewpoint.

worldvoyageur 15y ago
Though buried as an aside in the fourth part of the piece, the author does mention that he is the moderator on the Rybka forums. "In my capacity as Rybka forum moderator I have access to posting statistics." Indeed, his moderator status leads to one of his core debunking arguments. Some Dr. Hyatt guy, apparently a key individual behind the accusations against Rybka, appeared to be making the attack a full-time job, shifting the arguments as needed to defend a guilty conclusion. Though he doesn't say it, I suspect that noticing this was impetus for the author to seriously tackle a look at the charges from the defense perspective. Not that there is anything wrong with either side picking a conclusion (guilty, not guilty) and then working hard to support it. However, part of the credence given the initial charge is that it resulted from an independent search for truth. This does not appear to have been the case. Instead, the charges were the prosecution half of an adversarial process. In these articles, now we see the defense. Caveat: all I know about this matter is what I read in the four part article posted here.