11 ms·

It seems dubious to show that engines are sometimes bad at evaluating positions by giving a position with three black bishops on black squares.

lmilcin 5y ago
While unheard of, this is not illegal. You could theoretically convert two pawns to bishops and have three black bishops. Nobody would do that as it is usual to convert pawns to queens, but it is within the rules for you to choose. So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage.
- ioseph 5y ago
  I could imagine such a move making sense with the king on the backline such that promotion to queen would trigger a stalemate
  - johannes1234321 5y ago
    In rare cases this might happen, two times in a match very unlikely. The only situation where this might be worthwhile would be a forced mate in consequence. Otherwise the Queen is worthy enough that you can afford the tempo loss.
  - tzs 5y ago
    If it is in a tournament, it could even happen for reasons not related to the position itself. Let's say you have a winning position, but you are in severe time pressure. You really need a few moments to stop and think to figure out possible stalemate swindles your opponent might have. You are about to promote to a queen. You realize that both of your bishops are still on the board, and promote to a bishop instead of a queen. That lets you stop the clock and summon an arbiter to go find another bishop of the right color. While the arbiter if off dealing with that, you get your much needed thinking time.
  - deleted 5y ago
    [deleted]
- tialaramex 5y ago
  Underpromotions are one of those "toy problem" things. A human might finish a whole professional career having never once promoted a pawn to anything but a queen, but in theoretical problem positions - like the one you're talking about - they happen "all the time" because it's a fun twist. So it's understandable for a machine not to even bother modelling these weird cases I think.
  - perihelions 5y ago
    edit: This comment is wrong and useless; I'm leaving it up to preserve the logic of the replies. To quantify, here's raw counts from a representative sample of serious games: $ rg -Io '=\w' twic*.pgn | sort | uniq -c 365 =B 1551 =N 122386 =Q 1393 =R I see knight (N) under-promotions pretty regularly on chess streams. The common pattern is there isn't a free tempo to promote the pawn, unless promoting the pawn creates an immediate threat that holds the attacker's initiative; and a knight at that particular moment creates that opportunity.
    - vkou 5y ago
      This obviously can't be trivially gleaned from those statistics - but would you say most of the bishop/rook promotions are just showboating, or are they done to avoid stalemates? I, of course, understand why people would promote to a knight.
      - gamegoblin 5y ago
        I'm an 1850 player and when I am playing under extreme time pressure (e.g. < 2 seconds remaining), I occasionally underpromote to a rook because I can execute a rook+king endgame mechanically faster than a queen+king without an accidental misclick causing a stalemate.
        kadoban 5y ago
        That's interesting. Is it just easier because you can't accidentally move to diagonals, or it's easier in your mind because you've seen it more? If you can even tell the difference.
- dmurray 5y ago
  > So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage. It would not. Rybka (formerly the #1 chess engine) doesn't support promoting to bishops. The author chose an efficient move representation that couldn't handle more than four promotion types. Not quite the same situation, but I can be sure the difference is small even against an opponent trying to exploit it.
  - toast0 5y ago
    Isn't that a big advantage... if you can promote to bishop and the other player crashes, wouldn't you win by default or something?
    - bluGill 5y ago
      So long as you don't crash the engine just evaluates when it happens .bishop promotions are rare.
  - agalunar 5y ago
    Wait, there are only four promotions! Queen, rook, bishop, and knight – you're not allowed to keep a pawn on the back rank.
    - dragonwriter 5y ago
      Yes. It’s possible some engines represent bishop-on-black as a different thing than bishop-on-white, which would make 5 with bishops and 3 without.
    - WJW 5y ago
      Actually now that I think about it, promoting to a second king would be pretty sick. Then you have a spare so losing the original doesn't lose you the game immediately. I could see an incredible game where someone uses this to salvage an otherwise unescapable checkmate.
    - dmurray 5y ago
awb 5y ago
There are other more realistic examples like the Nakamura game sacrificing 2 useless rooks to give the engine a false sense of winning. Then the engine sacrifices a valuable pawn to avoid a 50 move draw. Turn out the pawn was much more valuable than the 2 rooks in the particular closed position and Nakamura goes on to win. Basically, the only way to beat an engine is to try to create an unusual situation that the AI hasn’t practiced before. But as others other out, AI suffers when it doesn’t have enough experience in a particular situation. So, the author is really just pointing out the extreme edge cases AI hasn’t mastered yet. But over time and getting these examples into the training process there’s no reason to believe that the AI couldn’t learn these situations as well.
- anotherqqq 5y ago
  Once you tell a human 'there's a tactic, go', given enough time we find it.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
contravariant 5y ago
Is it though? If humans instantly realise that the three black bishops have some special significance and a chess engine does not then apparently the chess engine lacks some understanding that humans do have.
- vkou 5y ago
  Sure, but it's 'meta' knowledge. If, for instance, the name of a chess problem includes the solution in it, written in plain English, no current chess engine is going to pick up on this bit of 'meta' information - yet any English-speaking human trying to solve the problem would be trivially able to use it. Does that make humans 'better' at chess than chess engines? Not really. Does it make humans 'better' at solving a given chess position, given some metacontextual clues? It does, but it is a bit contrived.
  - contravariant 5y ago
    It's not exactly 'meta' knowledge, it's something that anyone with a few hours of experience in chess would be able to pick up on. It doesn't require knowledge of anything other than chess. If an AI can't pick up on it then that suggests it lacks parts of what to humans would be a basic understanding of chess.
  - taneq 5y ago
    It does, however, seem to be the ‘secret sauce’ to humans’ ability to generalise our problem solving across domains.
  - akyu 5y ago
    It's not 'meta' knowledge, it's a basic rule of the game.
    - vkou 5y ago
      Promoting to a bishop is a rule of the game. Solving a chess puzzle where there are for some reason three black-square bishops - recognizing that the bishops are important to the puzzle is meta-knowledge.

lmilcin 5y ago

While unheard of, this is not illegal. You could theoretically convert two pawns to bishops and have three black bishops. Nobody would do that as it is usual to convert pawns to queens, but it is within the rules for you to choose. So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage.

ioseph 5y ago
I could imagine such a move making sense with the king on the backline such that promotion to queen would trigger a stalemate
- johannes1234321 5y ago
  In rare cases this might happen, two times in a match very unlikely. The only situation where this might be worthwhile would be a forced mate in consequence. Otherwise the Queen is worthy enough that you can afford the tempo loss.
- tzs 5y ago
  If it is in a tournament, it could even happen for reasons not related to the position itself. Let's say you have a winning position, but you are in severe time pressure. You really need a few moments to stop and think to figure out possible stalemate swindles your opponent might have. You are about to promote to a queen. You realize that both of your bishops are still on the board, and promote to a bishop instead of a queen. That lets you stop the clock and summon an arbiter to go find another bishop of the right color. While the arbiter if off dealing with that, you get your much needed thinking time.
- deleted 5y ago
  [deleted]
tialaramex 5y ago
Underpromotions are one of those "toy problem" things. A human might finish a whole professional career having never once promoted a pawn to anything but a queen, but in theoretical problem positions - like the one you're talking about - they happen "all the time" because it's a fun twist. So it's understandable for a machine not to even bother modelling these weird cases I think.
- perihelions 5y ago
  edit: This comment is wrong and useless; I'm leaving it up to preserve the logic of the replies. To quantify, here's raw counts from a representative sample of serious games: $ rg -Io '=\w' twic*.pgn | sort | uniq -c 365 =B 1551 =N 122386 =Q 1393 =R I see knight (N) under-promotions pretty regularly on chess streams. The common pattern is there isn't a free tempo to promote the pawn, unless promoting the pawn creates an immediate threat that holds the attacker's initiative; and a knight at that particular moment creates that opportunity.
  - vkou 5y ago
    This obviously can't be trivially gleaned from those statistics - but would you say most of the bishop/rook promotions are just showboating, or are they done to avoid stalemates? I, of course, understand why people would promote to a knight.
    - gamegoblin 5y ago
      I'm an 1850 player and when I am playing under extreme time pressure (e.g. < 2 seconds remaining), I occasionally underpromote to a rook because I can execute a rook+king endgame mechanically faster than a queen+king without an accidental misclick causing a stalemate.
      - kadoban 5y ago
        That's interesting. Is it just easier because you can't accidentally move to diagonals, or it's easier in your mind because you've seen it more? If you can even tell the difference.
dmurray 5y ago
> So if you plan to write chess engine it would be pretty stupid of you to not prepare it to face multiple black bishops. If I knew that it would give me a lot of advantage. It would not. Rybka (formerly the #1 chess engine) doesn't support promoting to bishops. The author chose an efficient move representation that couldn't handle more than four promotion types. Not quite the same situation, but I can be sure the difference is small even against an opponent trying to exploit it.
- toast0 5y ago
  Isn't that a big advantage... if you can promote to bishop and the other player crashes, wouldn't you win by default or something?
  - bluGill 5y ago
    So long as you don't crash the engine just evaluates when it happens .bishop promotions are rare.
- agalunar 5y ago
  Wait, there are only four promotions! Queen, rook, bishop, and knight – you're not allowed to keep a pawn on the back rank.
  - dragonwriter 5y ago
    Yes. It’s possible some engines represent bishop-on-black as a different thing than bishop-on-white, which would make 5 with bishops and 3 without.
  - WJW 5y ago
    Actually now that I think about it, promoting to a second king would be pretty sick. Then you have a spare so losing the original doesn't lose you the game immediately. I could see an incredible game where someone uses this to salvage an otherwise unescapable checkmate.
  - dmurray 5y ago

awb 5y ago

There are other more realistic examples like the Nakamura game sacrificing 2 useless rooks to give the engine a false sense of winning. Then the engine sacrifices a valuable pawn to avoid a 50 move draw. Turn out the pawn was much more valuable than the 2 rooks in the particular closed position and Nakamura goes on to win. Basically, the only way to beat an engine is to try to create an unusual situation that the AI hasn’t practiced before. But as others other out, AI suffers when it doesn’t have enough experience in a particular situation. So, the author is really just pointing out the extreme edge cases AI hasn’t mastered yet. But over time and getting these examples into the training process there’s no reason to believe that the AI couldn’t learn these situations as well.

anotherqqq 5y ago
Once you tell a human 'there's a tactic, go', given enough time we find it.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

contravariant 5y ago

Is it though? If humans instantly realise that the three black bishops have some special significance and a chess engine does not then apparently the chess engine lacks some understanding that humans do have.

vkou 5y ago
Sure, but it's 'meta' knowledge. If, for instance, the name of a chess problem includes the solution in it, written in plain English, no current chess engine is going to pick up on this bit of 'meta' information - yet any English-speaking human trying to solve the problem would be trivially able to use it. Does that make humans 'better' at chess than chess engines? Not really. Does it make humans 'better' at solving a given chess position, given some metacontextual clues? It does, but it is a bit contrived.
- contravariant 5y ago
  It's not exactly 'meta' knowledge, it's something that anyone with a few hours of experience in chess would be able to pick up on. It doesn't require knowledge of anything other than chess. If an AI can't pick up on it then that suggests it lacks parts of what to humans would be a basic understanding of chess.
- taneq 5y ago
  It does, however, seem to be the ‘secret sauce’ to humans’ ability to generalise our problem solving across domains.
- akyu 5y ago
  It's not 'meta' knowledge, it's a basic rule of the game.
  - vkou 5y ago
    Promoting to a bishop is a rule of the game. Solving a chess puzzle where there are for some reason three black-square bishops - recognizing that the bishops are important to the puzzle is meta-knowledge.