I get your point, but you also have to keep track of castling rights. If King or Rook have moved at any point (not necessarily on previous move), you can't cast

I get your point, but you also have to keep track of castling rights. If King or Rook have moved at any point (not necessarily on previous move), you can't castle even if they're in their starting position again.