I wonder if some of your difficulties are related to the particular structure of Reversi. The obvious heuristic in the endgame - maximize the number of your pi

I wonder if some of your difficulties are related to the particular structure of Reversi. The obvious heuristic in the endgame - maximize the number of your pieces - is actually reversed in the opening and midgame, where you want to minimize your number of pieces (other considerations like edge play and parity being equal). So it's possible your NN managed to do worse than random because it learned an endgame heuristic and generalized it improperly. If so, you may want to consider explicitly wiring in the turn counter as an input.