This delay also often happens when I switch between linux virtual consoles (Control+Alt+F7/F8), despite both of them using exactly the same X display settings.

This delay also often happens when I switch between linux virtual consoles (Control+Alt+F7/F8), despite both of them using exactly the same X display settings. Not always, though. The switch sometimes happens instantly. (At least, it has been this way since I replaced my Nvidia card with an AMD. I don't recall whether the Nvidia ever switched instantly.) I wish I knew what conditions make it instant, so I could try to make it happen consistently.