Myst 4 is played exactly like this, actually. You're surrounded by panoramic pre-rendered video, and clicking where you want to move to takes you to a new panor

Myst 4 is played exactly like this, actually. You're surrounded by panoramic pre-rendered video, and clicking where you want to move to takes you to a new panorama, analogous with the movement in the original game. (Myst 3 is similar, but it alternates between panoramas and pre-rendered video without doing both at the same time.)