I think some video games have this solved quite nicely: for an offscreen point of interest, show an indicator at the edge of the screen. That way someone knows

I think some video games have this solved quite nicely: for an offscreen point of interest, show an indicator at the edge of the screen. That way someone knows that it's there, and they can click to jump to it (or pan all the way there if they'd like). Definitely, never take away control of the map.