Another thing that seems overlooked is that we don't just randomly happen to look at objects from weird angles. If you're looking at a chair from that weird top

Another thing that seems overlooked is that we don't just randomly happen to look at objects from weird angles. If you're looking at a chair from that weird top-down angle, probably you walked to it previously and expect to see it like that. And you feel the direction of gravity in your ear, so you feel your viewpoint. The object itself may still be in a random orientation, but then just moving your head around a bit or turning the object clears up any confusion. We do heavily rely on context.