Unfortunately most implementations of orientation on these devices rely simply on the phone's IMU -- and since the phone is perpendicular to the ground, I'm gue

Unfortunately most implementations of orientation on these devices rely simply on the phone's IMU -- and since the phone is perpendicular to the ground, I'm guessing they don't make heavy use of the compass. Therefore, the only way to get the direction you are pointing is to extrapolate it from the changes in forces that the IMU senses. Unsurprisingly, over time, the error compounds and really destroys the experience -- as you have discovered!