In particular, INAV has problems with keeping the position of the horizon accurately, and usually thinks it's going straight while it's dramatically losing alti

In particular, INAV has problems with keeping the position of the horizon accurately, and usually thinks it's going straight while it's dramatically losing altitude. So, "all of the above", really. The biggest issue is that this usually happens during automatic modes, which are things like "return to home", which you usually use when you've lost control or video and you really want RTH to work reliably. I can't risk a plane falling onto a car or person, so I use ArduPilot, whose auto modes are impressively accurate.