The space of invertible matrices of a given size is not a compact connected Riemannian manifold, so besides the problem of non-uniqueness you probably don't hav

The space of invertible matrices of a given size is not a compact connected Riemannian manifold, so besides the problem of non-uniqueness you probably don't have existence of geodesics for every pair of points. You can't connect a matrix of positive determinant with one of negative determinant without going through non-invertible matrices. (EDIT: I somehow missed that you made the correction about disconnectedness yourself in a later paragraph.) Even restricting to matrices of positive determinant (the space of which is connected), I don't think there are always geodesics between any pair of matrices. For example in the 2x2 case, I bet there is no geodesic from I to -I.