A lot of the trouble of implementing numerical linear algebra is getting decent performance and numerical stability using finite-precision operations with bound

A lot of the trouble of implementing numerical linear algebra is getting decent performance and numerical stability using finite-precision operations with bounded numerical representations. How do you think this sort of generalized approach would work if you want things to behave nicely with, say, poorly conditioned matrices, or matrices with certain structure that might get clobbered when changing to another algebra? Are these things even necessary to consider in such a design? Or would this direction be mostly for bootstrapping a functional—but perhaps not performant or numerically stable—library?