If you have interactions between relatively few methods, then unit-testing those interactions shouldn't be too much of a problem. You wouldn't test all possible

If you have interactions between relatively few methods, then unit-testing those interactions shouldn't be too much of a problem. You wouldn't test all possible inputs, but focus on making sure that you have a test-case for each input that needs to be handled differently. Bugs will still creep through, but they will be substantially fewer. If you really do need tens of thousands of tests because you have so much interaction with unique states going on in your engine then you will probably never ship a product with any reasonable quality. And that floating point operations lacks precision is just the way things are, and you need to understand how to write stable code despite it if you want a stable graphics engine.