4 ms·

Our code base is primarily Python. We do use type checking, but we risk having problems that you simply don't have in strongly typed languages. Unit test save

Our code base is primarily Python. We do use type checking, but we risk having problems that you simply don't have in strongly typed languages. Unit test save the day every time I need to refactor large parts of the code base or make a change that affects the entire code base. They don't necessarily find problems I didn't think of, but they make these kind of large refactoring a lot easier. Recently we replaced the ORM. I replaced the code that handles setting things up, and wrote the code to configure the new ORM. Then I just ran the tests. Almost all of them failed because the imports and syntax are slightly different. So, I would just take it test by test and kept fixing things until the tests passed. When all the tests were passing I was reasonably sure everything was still working as expected. Sure, we also face the problems you're facing. Sometimes the tests just fail for unrelated reasons. Or somebody wrote a test that is a bit flaky and fails from time to time. Usually we just consider this the cost of doing business so to say. When tests are failing every time the code changes, that usually indicates a problem with how you write tests. Stick to the single assert principle, make sure tests are simple, don't require a lot of set up etc. complex tests are usually a sign of complex code. In dynamic languages, mocks are very popular, but I've found them to be the source of having to change things. They are often a sign that the code is poorly split and untestable. Instead, write smaller, isolated modules and possibly use dependency injection, making the "mocking" part of the code.

mbrock 8y ago
Basic idea: to verify your robot in a quick and harmless way, disconnect its actuators, feed fake data to its sensors, observe its stream of decisions, and verify that it behaves according to some rules. If your robot has different parts with their own rules of behavior, then you should be able to detach them to test them on their own, or in a much simpler test harness. I think the complaints about unit tests often amount to “we’re taking the robot apart into too many parts that actually depend on each other in complex ways, so testing them independently is tedious and gives us little real information.” Another kind of complaint is “our robot has a huge amount of functions that all interact in very complicated ways and we haven’t specified any overarching principles so our tests are just a lot of arbitrary scripts that we have to change constantly to accomodate the robot’s ever-changing repertoire of fascinating behaviors.” Or “there is no exhaustive list of our robot’s sensors and actuators, and many of them are complex third party products which we integrate without any adapters, so it’s nearly impossible to take the thing apart and provide a realistic simulated environment.” I think the underlying complaint is something like “we were told that there was an easy method to make our software correct, and it’s disappointing that it doesn’t work well without considerable investment into domain modeling, architectural work, mathematical analysis, etc.”

mbrock 8y ago

Basic idea: to verify your robot in a quick and harmless way, disconnect its actuators, feed fake data to its sensors, observe its stream of decisions, and verify that it behaves according to some rules. If your robot has different parts with their own rules of behavior, then you should be able to detach them to test them on their own, or in a much simpler test harness. I think the complaints about unit tests often amount to “we’re taking the robot apart into too many parts that actually depend on each other in complex ways, so testing them independently is tedious and gives us little real information.” Another kind of complaint is “our robot has a huge amount of functions that all interact in very complicated ways and we haven’t specified any overarching principles so our tests are just a lot of arbitrary scripts that we have to change constantly to accomodate the robot’s ever-changing repertoire of fascinating behaviors.” Or “there is no exhaustive list of our robot’s sensors and actuators, and many of them are complex third party products which we integrate without any adapters, so it’s nearly impossible to take the thing apart and provide a realistic simulated environment.” I think the underlying complaint is something like “we were told that there was an easy method to make our software correct, and it’s disappointing that it doesn’t work well without considerable investment into domain modeling, architectural work, mathematical analysis, etc.”