3 ms·

> I want solid verification, and a strong pattern to repeat verification no matter what. Everybody has a time budget for test execution time. I doubt that you

> I want solid verification, and a strong pattern to repeat verification no matter what. Everybody has a time budget for test execution time. I doubt that you will wait one year for test suite to finish. It is feasible to test all possible inputs for “int x+ int y”, but it is not feasible to test all inputs for usual GUI application. Trade offs have to be recognised, some balance must be struck. Speed is one of the factors that has to be considered.

shepherdjerred 2y ago
You don’t have to consider every possibility but you can randomly search the tree of every possible state.
- stoperaticless 2y ago
  One needs expected result defined for each of those. I doubt that is feasible.
  - shepherdjerred 2y ago
    You'd define invariants that must be met. This has been done before. https://en.wikipedia.org/wiki/Search-based_software_engineering https://en.wikipedia.org/wiki/Search-based_software_engineer... e.g. testing implementations of Paxos: https://github.com/emichael/dslabs https://github.com/emichael/dslabs At Facebook: https://engineering.fb.com/2018/05/02/developer-tools/sapienz-intelligent-automated-software-testing-at-scale/ https://engineering.fb.com/2018/05/02/developer-tools/sapien... As an example, your search graph might define your nodes as possible application states and the edges as actions to transition to a new state. Some edges might be "create a post" and "change post privacy settings". You might then define an invariant saying "after a user creates a post the new post is in their follower's timeline if privacy settings allow". Your search might eventually come across some set of actions that lead to this invariant being violated which is a discovered bug, e.g. if a user A creates a post -> user A changes settings to private -> user B can still see the post. Since the search space is infinite, the effectiveness is heavily influenced by how much time you can spend searching the tree & how fast you can evaluate states. This is a kind of exploratory test & wouldn't replace unit/integration tests, but it does let you catch certain kinds of bugs that would be impossible to find otherwise.

shepherdjerred 2y ago

You don’t have to consider every possibility but you can randomly search the tree of every possible state.

stoperaticless 2y ago
One needs expected result defined for each of those. I doubt that is feasible.
- shepherdjerred 2y ago
  You'd define invariants that must be met. This has been done before. https://en.wikipedia.org/wiki/Search-based_software_engineering https://en.wikipedia.org/wiki/Search-based_software_engineer... e.g. testing implementations of Paxos: https://github.com/emichael/dslabs https://github.com/emichael/dslabs At Facebook: https://engineering.fb.com/2018/05/02/developer-tools/sapienz-intelligent-automated-software-testing-at-scale/ https://engineering.fb.com/2018/05/02/developer-tools/sapien... As an example, your search graph might define your nodes as possible application states and the edges as actions to transition to a new state. Some edges might be "create a post" and "change post privacy settings". You might then define an invariant saying "after a user creates a post the new post is in their follower's timeline if privacy settings allow". Your search might eventually come across some set of actions that lead to this invariant being violated which is a discovered bug, e.g. if a user A creates a post -> user A changes settings to private -> user B can still see the post. Since the search space is infinite, the effectiveness is heavily influenced by how much time you can spend searching the tree & how fast you can evaluate states. This is a kind of exploratory test & wouldn't replace unit/integration tests, but it does let you catch certain kinds of bugs that would be impossible to find otherwise.