2 ms·

On a small scale I find this way of thinking useful in software development. Say that you are implementing a role-based access control system. And then ask yo

On a small scale I find this way of thinking useful in software development. Say that you are implementing a role-based access control system. And then ask yourself how can my RBAC system fail? Well one way it could fail is if someone without the necessary role can operate on something that they ought not have access to. So then create roles A, B, and C, and create a resource and say that group A can rw, and group B can r this resource. Then create a user and give that user role C. And then try to use this user in that role and try to read the resource. If it can, fail the test. Then another test where you try to write using such a user. If the resource was modified, fail the test. This is as opposed to only thinking about what you want to happen, in which case you might be writing tests that only ensure that those that should be able to read/write can do so. And of course you want to test that too. But ensuring that those that should not be able to access a resource cannot is the more important thing to be sure of, and also the type of thing that might slip by unnoticed. A system where those that should be able to access resources cannot, this will be detected in normal operation anyway, by way of users performing their usual tasks. But accessing resources you should not is the most critical and could go by unnoticed for a long time. And that is why the tests that are written in the backwards thinking fashion are the most important ones.

iandanforth 5y ago
I'm not sure "negative test cases" and "thinking about the problem in reverse" are the same thing. Thinking about larger systems is much more amenable to this approach. Taking your example if we extend RBAC to an auth/authz service we can say the following. I want my service to be unreliable. In what ways? I want it to be occasionally inaccessible or non-responsive, produce results that are non-deterministic and inaccurate. Taking one of those, how would I produce non-deterministic results? I'd make every operation tied to a PRNG, or be a function of something external to the system I don't have control over. Or maybe it could look non-deterministic if I make my logic dependent on time. How would flesh out an unreliable function based on time? I'd offer functionality proposed to tie RBAC to the local clock of the user. This way I'd have to deal with timezones, differences in user localities, relativity, leap years/seconds etc. Functions built on that would likely give the impression of being non-deterministic across many users. (Even assuming correct implementation). Ok what about being inaccessible vs non-responsive? The easiest thing to do is just shut the system down of course to maximize this objective, but we want the worst possible system so that's one that may or may not be there and may or may not respond. I could set up my load balancer to include nodes that don't exist, I could run jobs on the service boxes at regular intervals that consumed 100% CPU preventing their responses. I could ... ... And you can go on and on trying to find all the design and config choices that would make for a truly maddening service. Then you say OK from the product feature level down how can I avoid doing any of those things.

iandanforth 5y ago

I'm not sure "negative test cases" and "thinking about the problem in reverse" are the same thing. Thinking about larger systems is much more amenable to this approach. Taking your example if we extend RBAC to an auth/authz service we can say the following. I want my service to be unreliable. In what ways? I want it to be occasionally inaccessible or non-responsive, produce results that are non-deterministic and inaccurate. Taking one of those, how would I produce non-deterministic results? I'd make every operation tied to a PRNG, or be a function of something external to the system I don't have control over. Or maybe it could look non-deterministic if I make my logic dependent on time. How would flesh out an unreliable function based on time? I'd offer functionality proposed to tie RBAC to the local clock of the user. This way I'd have to deal with timezones, differences in user localities, relativity, leap years/seconds etc. Functions built on that would likely give the impression of being non-deterministic across many users. (Even assuming correct implementation). Ok what about being inaccessible vs non-responsive? The easiest thing to do is just shut the system down of course to maximize this objective, but we want the worst possible system so that's one that may or may not be there and may or may not respond. I could set up my load balancer to include nodes that don't exist, I could run jobs on the service boxes at regular intervals that consumed 100% CPU preventing their responses. I could ... ... And you can go on and on trying to find all the design and config choices that would make for a truly maddening service. Then you say OK from the product feature level down how can I avoid doing any of those things.