4 ms·

> What are your thoughts on automated/unit tests used to guarantee expectations do not change in future revisions (a regression suite) and also used to present

> What are your thoughts on automated/unit tests used to guarantee expectations do not change in future revisions (a regression suite) and also used to present example inputs usages of the API that was designed? To be honest, this is one idea I was toying around with. As in, have a test suite that generates a "state" file for your program and then, in any given patch, list the items in the state that you'd expect to change. That way, one can basically catch a lot of "bugs" which often boil down to "this change I made here to affect X is also unexpectedly affecting Y". My main problem with this are: 1. I see nobody doing this, and I'm not sure why. 2. The tooling for this doesn't really seem to exist, and I'm not sure how easy one could bring it into existence and/or if a generic version of it could be written. 3. This breaks down with a lot of software where a tiny change can affect, well, everything (e.g. modifying a random seed or changing an error-checking constant)

t-writescode 6y ago
> As in, have a test suite that generates a "state" file for your program What do you mean here? Do you mean you would run a specific function with specific inputs and record specific outputs and compare them?
- george3d6 6y ago
  Yes, or rather, run your whole program with specific inputs then (e.g. by using a debugger) capture as much of it's state as you can and compare that with the state from the next run of those same inputs. E.g. given a program with variables a1 = array a2 = int a3 = string Running might get the state flow: a1 = [] a2 = 10 a3 = 'abc' a3 = 'dd1' a2 = 200 a1 = [200,'dd1'] Then, if a change is made, the programmer can say "I expect this change to only affect the state of a2" and if you get the state flow: a1 = [] a2 = 11 a3 = 'abc' a3 = 'dd1' a2 = 500 a1 = [200,'dd1'] Then the test passes If you get the state flow: a1 = [] a2 = 11 a3 = 'abc' a3 = 'dd1' a2 = 500 a1 = [500,'dd1'] The test fails with error "The change you expected to only affect a2 also had an effect upon a1" But again, the actual state you capture here and the way you check for equality (e.g. to you check for equality among all the states during the whole runtime, or just in the final state of the program ? If the former how is state-change ordering handled ? If the later, what about relevant state that is out of scope by the time the program finishes running ?)
  - t-writescode 6y ago
    How is this different from a set of unit tests where you give the function under test each of those inputs and have the expected output be the same for each entry as the given list of outputs? e.g. [TestCase(null, null)] [TestCase(10, 11)] [TestCase('abc', 'abc')] [TestCase('dd1', 'dd1')] [TestCase(500, 500)] [TestCase([200, 'dd1'], [200, 'dd1'])] public void StateChanges(input, expected) { actual = DoThing(input); Assert.AreEqual(actual, expected); }
    - exdsq 6y ago
      Because this is only checking the individual function, not the entire systems state. If I have a function that adds two ints I could test it with [TestCase([2, 2], 4)] and this would pass but if it also changed the value of an object there could be an issue a unit test (and indeed integration/e2e test) wouldn't catch.
bdavis__ 6y ago
The problem is what do you do when you change the code? System state is different; how do you know it is correct? Also, un-tenable with any code that is non-deterministic, like multiple threads or external I/O.
mdoms 6y ago
> To be honest, this is one idea I was toying around with. As in, have a test suite that generates a "state" file for your program and then, in any given patch, list the items in the state that you'd expect to change. It is very clear that you have not worked on large, complex systems.

t-writescode 6y ago

> As in, have a test suite that generates a "state" file for your program What do you mean here? Do you mean you would run a specific function with specific inputs and record specific outputs and compare them?

george3d6 6y ago
Yes, or rather, run your whole program with specific inputs then (e.g. by using a debugger) capture as much of it's state as you can and compare that with the state from the next run of those same inputs. E.g. given a program with variables a1 = array a2 = int a3 = string Running might get the state flow: a1 = [] a2 = 10 a3 = 'abc' a3 = 'dd1' a2 = 200 a1 = [200,'dd1'] Then, if a change is made, the programmer can say "I expect this change to only affect the state of a2" and if you get the state flow: a1 = [] a2 = 11 a3 = 'abc' a3 = 'dd1' a2 = 500 a1 = [200,'dd1'] Then the test passes If you get the state flow: a1 = [] a2 = 11 a3 = 'abc' a3 = 'dd1' a2 = 500 a1 = [500,'dd1'] The test fails with error "The change you expected to only affect a2 also had an effect upon a1" But again, the actual state you capture here and the way you check for equality (e.g. to you check for equality among all the states during the whole runtime, or just in the final state of the program ? If the former how is state-change ordering handled ? If the later, what about relevant state that is out of scope by the time the program finishes running ?)
- t-writescode 6y ago
  How is this different from a set of unit tests where you give the function under test each of those inputs and have the expected output be the same for each entry as the given list of outputs? e.g. [TestCase(null, null)] [TestCase(10, 11)] [TestCase('abc', 'abc')] [TestCase('dd1', 'dd1')] [TestCase(500, 500)] [TestCase([200, 'dd1'], [200, 'dd1'])] public void StateChanges(input, expected) { actual = DoThing(input); Assert.AreEqual(actual, expected); }
  - exdsq 6y ago
    Because this is only checking the individual function, not the entire systems state. If I have a function that adds two ints I could test it with [TestCase([2, 2], 4)] and this would pass but if it also changed the value of an object there could be an issue a unit test (and indeed integration/e2e test) wouldn't catch.

bdavis__ 6y ago

The problem is what do you do when you change the code? System state is different; how do you know it is correct? Also, un-tenable with any code that is non-deterministic, like multiple threads or external I/O.

mdoms 6y ago

> To be honest, this is one idea I was toying around with. As in, have a test suite that generates a "state" file for your program and then, in any given patch, list the items in the state that you'd expect to change. It is very clear that you have not worked on large, complex systems.