3 ms·

In corporate app development, I would see tests to check that the mocks return the expected values. Like, what are we even doing here?

anilakar 8mo ago
Someone was asked to test untestable code so verifying mock contents was the best they could come up with.
- mvpmvh 8mo ago
  No. Someone was asked to meet an arbitrary code coverage threshold. I'm dealing with this malicious compliance/weaponized incompetence at $current_job
  - dwoldrich 8mo ago
    How will you deal with it? I successfully convinced $big_important_group at $day_job to not implement a policy of failing their builds when code coverage dips below their target threshold > 90%. (Insane target, but that's a different conversation.) I convinced them that if they wanted to treat uncovered lines of code as tech debt, they needed to add an epic stories to their backlog to write tests. And their artificially setting some high target coverage threshold will produce garbage because developers will write do-nothing tests in order to get their work done and not trip the alarms. I argued that failing the builds on code coverage would be unfair because the tech debt created by past developers would unfairly hinder random current-day devs getting their work done. Instead, I recommended they pick their current coverage percentage (it was < 10% at the time) and set the threshold to that simply to prevent backsliding as new code was added. Then, as their backlogged, legit tests were implemented, ratchet up the coverage threshold to the new high water mark. This meant all new code would get tests written for them. And, instead of failing builds, I recommended email blasts to the whole team to indicate there was some recent backsliding in the testing regime and the codebase had grown without accompanying tests. It was not a huge shame event, but good a motivator to the team to keep up the quality. SonarQube was great for long-term tracking of coverage stats. Finally, I argued the coverage tool needed to have very liberal "ignore" rules that were agreed to by all members of the team (including managers). Anything that did not represent testable logic written by the team: generated code, configurations, tests themselves, should not count against their code coverage percentages.
RobRivera 8mo ago
I'm trying to wrap my head around here. So there are tests that leverage mocks. Those mocks help validate software is performing as desired by enabling tests to see the software behaves as desired in varying contexts. If the software fails, it is because the mocks exposed that under certain inputs, undesired behavior occurs, an assert fails, and a red line flags the test output. Validating that the mocks return the desired output.... Maybe there is a desire that the mocks return a stream of random numbers and the mock validation tests asserts said stream adheres to a particular distribution? Maybe someone in the past pushed a bad mock into prod, that mock validated a test that would have failed given better mock, and a post mortem when the bad software, now pushed into prod, was traced to a bad mock derived a requirement that all mocks must be validated?
- bensyverson 8mo ago
  Yeah, seems plausible, or it was just "belt and suspenders." Sure made a lot of pretty green checkmarks.
bdangubic 8mo ago
we use this https://github.com/auchenberg/volkswagen https://github.com/auchenberg/volkswagen
steve_adams_86 8mo ago
# abstract internals for no reason func doThing(x: bool) if (x) return true else return false # make sure our logic works as expected assert(doThing(true)) # ??? # profit It's excellent software engineering because there are tests
dwoldrich 8mo ago
You could ask the same thing about tests themselves. And I'm not talking about tests that don't exercise the code in a meaningful manner like your assertions on mocks(?!) I'm saying you could make the same argument about useful tests themselves. What is testing that the tests are correct? Uncle Bob would say the production code is testing the tests but only in the limited, one-time, acceptance case where the programmer who watches the test fail, implements code, and then watches it pass (in the ideal test-driven development scenario.) But what we do all boils down to acceptance. A human user or stakeholder continues to accept the code as correct equals a job well done. Of course, this is itself a flawed check because humans are flawed and miss things and they don't know what they want anyhow. The Agile Manifesto and Extreme Programming was all about organizing to make course corrections as cheap as possible to accommodate fickle humanity. > Like, what are we even doing here? What ARE we doing? A slapdash job on the whole. And, AI is just making slapdash more acceptable and accepted because it is so clever and the boards of directors are busy running this next latest craze into the dirt. "Baffle 'em with bullsh*t" works in every sector of life and lets people get away with all manner of sins. I think what we SHOULD be doing is plying our craft. We should be using AI as a thinking tool, and not treat it like a replacement for ourselves and our thinking.

anilakar 8mo ago

Someone was asked to test untestable code so verifying mock contents was the best they could come up with.

mvpmvh 8mo ago
No. Someone was asked to meet an arbitrary code coverage threshold. I'm dealing with this malicious compliance/weaponized incompetence at $current_job
- dwoldrich 8mo ago
  How will you deal with it? I successfully convinced $big_important_group at $day_job to not implement a policy of failing their builds when code coverage dips below their target threshold > 90%. (Insane target, but that's a different conversation.) I convinced them that if they wanted to treat uncovered lines of code as tech debt, they needed to add an epic stories to their backlog to write tests. And their artificially setting some high target coverage threshold will produce garbage because developers will write do-nothing tests in order to get their work done and not trip the alarms. I argued that failing the builds on code coverage would be unfair because the tech debt created by past developers would unfairly hinder random current-day devs getting their work done. Instead, I recommended they pick their current coverage percentage (it was < 10% at the time) and set the threshold to that simply to prevent backsliding as new code was added. Then, as their backlogged, legit tests were implemented, ratchet up the coverage threshold to the new high water mark. This meant all new code would get tests written for them. And, instead of failing builds, I recommended email blasts to the whole team to indicate there was some recent backsliding in the testing regime and the codebase had grown without accompanying tests. It was not a huge shame event, but good a motivator to the team to keep up the quality. SonarQube was great for long-term tracking of coverage stats. Finally, I argued the coverage tool needed to have very liberal "ignore" rules that were agreed to by all members of the team (including managers). Anything that did not represent testable logic written by the team: generated code, configurations, tests themselves, should not count against their code coverage percentages.

RobRivera 8mo ago

I'm trying to wrap my head around here. So there are tests that leverage mocks. Those mocks help validate software is performing as desired by enabling tests to see the software behaves as desired in varying contexts. If the software fails, it is because the mocks exposed that under certain inputs, undesired behavior occurs, an assert fails, and a red line flags the test output. Validating that the mocks return the desired output.... Maybe there is a desire that the mocks return a stream of random numbers and the mock validation tests asserts said stream adheres to a particular distribution? Maybe someone in the past pushed a bad mock into prod, that mock validated a test that would have failed given better mock, and a post mortem when the bad software, now pushed into prod, was traced to a bad mock derived a requirement that all mocks must be validated?

bensyverson 8mo ago
Yeah, seems plausible, or it was just "belt and suspenders." Sure made a lot of pretty green checkmarks.

bdangubic 8mo ago

we use this https://github.com/auchenberg/volkswagen https://github.com/auchenberg/volkswagen

steve_adams_86 8mo ago

# abstract internals for no reason func doThing(x: bool) if (x) return true else return false # make sure our logic works as expected assert(doThing(true)) # ??? # profit It's excellent software engineering because there are tests

dwoldrich 8mo ago

You could ask the same thing about tests themselves. And I'm not talking about tests that don't exercise the code in a meaningful manner like your assertions on mocks(?!) I'm saying you could make the same argument about useful tests themselves. What is testing that the tests are correct? Uncle Bob would say the production code is testing the tests but only in the limited, one-time, acceptance case where the programmer who watches the test fail, implements code, and then watches it pass (in the ideal test-driven development scenario.) But what we do all boils down to acceptance. A human user or stakeholder continues to accept the code as correct equals a job well done. Of course, this is itself a flawed check because humans are flawed and miss things and they don't know what they want anyhow. The Agile Manifesto and Extreme Programming was all about organizing to make course corrections as cheap as possible to accommodate fickle humanity. > Like, what are we even doing here? What ARE we doing? A slapdash job on the whole. And, AI is just making slapdash more acceptable and accepted because it is so clever and the boards of directors are busy running this next latest craze into the dirt. "Baffle 'em with bullsh*t" works in every sector of life and lets people get away with all manner of sins. I think what we SHOULD be doing is plying our craft. We should be using AI as a thinking tool, and not treat it like a replacement for ourselves and our thinking.