2 ms·

toil of process, which they tend to inflate and extend. I don’t know what to say about this. I remember this one dev who was super obsessed about code coverage

toil of process, which they tend to inflate and extend. I don’t know what to say about this. I remember this one dev who was super obsessed about code coverage (like, the actual coverage number - this detail is important). I don’t think this person even gave a shit about the tests, it just has to tick off that 90%+ test coverage metric which he was obsessed with, and the code needed to look exhaustive. I know this because that’s what his tests were exactly like. The tests were like the lamest tests ever, but they appeared comprehensive and exhaustive. It never really added much to the app, and it was just insane busy work. How do you tell a business this an absolute waste of money? You can’t because culturally code coverage is good. So I don’t even know who to blame here. This kind of shit happens constantly because we built some kinda tech culture that incentivizes it. We as an industry seem to have a really hard to time telling a 20 something (who may or may not be jacked up on stimulants) to settle the fuck down.

chachacharge 2y ago
Code coverage should be easy. if it's costing time, you're doing it wrong. if you're trying to apply it to an old project that did not use code coverage from the start: This is one way to do it wrong. you couldn't hold a gun to my head and make me do that. you can have a little bit of code coverage in this case, but not 85% - no. The moment it proves that it will be costing more time than it saves is the moment you can count me out.
- bloomingkales 2y ago
  No no, let me be clear. It’s a business, my natural instincts were “this is a waste of money even if it doesn’t take much work”. It’s a business! It’s valid feedback. This is not true for all businesses, but for our business it was true. How do you tell many many many developers that the reason they shouldn’t focus on certain things is because it’s a waste of money? This not a topic that comes up a lot because most developers are divorced from the bottom line. Anyway, this topic can head off in all kinds of directions.
  - chachacharge 2y ago
    well you said it was "insane busy work" and I do not see how that is possible unless applied incorrectly. 90% plus coverage is some smoke ,for example. I don't use code coverage on most of my projects though because the teams are just one or two people, and the projects only live a few years without ever being updated. The teams would have to be bigger the updates more frequent, and the project life span longer.
kazinator 2y ago
I thought of a good way to break down why obsession with code coverage is poor. The same program can be represented in many different ways. And code is data. We can take the program and compile it to a byte code. We then represent the byte code as an array of bytes which is bundled with the virtual machine to make the working program. The byte code uses every instruction of the virtual machine. And so, wee, we have 100% code coverage. Of course not 100% of the byte code array is executed by the VM, but pay no attention to that. Code coverage is about code, not data! The only coverage that matters is the coverage of the state space: hitting all the cases that can occur. There isn't a nice one-to-one correspondence between control flow paths and cases. A routine which adds to unsigned 16-bit integers has 65536 x 65536 cases, but nowhere near that many paths through its control graph.
almostnormal 2y ago
> [...] I remember this one dev who was super obsessed about code coverage [...] Oh, you are lucky. Where I work that's the person who defines how to test and which tools to use. Look at the code, create tests to paint everything green. Whether the code actually works does not matter. I don't think these tests have ever caught any kind of bug. That isn't even their purpose. > So I don’t even know who to blame here. Money. If tests can be written without understanding what the code is supposed to do, they can be written by people who do not understand. Which appears to be the cheapest way to get passing tests. Not to find any bugs, but that's not what the process requires.
Izkata 2y ago
> How do you tell a business this an absolute waste of money? You can’t because culturally code coverage is good. Theoretically yes, but practically not as much as it should be, because we've chosen a bad way to represent the metric. It should be an absolute number and flipped - we don't care about the covered lines, we care about the uncovered lines. Like instead of "90% of the code is covered by tests" it should be "234 lines aren't covered by any test". Bonus, this way a change that deletes covered code won't negatively affect the metric.

chachacharge 2y ago

Code coverage should be easy. if it's costing time, you're doing it wrong. if you're trying to apply it to an old project that did not use code coverage from the start: This is one way to do it wrong. you couldn't hold a gun to my head and make me do that. you can have a little bit of code coverage in this case, but not 85% - no. The moment it proves that it will be costing more time than it saves is the moment you can count me out.

bloomingkales 2y ago
No no, let me be clear. It’s a business, my natural instincts were “this is a waste of money even if it doesn’t take much work”. It’s a business! It’s valid feedback. This is not true for all businesses, but for our business it was true. How do you tell many many many developers that the reason they shouldn’t focus on certain things is because it’s a waste of money? This not a topic that comes up a lot because most developers are divorced from the bottom line. Anyway, this topic can head off in all kinds of directions.
- chachacharge 2y ago
  well you said it was "insane busy work" and I do not see how that is possible unless applied incorrectly. 90% plus coverage is some smoke ,for example. I don't use code coverage on most of my projects though because the teams are just one or two people, and the projects only live a few years without ever being updated. The teams would have to be bigger the updates more frequent, and the project life span longer.

kazinator 2y ago

I thought of a good way to break down why obsession with code coverage is poor. The same program can be represented in many different ways. And code is data. We can take the program and compile it to a byte code. We then represent the byte code as an array of bytes which is bundled with the virtual machine to make the working program. The byte code uses every instruction of the virtual machine. And so, wee, we have 100% code coverage. Of course not 100% of the byte code array is executed by the VM, but pay no attention to that. Code coverage is about code, not data! The only coverage that matters is the coverage of the state space: hitting all the cases that can occur. There isn't a nice one-to-one correspondence between control flow paths and cases. A routine which adds to unsigned 16-bit integers has 65536 x 65536 cases, but nowhere near that many paths through its control graph.

almostnormal 2y ago

> [...] I remember this one dev who was super obsessed about code coverage [...] Oh, you are lucky. Where I work that's the person who defines how to test and which tools to use. Look at the code, create tests to paint everything green. Whether the code actually works does not matter. I don't think these tests have ever caught any kind of bug. That isn't even their purpose. > So I don’t even know who to blame here. Money. If tests can be written without understanding what the code is supposed to do, they can be written by people who do not understand. Which appears to be the cheapest way to get passing tests. Not to find any bugs, but that's not what the process requires.

Izkata 2y ago

> How do you tell a business this an absolute waste of money? You can’t because culturally code coverage is good. Theoretically yes, but practically not as much as it should be, because we've chosen a bad way to represent the metric. It should be an absolute number and flipped - we don't care about the covered lines, we care about the uncovered lines. Like instead of "90% of the code is covered by tests" it should be "234 lines aren't covered by any test". Bonus, this way a change that deletes covered code won't negatively affect the metric.