4 ms·

One issue with software methodology assessment is the difficulty in getting consensus on ways to determine code quality. One potential experiment I've been thi

by keyist 16y ago

One issue with software methodology assessment is the difficulty in getting consensus on ways to determine code quality. One potential experiment I've been thinking of to work around this is by seeing if 3rd parties can tell if a methodology was used. To abuse OP's analogy of debug statements, this is essentially a "assert this practice makes a visible difference". Have Alice and Bob sporadically use practice X on a group of changesets and track the time spent on each. Eve and other coworkers then go through the resulting list and try to determine which changesets used practice X. There will need to be practice-specific variations of course. For example if experimenting for TDD, the changesets obviously need to include tests and code in the same commit (without indicating which came first). If experimenting for pairing, should probably report the time elapsed as the pair worked on it, instead of summing the time spent by both developers. Just thought I'd throw this idea out here where it's somewhat on-topic and see what y'all think about its feasibility. If anyone tries this out I'd love to hear the results!

ezy 16y ago
I think code quality misses the issue. The right metric (IMO) is actually fairly easy: bug count perhaps normalized by project size or reach (number of customers). Whether the software does what it is advertised to do is the very definition of software quality. There is nothing more or less than that. Code quality is obviously a component of software quality, but it's not the only one. In fact, there's no study out there right now (that I'm aware of) that says messy code leads to bad software quality -- just a hypothesis and lots of anecdotes. I would think that it probably is true, but it's nowhere near self-evident.
- nradov 16y ago
  Bug count is a poor metric for this. Not all bugs are created equal; some have orders of magnitude more impact than others. And then there's the philosophical question about what is or isn't a "bug". Most software isn't advertised with detailed functional and non-functional requirements covering every possible use case. So if the software doesn't do something the way a user wants it to is that a bug, missing feature, or simply working as designed?
  - ezy 16y ago
    Ok, you're taking "advertise" too literally. What I meant was that software typically is thrown out to the world to serve a certain function, I wasn't making a comment on advertising materials :-). I see what you're saying, but when you build something you typically have an idea of how it'll work, even if you don't have a precise specification. My view is that if the software is of good quality, it will work like that. If it's of bad quality, it'll sort kinds work some of the time like you expected. :-) In general, saying that certain bugs aren't really is usually a cop-out (not saying you're coping out, just that this is what I hear as a typical excuse). In industry, you're usually quickly taking feature requests out of the "bug" category very quickly, and "working as designed" is usually the result of misunderstandings, not software failure -- and is also quickly pushed out of the bug category. Anything that pings back and forth a lot is probably a result of bad feature planning, which, again, reflects on the software quality.
    - bmm6o 16y ago
      It's not a cop-out at all. The only way to precisely define "bug" in your original comment is as behavior that deviates from the specification - you need a formal definition if you want to count them. You're right that if the user says "when I click 'save', your program formats my hard drive", "yes, that is in the spec" is not really a defensible response. The user doesn't care that this is a flaw in the specification and not the implementation. But there is a huge middle ground where the specification and the user's expectations disagree (since the user didn't read the spec), and both are reasonable. Counts as a "bug" or not?
      - ezy 16y ago
        A "bug" is simply a reported issue that the developers feel must be fixed at some time in the future. It is defined just by existing in the database, and not being deferred or reclassified as not a bug. That's what I mean by bug count. It's totally unambiguous because the criteria is simply that the developers in question on a particular project perceive it as a problem that needs to be fixed. Like I said, I think this is basically the definition of software quality -- does it do what it was supposed to do. Of course, you can cheat it by having people be completely dishonest (or incompetent) about filing or classifying bugs, but that comes with the territory. We have to rely on people telling the truth and being somewhat competent in other studies as well -- there is no perfect control. So, I stand by my assertion that trying to define a fuzzy area for bugs is a cop-out, because if it's actually fuzzy then it will be reclassified by the developers in question most of the time. Consider an analogy that's driving my thoughts on this subject: If you have some physical product you're producing -- if issues come up that make the designers, manufacturers think about rebuilding/modifying it to address the issues, aren't those issue a problem with the quality of the original product?
- flogic 16y ago
  The only real measure is benefit - cost. Sometimes quality costs more than bugs. Take a simple editor macro. You record some simple set of actions and have the editor apply them en-mass. They may not be correct, but just getting the job done likely outweighs the faults.

by keyist 16y ago

ezy 16y ago

I think code quality misses the issue. The right metric (IMO) is actually fairly easy: bug count perhaps normalized by project size or reach (number of customers). Whether the software does what it is advertised to do is the very definition of software quality. There is nothing more or less than that. Code quality is obviously a component of software quality, but it's not the only one. In fact, there's no study out there right now (that I'm aware of) that says messy code leads to bad software quality -- just a hypothesis and lots of anecdotes. I would think that it probably is true, but it's nowhere near self-evident.

nradov 16y ago
Bug count is a poor metric for this. Not all bugs are created equal; some have orders of magnitude more impact than others. And then there's the philosophical question about what is or isn't a "bug". Most software isn't advertised with detailed functional and non-functional requirements covering every possible use case. So if the software doesn't do something the way a user wants it to is that a bug, missing feature, or simply working as designed?
- ezy 16y ago
  Ok, you're taking "advertise" too literally. What I meant was that software typically is thrown out to the world to serve a certain function, I wasn't making a comment on advertising materials :-). I see what you're saying, but when you build something you typically have an idea of how it'll work, even if you don't have a precise specification. My view is that if the software is of good quality, it will work like that. If it's of bad quality, it'll sort kinds work some of the time like you expected. :-) In general, saying that certain bugs aren't really is usually a cop-out (not saying you're coping out, just that this is what I hear as a typical excuse). In industry, you're usually quickly taking feature requests out of the "bug" category very quickly, and "working as designed" is usually the result of misunderstandings, not software failure -- and is also quickly pushed out of the bug category. Anything that pings back and forth a lot is probably a result of bad feature planning, which, again, reflects on the software quality.
  - bmm6o 16y ago
    It's not a cop-out at all. The only way to precisely define "bug" in your original comment is as behavior that deviates from the specification - you need a formal definition if you want to count them. You're right that if the user says "when I click 'save', your program formats my hard drive", "yes, that is in the spec" is not really a defensible response. The user doesn't care that this is a flaw in the specification and not the implementation. But there is a huge middle ground where the specification and the user's expectations disagree (since the user didn't read the spec), and both are reasonable. Counts as a "bug" or not?
    - ezy 16y ago
      A "bug" is simply a reported issue that the developers feel must be fixed at some time in the future. It is defined just by existing in the database, and not being deferred or reclassified as not a bug. That's what I mean by bug count. It's totally unambiguous because the criteria is simply that the developers in question on a particular project perceive it as a problem that needs to be fixed. Like I said, I think this is basically the definition of software quality -- does it do what it was supposed to do. Of course, you can cheat it by having people be completely dishonest (or incompetent) about filing or classifying bugs, but that comes with the territory. We have to rely on people telling the truth and being somewhat competent in other studies as well -- there is no perfect control. So, I stand by my assertion that trying to define a fuzzy area for bugs is a cop-out, because if it's actually fuzzy then it will be reclassified by the developers in question most of the time. Consider an analogy that's driving my thoughts on this subject: If you have some physical product you're producing -- if issues come up that make the designers, manufacturers think about rebuilding/modifying it to address the issues, aren't those issue a problem with the quality of the original product?
flogic 16y ago
The only real measure is benefit - cost. Sometimes quality costs more than bugs. Take a simple editor macro. You record some simple set of actions and have the editor apply them en-mass. They may not be correct, but just getting the job done likely outweighs the faults.