4 ms·

I have two questions for y'all on this. I've heard it said for years that studies show the number of bugs grows roughly linearly with code size and that this h

by gruseom 15y ago

I have two questions for y'all on this. I've heard it said for years that studies show the number of bugs grows roughly linearly with code size and that this holds true across any programming language. (It's repeated, for example, at http://c2.com/cgi/wiki?LinesOfCode http://c2.com/cgi/wiki?LinesOfCode) I think I've even seen references to such studies, but I don't remember where. So, HN: what are these studies? Anybody know? (I mean besides the one referenced by the OP on class size. I believe this meme goes further back than that.) My second question is about how to measure code size. PG said a few years ago: why not just count tokens? I've thought about this ever since and I don't see what's wrong with it. Raw character count is obviously a lame metric, and LOC isn't much better. But token count seems like an apples-to-apples comparison that is easy to measure objectively and leaves out noise such as name length and whitespace. So: what's wrong with token count as a measurement of code size?

jcromartie 15y ago
In the Computer Language Benchmarks Game, they compare GZip'ed code size. I disagree with removing comments, since comments tend to be important, and comments generally document weirdness and bugs inherent in the platform/library/language. http://shootout.alioth.debian.org/help.php#gzbytes http://shootout.alioth.debian.org/help.php#gzbytes
watmough 15y ago
"I've heard it said for years that studies show the number of bugs grows roughly linearly with code size and that this holds true across any programming language." I have absolutely no reason to doubt this, but I suspect this does not look deeply enough at the process. Bugs, normally, get fixed, especially towards the end of a project, and it is much easier to eradicate bugs, and verify eradication, in a small project, than it is in a large one. Poor bug-fixing, in the late stages of a large, buggy project, may well introduce further bugs, as well as discovering latent bugs, masked by the original ones.
deleted 15y ago
[deleted]
jacques_chester 15y ago
> So, HN: what are these studies? If you have access to the publication libraries of the ACM and IEEE, you'll find that they publish most of this literature. (It costs money for both, unfortunately). > My second question is about how to measure code size. PG said a few years ago: why not just count tokens? Some schemes do so, it depends on how you define "SLOC". The classic problem is if-thens. How many lines is this? if ( foo ) then bar else baz Or this? if ( foo ) then bar else baz Or this? if ( foo ) then bar else baz In the literature you'll see a distinction between "physical" lines of code and "logical" lines of code. The latter is close to token-based. > So: what's wrong with token count as a measurement of code size? I vaguely recall that some properties correlate with physical lines and others with logical. I can't recall what and which, sorry.
- gruseom 15y ago
  If you have access to the publication libraries of the ACM and IEEE, you'll find that they publish most of this literature. I know where to find research literature. I'm asking for specific citations. Is the claim an urban legend? If "studies show" X, one ought to be able to point to the studies. How many lines is this? All three of your examples have the same number of tokens, so to judge by them alone, token count is not just a good measurement of code size, it's a perfect one. My question is what's wrong with it. I don't see how "logical lines", whatever that is, can possibly be simpler than counting tokens. In fact I don't see how anything can be simpler than counting tokens, since it's easy to know what it means, a tokenizer is always available, and everything irrelevant to the program is by definition dropped.
  - jacques_chester 15y ago
    > I'm asking for specific citations. None to hand right now. > I don't see how "logical lines", whatever that is, can possibly be simpler than counting tokens. I wasn't rejecting token counts per se. I think that it's a useful metric too. What I was trying to convey is that "logical lines" is the term used in the literature. Logical lines can cover token counts if you define 1 token = 1 logical line. Or it might not. Either way, you have to settle on a definition.
anon_d 15y ago
I actually think raw byte-count is a pretty good metric. Documentation size and variable name length are also symptoms of complexity. Specifically I think wc -c $(find -type f) (how many total bytes) and find -type f | wc -c (how many files and directories) make good metrics. Obviously you need to filter out data files and such. This has some problems (e.g. spaces vs. tabs, utf8, etc.), but all of these size metrics will be pretty loose.
- jacques_chester 15y ago
  One metric I've seen is gzip-compressed size, which has the nice property that it identifies the size of the incompressible elements -- ie it discounts repetitive boilerplate. Another interesting set of metrics is Halstead's "software science" metrics[1]. They fell out of favour because initially they were hard to count and didn't seem to correlate with anything else. [1] http://en.wikipedia.org/wiki/Halstead_complexity_measures http://en.wikipedia.org/wiki/Halstead_complexity_measures
  - gruseom 15y ago
    But repetitive boilerplate is exactly the last thing that should get away scot-free in a measurement of code size.
    - jacques_chester 15y ago
      It depends on what you want to know. Pure physical lines is one thing, "size" is a another.
      - gruseom 15y ago
        I want a way to measure how complicated a program is that's independent of language and obviously extraneous things like line length.
        jacques_chester 15y ago
cpeterso 15y ago
A common LOC metric for languages in the C family is to count semicolons ("real" lines of code).
- gruseom 15y ago
  I used to use that all the time. It's great because it's quick, and a pretty damn good sloppy metric. I remember joking that the extra semicolons in for loops and the lack of semicolons in if/while statements balance each other out. People tend not to use semicolons in comments, either, except in commented-out code (which some of us have a pet peeve against anyway).
  - alok-g 15y ago
    It's very easy to correct for both for loops and if/while.
    - gruseom 15y ago
      Yes, but then the metric is no longer so charmingly trivial.
tansey 15y ago
Here's one study showing that complexity grows exponentially with the size of the program: http://www.cc.gatech.edu/sparc/Resources/readings/berry.pdf http://www.cc.gatech.edu/sparc/Resources/readings/berry.pdf Here's a study showing addressing changed requirements or ﬁxing program defects requires a program maintenance effort that is directly proportional to the size of a program: http://dl.acm.org/citation.cfm?id=358228 http://dl.acm.org/citation.cfm?id=358228
- gruseom 15y ago
  Thanks, I'll take a look. This point from the abstract of the latter is highly reminiscent of the paper cited by the OP: "Repair maintenance is more highly correlated with the number of lines of source code in the program than it is to software science metrics." Edit: I read it. This must certainly be one of the studies people are referring to; it covers exactly the question of interest. The major limitation is that all the programs were in PL/I, so it says nothing about language-independence. The important finding is that line count was the most highly correlated variable with bug count of those studied, quite a bit more so than more complex metrics were (Halstead's E). It's also interesting that although the authors write, "There are some very large programs in this set," the largest program was in fact only a very modest (by our standards) 6572 lines of PL/I.
  - deleted 15y ago
    [deleted]
GregBuchholz 15y ago
Les Hatton has some of what you are looking for. See "Re-examining the fault density - component size connection": http://www.leshatton.org/IEEE_Soft_97b.html http://www.leshatton.org/IEEE_Soft_97b.html ...and some of his other work: http://www.leshatton.org/index_CS.html http://www.leshatton.org/index_CS.html
- gruseom 15y ago
  Thanks. I'll take a look at those too.
ljlolel 15y ago
The Mythical Man-Month
- gruseom 15y ago
  That's a marvelous book-length essay, but not a formal study. Or does Brooks cite research on this?

by gruseom 15y ago

jcromartie 15y ago

In the Computer Language Benchmarks Game, they compare GZip'ed code size. I disagree with removing comments, since comments tend to be important, and comments generally document weirdness and bugs inherent in the platform/library/language. http://shootout.alioth.debian.org/help.php#gzbytes http://shootout.alioth.debian.org/help.php#gzbytes

watmough 15y ago

"I've heard it said for years that studies show the number of bugs grows roughly linearly with code size and that this holds true across any programming language." I have absolutely no reason to doubt this, but I suspect this does not look deeply enough at the process. Bugs, normally, get fixed, especially towards the end of a project, and it is much easier to eradicate bugs, and verify eradication, in a small project, than it is in a large one. Poor bug-fixing, in the late stages of a large, buggy project, may well introduce further bugs, as well as discovering latent bugs, masked by the original ones.

deleted 15y ago

[deleted]

jacques_chester 15y ago

> So, HN: what are these studies? If you have access to the publication libraries of the ACM and IEEE, you'll find that they publish most of this literature. (It costs money for both, unfortunately). > My second question is about how to measure code size. PG said a few years ago: why not just count tokens? Some schemes do so, it depends on how you define "SLOC". The classic problem is if-thens. How many lines is this? if ( foo ) then bar else baz Or this? if ( foo ) then bar else baz Or this? if ( foo ) then bar else baz In the literature you'll see a distinction between "physical" lines of code and "logical" lines of code. The latter is close to token-based. > So: what's wrong with token count as a measurement of code size? I vaguely recall that some properties correlate with physical lines and others with logical. I can't recall what and which, sorry.

gruseom 15y ago
If you have access to the publication libraries of the ACM and IEEE, you'll find that they publish most of this literature. I know where to find research literature. I'm asking for specific citations. Is the claim an urban legend? If "studies show" X, one ought to be able to point to the studies. How many lines is this? All three of your examples have the same number of tokens, so to judge by them alone, token count is not just a good measurement of code size, it's a perfect one. My question is what's wrong with it. I don't see how "logical lines", whatever that is, can possibly be simpler than counting tokens. In fact I don't see how anything can be simpler than counting tokens, since it's easy to know what it means, a tokenizer is always available, and everything irrelevant to the program is by definition dropped.
- jacques_chester 15y ago
  > I'm asking for specific citations. None to hand right now. > I don't see how "logical lines", whatever that is, can possibly be simpler than counting tokens. I wasn't rejecting token counts per se. I think that it's a useful metric too. What I was trying to convey is that "logical lines" is the term used in the literature. Logical lines can cover token counts if you define 1 token = 1 logical line. Or it might not. Either way, you have to settle on a definition.

anon_d 15y ago

I actually think raw byte-count is a pretty good metric. Documentation size and variable name length are also symptoms of complexity. Specifically I think wc -c $(find -type f) (how many total bytes) and find -type f | wc -c (how many files and directories) make good metrics. Obviously you need to filter out data files and such. This has some problems (e.g. spaces vs. tabs, utf8, etc.), but all of these size metrics will be pretty loose.

jacques_chester 15y ago
One metric I've seen is gzip-compressed size, which has the nice property that it identifies the size of the incompressible elements -- ie it discounts repetitive boilerplate. Another interesting set of metrics is Halstead's "software science" metrics[1]. They fell out of favour because initially they were hard to count and didn't seem to correlate with anything else. [1] http://en.wikipedia.org/wiki/Halstead_complexity_measures http://en.wikipedia.org/wiki/Halstead_complexity_measures
- gruseom 15y ago
  But repetitive boilerplate is exactly the last thing that should get away scot-free in a measurement of code size.
  - jacques_chester 15y ago
    It depends on what you want to know. Pure physical lines is one thing, "size" is a another.
    - gruseom 15y ago
      I want a way to measure how complicated a program is that's independent of language and obviously extraneous things like line length.
      - jacques_chester 15y ago

cpeterso 15y ago

A common LOC metric for languages in the C family is to count semicolons ("real" lines of code).

gruseom 15y ago
I used to use that all the time. It's great because it's quick, and a pretty damn good sloppy metric. I remember joking that the extra semicolons in for loops and the lack of semicolons in if/while statements balance each other out. People tend not to use semicolons in comments, either, except in commented-out code (which some of us have a pet peeve against anyway).
- alok-g 15y ago
  It's very easy to correct for both for loops and if/while.
  - gruseom 15y ago
    Yes, but then the metric is no longer so charmingly trivial.

tansey 15y ago

Here's one study showing that complexity grows exponentially with the size of the program: http://www.cc.gatech.edu/sparc/Resources/readings/berry.pdf http://www.cc.gatech.edu/sparc/Resources/readings/berry.pdf Here's a study showing addressing changed requirements or ﬁxing program defects requires a program maintenance effort that is directly proportional to the size of a program: http://dl.acm.org/citation.cfm?id=358228 http://dl.acm.org/citation.cfm?id=358228

gruseom 15y ago
Thanks, I'll take a look. This point from the abstract of the latter is highly reminiscent of the paper cited by the OP: "Repair maintenance is more highly correlated with the number of lines of source code in the program than it is to software science metrics." Edit: I read it. This must certainly be one of the studies people are referring to; it covers exactly the question of interest. The major limitation is that all the programs were in PL/I, so it says nothing about language-independence. The important finding is that line count was the most highly correlated variable with bug count of those studied, quite a bit more so than more complex metrics were (Halstead's E). It's also interesting that although the authors write, "There are some very large programs in this set," the largest program was in fact only a very modest (by our standards) 6572 lines of PL/I.
- deleted 15y ago
  [deleted]

GregBuchholz 15y ago

Les Hatton has some of what you are looking for. See "Re-examining the fault density - component size connection": http://www.leshatton.org/IEEE_Soft_97b.html http://www.leshatton.org/IEEE_Soft_97b.html ...and some of his other work: http://www.leshatton.org/index_CS.html http://www.leshatton.org/index_CS.html

gruseom 15y ago
Thanks. I'll take a look at those too.

ljlolel 15y ago

The Mythical Man-Month

gruseom 15y ago
That's a marvelous book-length essay, but not a formal study. Or does Brooks cite research on this?