9 ms·

The “Bug-O” Notation

203 points by danabramov 8y ago

qlk1123 8y ago
This is a good attempt to kickstart the discussion of some quantitative metric about bugs and the time one needs to deal with it, but I wonder if there wasn't any software engineering theories or practices to address this. If anyone know other related topics, please share them, thanks!
- jakear 8y ago
  This is somewhat similar to the concept of “Cyclomatic Complexity”, which is the number of independent paths though a piece of code, and can be statically analyzed. https://en.m.wikipedia.org/wiki/Cyclomatic_complexity https://en.m.wikipedia.org/wiki/Cyclomatic_complexity
  - danabramov 8y ago
    Note one difference is that I'm talking about characteristics of an API, not particular code snippet. Although it's relevant.
  - amelius 8y ago
    This makes me wonder, are software metrics still a thing?
    - rbanffy 8y ago
      Yes. At least for me.
  - jerf 8y ago
    I like this concept better than cyclomatic complexity, precisely because this is what cyclomatic complexity is trying to poorly approximate. But it can't be automatically derived in the general case, or even terribly easily in non-general cases, either, unlike cyclomatic complexity. I use a descendant of gometalinter on all my Go code, and one of the things it ships with is cyclomatic complexity, and I always turn it off. Precisely because I tend to think like this, it often gives my code terribly wrong ratings because it can't see that there's actually only a finite set of paths through the code that are possible, and also as the blog post demonstrates, sometimes adding more code actually simplifies the conceptual flow. But cyclomatic complexity will generally say the "more code" is even more cyclomatically complex. There's also some Go-specific elements to my distaste for the measure, though; since in Go right now handling an error is automatically an if statement, that'll hit you right in the metrics. But in many cases, something, err := GetSomething() if err != nil { return errwrap.Wrapf("while trying to get something: {{err}}", err) } Officially that's an if statement; unofficially, it ought to be a cyclomatic complexity of zero. Now, it isn't technically a free if clause, in the sense that a bug could live there, but if you're going to count the exception-based equivalent as zero (and it has the same callout; bugs can lie in your exception handling exactly the same way), then this ought to be practically zero too. Consequently, Go functions tend to get smacked with much higher complexity numbers than exception-equivalent code. Yeah, it's probably not a one-to-one, but it's certainly not as lopsided as the metric makes it seem.
    - sigil 8y ago
      Does gometalinter report absolute cyclomatic complexity numbers, or does it report density (cyclomatic complexity / sloc)?
- l0b0 8y ago
  I've found TDD to be a good (but not perfect, as some people complain) way to keep complexity down - when you force yourself to test every state you very quickly start focussing on how to keep the number of possible states down.
  - AllegedAlec 8y ago
    I think that a more important reason is that TDD forces you to write code in small modules which can be tested independently. Because of this, you have small to medium size modules to reason about, rather than one huge class where you wouldn't know where to start.
- jtolmar 8y ago
  There are various studies of defect density (bugs per 1000 lines of code) versus whatever factor someone thought would influence bugs. "Software defect density" seem to be the magic search terms for finding actual studies instead of blog posts. I unfortunately can't find it, but I remember there being a study (by Microsoft?) of defect density for projects with different methodologies (scrum, TDD, whatever), which found lines of code is more correlated with defect rate than everything else they tried. They took that as a failure; I've always taken that to mean that reducing lines of code is the most important first step (or probably more accurately reducing tokens of executable code; one liners don't help and type annotations don't hurt). While attempting to find that study, I found a study [1] that claims to have found an empirical sweet spot in defect density by module size - apparently 400 lines for assembly modules or 200 lines for ADA modules. Those are some weird languages to have numbers on, but maybe something else in that paper's family tree has something more relatable. [1] http://www.cs.colostate.edu/~malaiya/p/denton_2000.pdf http://www.cs.colostate.edu/~malaiya/p/denton_2000.pdf
  - beaconstudios 8y ago
    even reducing tokens isn't a very good measure - lines of code is a proxy for complexity of implementation. I think a good measure (albeit a heuristic one) is how concise the implementation is relative to the complexity of the problem - or in other words, the ratio of accidental to essential complexity.
- biggio 8y ago
  It's a broad topic and called "code metrics" or "software metrics".
jonahx 8y ago
> The problem with this code isn’t that it’s “ugly”. We’re not talking about aesthetics. The problem is that if there is a bug in this code, I don’t know where to start looking. But this is precisely what people mean when they say it's ugly. "Ugly" is what this gets translated as by your limbic system which registers in an instant: "fuck this big mess and the work it's going to be to reason about." Aesthetics and usability are deeply entangled here, and the problem with that code is absolutely an aesthetic one. I want to say, more so. Because I experience it that way first, and only later do I enumerate the measurable reasons it's likely to be more buggy because of it.
- alephr 8y ago
  the problem is that "ugly" isn't precise and can get translated from other reactions that aren't based on experience but are instead just cargo cutting. it's great to use the faster parts of our brains to reason but still important to make sure we're not accepting the mistakes they might make.
  - stupidcar 8y ago
    It's also an emotive word that can make discussion more heated. If you're reviewing somebody's code and you call it "ugly", they're liable to get upset, and it's too imprecise for them to act on. It's far more helpful and less combative to explain, in concrete terms, what you think its problems are, and how a different approach could solve them.
- rzwitserloot 8y ago
  Ugly, given that it is an emotional word, applies to any code that you don't like. WHY do you not like some code? To avoid pointless discussions about taste and gut feelings, I'd bring it down to falsifiable statements. "This is hard to maintain, for reasons X and Y" is such a statement. As is: "This is hard to test for reason: It needlessly engages in combinatorial explosion" is another. Yes, you could shorten that to 'ugly'. Good for you. Nevertheless, I've seen tons of discussions referring to 'ugly' code where the reasons were solid but completely different from maintainability (for example: "It is longer than needed", "It does is not idiomatic for this language"), or even not particularly solid ("It is ugly because it isn't functional", "It is ugly because the style guide says all APIs should always be interfaces", "It is ugly because I see tabs, not spaces").
PezzaDev 8y ago
After seeing callbacks cause a lot of "hard to debug" scenarios for me and for others, I'm starting to believe they are gotos in disguise.
- danabramov 8y ago
  They are.
- pjc50 8y ago
  Even worse: it's a "come from", a statement normally only seen in INTERCAL.
  - rbanffy 8y ago
    At least "come from" states clearly where the code is coming from. Now... When your callbacks are names and not inline you can't really make that assumption.
lioeters 8y ago
Otherwise known as cyclomatic complexity?
- danabramov 8y ago
  Cyclomatic complexity refers to a specific code snippet. I am talking about inherent characteristics of a particular API and how the patterns it encourages influences debugging. For example: >In fact, if any value looks wrong in the DOM of a React app, you can trace where it comes from by looking at the code of components above it in the React tree one by one. No matter the app size, tracing a rendered value is (tree height).
  - beatgammit 8y ago
    Really, all of this can be pared down to: reduce mutable state. That's what React does, and it's why code written in functional languages like Haskell tend to be much easier to reason about. Basically, the fewer things you need to know about in a given context, the less likely you are too mess it up.
titanix2 8y ago
Funny how the post is about bug complexity, says that "Some API and language designs make a whole class of mistakes impossible. Some create endless problems." and then proceed to an example in Javascript... for which a whole language (Typescript) was invented precisely because it does not scales well.
- danabramov 8y ago
  My audience is JavaScript developers.
- riskable 8y ago
  I was going to say something similar but then I thought about it a moment... If I were going to write an article about maddening complexity in code that seems like it should be straightforward I too would choose JavaScript for my examples!
8lall0 8y ago
tl;dr avoid too much nesting, prevent DRY and make small but useful functions. An advice that is valid since forever.
- danabramov 8y ago
  You can split the first example in small functions and it'll still be a mess. The post is not about duplication, function size, or nesting. We gained predictability by preventing _accumulation_ of errors — which we got by always restarting rendering from scratch.
  - dyarosla 8y ago
    Well- the React example doesn’t really restart rendering from scratch (as you yourself mentioned). And in larger applications you’d still prefer to not ‘redraw from scratch’ in React either and build on top of a previous setup. Really what you’re doing is preferring more declarative programming practices over procedural. And as a consequence, more declarative APIs like React.
    - danabramov 8y ago
      For the app developer, React code _does_ restart rendering from scratch. You have no access to the past rendered state from the render method (at least if you don’t intentionally escape the declarative model). This isn’t true only for bugs in React. What happens internally in React is another question. But the point of React abstraction is to let _you_ think in terms of “render from scratch”.
  - deleted 8y ago
    [deleted]
  - k__ 8y ago
    Funny. I tried to write software like that 10 years ago already, but I always failed at performance. React was just what I was missing.
- yen223 8y ago
userbinator 8y ago
I'm not familiar with the language, and yet I remain unconvinced because I could still understand what the code in the first example is doing, and more importantly, see it all at a glance. I don't have to jump around between functions to follow the logic. More verbose code means more opportunity for errors and more code to look through and debug. That said, I'd probably write something closer to this (once again, not sure if this is even valid, but you can just treat it as pseudocode): // don't create these every time, we're just changing their visibility let retryButton = createRetryButton(); // calls trySubmit() when clicked... let successMessage = createSuccessMessage(); let spinner = createSpinner(); let errorMessage = createErrorMessage(); ... function trySubmit() { // assume these don't cause errors if the child isn't already there formStatus.removeChild(errorMessage); formStatus.removeChild(retryButton); formStatus.appendChild(spinner); succ = submitForm(); formStatus.removeChild(spinner); if(succ) formStatus.appendChild(successMessage); else { errorMessage.setMessage(error); formStatus.appendChild(errorMessage); formStatus.appendChild(retryButton) } }
- danabramov 8y ago
  I’m not saying the first example is hard to _understand_. I’m saying that it’s hard to be confident it works or debug when it has a bad visual output because the unnecessary statefulness creates compounding mistakes over time. My second example (that doesn’t have this problem) is pretty much what you wrote.
- jerf 8y ago
  "yet I remain unconvinced because I could still understand what the code in the first example is doing, and more importantly, see it all at a glance." This isn't because it's good code. It's because it's code that fits in a blog post, so of course you understand it quickly. If you didn't understand it quickly, the author's point couldn't be conveyed properly. The question isn't really what happens when you have 10 lines of dodgy code. The question is, what happens when you have 10,000 lines of dodgy code? (In a classroom, on a chalkboard, bubble sort isn't any slower than any other sort, and somewhat easier to draw, honestly. Quicksort is rather a lot of stuff to draw. But unless your code solely deals with lists small enough to fit comfortably on a blackboard with work room to spare, you are well advised not to use bubblesort.)
pjc50 8y ago
So what I think this points to is the importance of what I'm going to call "psychological reversibility" in programming: the ability for the developer (with the aid of their tools) to answer three kinds of question: - how did this (wrong) control flow get here? Normally simple with stack traces, may be more complex with event systems. Part of the original rationale against GOTO. - how did this (wrong) data item or variable get here? - how did this piece of state get into this (wrong) state? This is often very hard to answer if the state is outside the program. DOMs are a large piece of global state that everyone working in the browser environment has to deal with.
balena 8y ago
https://en.m.wikipedia.org/wiki/Cyclomatic_complexity https://en.m.wikipedia.org/wiki/Cyclomatic_complexity
- bcaa7f3a8bbc 8y ago
  There is a GCC plugin that can automatically calculate the cyclomatic complexity for programmers, and it has been built into the Linux kernel source tree. I don't know if there is any public statistics, but it would sound great for code reviewers.
j2kun 8y ago
Cute, but it's still just asymptotic complexity... You can use asymptotic notation (big-O) to describe anything that grows. It doesn't have to be code runtime.
jkravitz61 8y ago
This notation is already taken in the computer science world and is used in robotics planning algorithms. https://www.cs.cmu.edu/~motionplanning/lecture/Chap2-Bug-Alg_howie.pdf https://www.cs.cmu.edu/~motionplanning/lecture/Chap2-Bug-Alg...

203 points by danabramov 8y ago

This is a good attempt to kickstart the discussion of some quantitative metric about bugs and the time one needs to deal with it, but I wonder if there wasn't any software engineering theories or practices to address this. If anyone know other related topics, please share them, thanks!

jakear 8y ago
This is somewhat similar to the concept of “Cyclomatic Complexity”, which is the number of independent paths though a piece of code, and can be statically analyzed. https://en.m.wikipedia.org/wiki/Cyclomatic_complexity https://en.m.wikipedia.org/wiki/Cyclomatic_complexity
- danabramov 8y ago
  Note one difference is that I'm talking about characteristics of an API, not particular code snippet. Although it's relevant.
- amelius 8y ago
  This makes me wonder, are software metrics still a thing?
  - rbanffy 8y ago
    Yes. At least for me.
- jerf 8y ago
  I like this concept better than cyclomatic complexity, precisely because this is what cyclomatic complexity is trying to poorly approximate. But it can't be automatically derived in the general case, or even terribly easily in non-general cases, either, unlike cyclomatic complexity. I use a descendant of gometalinter on all my Go code, and one of the things it ships with is cyclomatic complexity, and I always turn it off. Precisely because I tend to think like this, it often gives my code terribly wrong ratings because it can't see that there's actually only a finite set of paths through the code that are possible, and also as the blog post demonstrates, sometimes adding more code actually simplifies the conceptual flow. But cyclomatic complexity will generally say the "more code" is even more cyclomatically complex. There's also some Go-specific elements to my distaste for the measure, though; since in Go right now handling an error is automatically an if statement, that'll hit you right in the metrics. But in many cases, something, err := GetSomething() if err != nil { return errwrap.Wrapf("while trying to get something: {{err}}", err) } Officially that's an if statement; unofficially, it ought to be a cyclomatic complexity of zero. Now, it isn't technically a free if clause, in the sense that a bug could live there, but if you're going to count the exception-based equivalent as zero (and it has the same callout; bugs can lie in your exception handling exactly the same way), then this ought to be practically zero too. Consequently, Go functions tend to get smacked with much higher complexity numbers than exception-equivalent code. Yeah, it's probably not a one-to-one, but it's certainly not as lopsided as the metric makes it seem.
  - sigil 8y ago
    Does gometalinter report absolute cyclomatic complexity numbers, or does it report density (cyclomatic complexity / sloc)?
l0b0 8y ago
I've found TDD to be a good (but not perfect, as some people complain) way to keep complexity down - when you force yourself to test every state you very quickly start focussing on how to keep the number of possible states down.
- AllegedAlec 8y ago
  I think that a more important reason is that TDD forces you to write code in small modules which can be tested independently. Because of this, you have small to medium size modules to reason about, rather than one huge class where you wouldn't know where to start.
jtolmar 8y ago
There are various studies of defect density (bugs per 1000 lines of code) versus whatever factor someone thought would influence bugs. "Software defect density" seem to be the magic search terms for finding actual studies instead of blog posts. I unfortunately can't find it, but I remember there being a study (by Microsoft?) of defect density for projects with different methodologies (scrum, TDD, whatever), which found lines of code is more correlated with defect rate than everything else they tried. They took that as a failure; I've always taken that to mean that reducing lines of code is the most important first step (or probably more accurately reducing tokens of executable code; one liners don't help and type annotations don't hurt). While attempting to find that study, I found a study [1] that claims to have found an empirical sweet spot in defect density by module size - apparently 400 lines for assembly modules or 200 lines for ADA modules. Those are some weird languages to have numbers on, but maybe something else in that paper's family tree has something more relatable. [1] http://www.cs.colostate.edu/~malaiya/p/denton_2000.pdf http://www.cs.colostate.edu/~malaiya/p/denton_2000.pdf
- beaconstudios 8y ago
  even reducing tokens isn't a very good measure - lines of code is a proxy for complexity of implementation. I think a good measure (albeit a heuristic one) is how concise the implementation is relative to the complexity of the problem - or in other words, the ratio of accidental to essential complexity.
biggio 8y ago
It's a broad topic and called "code metrics" or "software metrics".

jonahx 8y ago

> The problem with this code isn’t that it’s “ugly”. We’re not talking about aesthetics. The problem is that if there is a bug in this code, I don’t know where to start looking. But this is precisely what people mean when they say it's ugly. "Ugly" is what this gets translated as by your limbic system which registers in an instant: "fuck this big mess and the work it's going to be to reason about." Aesthetics and usability are deeply entangled here, and the problem with that code is absolutely an aesthetic one. I want to say, more so. Because I experience it that way first, and only later do I enumerate the measurable reasons it's likely to be more buggy because of it.

alephr 8y ago
the problem is that "ugly" isn't precise and can get translated from other reactions that aren't based on experience but are instead just cargo cutting. it's great to use the faster parts of our brains to reason but still important to make sure we're not accepting the mistakes they might make.
- stupidcar 8y ago
  It's also an emotive word that can make discussion more heated. If you're reviewing somebody's code and you call it "ugly", they're liable to get upset, and it's too imprecise for them to act on. It's far more helpful and less combative to explain, in concrete terms, what you think its problems are, and how a different approach could solve them.
rzwitserloot 8y ago
Ugly, given that it is an emotional word, applies to any code that you don't like. WHY do you not like some code? To avoid pointless discussions about taste and gut feelings, I'd bring it down to falsifiable statements. "This is hard to maintain, for reasons X and Y" is such a statement. As is: "This is hard to test for reason: It needlessly engages in combinatorial explosion" is another. Yes, you could shorten that to 'ugly'. Good for you. Nevertheless, I've seen tons of discussions referring to 'ugly' code where the reasons were solid but completely different from maintainability (for example: "It is longer than needed", "It does is not idiomatic for this language"), or even not particularly solid ("It is ugly because it isn't functional", "It is ugly because the style guide says all APIs should always be interfaces", "It is ugly because I see tabs, not spaces").

PezzaDev 8y ago

After seeing callbacks cause a lot of "hard to debug" scenarios for me and for others, I'm starting to believe they are gotos in disguise.

danabramov 8y ago
They are.
pjc50 8y ago
Even worse: it's a "come from", a statement normally only seen in INTERCAL.
- rbanffy 8y ago
  At least "come from" states clearly where the code is coming from. Now... When your callbacks are names and not inline you can't really make that assumption.

lioeters 8y ago

Otherwise known as cyclomatic complexity?

danabramov 8y ago
Cyclomatic complexity refers to a specific code snippet. I am talking about inherent characteristics of a particular API and how the patterns it encourages influences debugging. For example: >In fact, if any value looks wrong in the DOM of a React app, you can trace where it comes from by looking at the code of components above it in the React tree one by one. No matter the app size, tracing a rendered value is (tree height).
- beatgammit 8y ago
  Really, all of this can be pared down to: reduce mutable state. That's what React does, and it's why code written in functional languages like Haskell tend to be much easier to reason about. Basically, the fewer things you need to know about in a given context, the less likely you are too mess it up.

titanix2 8y ago

Funny how the post is about bug complexity, says that "Some API and language designs make a whole class of mistakes impossible. Some create endless problems." and then proceed to an example in Javascript... for which a whole language (Typescript) was invented precisely because it does not scales well.

danabramov 8y ago
My audience is JavaScript developers.
riskable 8y ago
I was going to say something similar but then I thought about it a moment... If I were going to write an article about maddening complexity in code that seems like it should be straightforward I too would choose JavaScript for my examples!

8lall0 8y ago

tl;dr avoid too much nesting, prevent DRY and make small but useful functions. An advice that is valid since forever.

danabramov 8y ago
You can split the first example in small functions and it'll still be a mess. The post is not about duplication, function size, or nesting. We gained predictability by preventing _accumulation_ of errors — which we got by always restarting rendering from scratch.
- dyarosla 8y ago
  Well- the React example doesn’t really restart rendering from scratch (as you yourself mentioned). And in larger applications you’d still prefer to not ‘redraw from scratch’ in React either and build on top of a previous setup. Really what you’re doing is preferring more declarative programming practices over procedural. And as a consequence, more declarative APIs like React.
  - danabramov 8y ago
    For the app developer, React code _does_ restart rendering from scratch. You have no access to the past rendered state from the render method (at least if you don’t intentionally escape the declarative model). This isn’t true only for bugs in React. What happens internally in React is another question. But the point of React abstraction is to let _you_ think in terms of “render from scratch”.
- deleted 8y ago
  [deleted]
- k__ 8y ago
  Funny. I tried to write software like that 10 years ago already, but I always failed at performance. React was just what I was missing.
yen223 8y ago

userbinator 8y ago

I'm not familiar with the language, and yet I remain unconvinced because I could still understand what the code in the first example is doing, and more importantly, see it all at a glance. I don't have to jump around between functions to follow the logic. More verbose code means more opportunity for errors and more code to look through and debug. That said, I'd probably write something closer to this (once again, not sure if this is even valid, but you can just treat it as pseudocode): // don't create these every time, we're just changing their visibility let retryButton = createRetryButton(); // calls trySubmit() when clicked... let successMessage = createSuccessMessage(); let spinner = createSpinner(); let errorMessage = createErrorMessage(); ... function trySubmit() { // assume these don't cause errors if the child isn't already there formStatus.removeChild(errorMessage); formStatus.removeChild(retryButton); formStatus.appendChild(spinner); succ = submitForm(); formStatus.removeChild(spinner); if(succ) formStatus.appendChild(successMessage); else { errorMessage.setMessage(error); formStatus.appendChild(errorMessage); formStatus.appendChild(retryButton) } }

danabramov 8y ago
I’m not saying the first example is hard to _understand_. I’m saying that it’s hard to be confident it works or debug when it has a bad visual output because the unnecessary statefulness creates compounding mistakes over time. My second example (that doesn’t have this problem) is pretty much what you wrote.
jerf 8y ago
"yet I remain unconvinced because I could still understand what the code in the first example is doing, and more importantly, see it all at a glance." This isn't because it's good code. It's because it's code that fits in a blog post, so of course you understand it quickly. If you didn't understand it quickly, the author's point couldn't be conveyed properly. The question isn't really what happens when you have 10 lines of dodgy code. The question is, what happens when you have 10,000 lines of dodgy code? (In a classroom, on a chalkboard, bubble sort isn't any slower than any other sort, and somewhat easier to draw, honestly. Quicksort is rather a lot of stuff to draw. But unless your code solely deals with lists small enough to fit comfortably on a blackboard with work room to spare, you are well advised not to use bubblesort.)

pjc50 8y ago

So what I think this points to is the importance of what I'm going to call "psychological reversibility" in programming: the ability for the developer (with the aid of their tools) to answer three kinds of question: - how did this (wrong) control flow get here? Normally simple with stack traces, may be more complex with event systems. Part of the original rationale against GOTO. - how did this (wrong) data item or variable get here? - how did this piece of state get into this (wrong) state? This is often very hard to answer if the state is outside the program. DOMs are a large piece of global state that everyone working in the browser environment has to deal with.

balena 8y ago

https://en.m.wikipedia.org/wiki/Cyclomatic_complexity https://en.m.wikipedia.org/wiki/Cyclomatic_complexity

bcaa7f3a8bbc 8y ago
There is a GCC plugin that can automatically calculate the cyclomatic complexity for programmers, and it has been built into the Linux kernel source tree. I don't know if there is any public statistics, but it would sound great for code reviewers.

j2kun 8y ago

Cute, but it's still just asymptotic complexity... You can use asymptotic notation (big-O) to describe anything that grows. It doesn't have to be code runtime.

jkravitz61 8y ago

This notation is already taken in the computer science world and is used in robotics planning algorithms. https://www.cs.cmu.edu/~motionplanning/lecture/Chap2-Bug-Alg_howie.pdf https://www.cs.cmu.edu/~motionplanning/lecture/Chap2-Bug-Alg...