6 ms·

Hype: "Static Typing reduces bugs." Shower: A review of all the available literature (up to 2014), showing that the solid research is inconclusive, whi

Hype: "Static Typing reduces bugs." Shower: A review of all the available literature (up to 2014), showing that the solid research is inconclusive, while the conclusive research had methodological issues. Static typing lets you do more complicated things by offloading a subset of complexity-management to robots. The remaining human-managed complexity expands until new development slows to a crawl, and no further human-managed complexity can be admitted to the system, similar to adding more lanes on a freeway.

junon 3y ago
Dynamic typing doesn't change/improve this, though, so I'm not sure what the point being made is. I'm also not sure I agree with it at its premise anyway.
brink 3y ago
I know, right? When did obvious things become hype and "need literature"? Type bugs are an obvious thing and happen all the time. Not hype, but an obvious fact: "Static Typing reduces type related bugs."
- the_gastropod 3y ago
  I think the important question is: at what cost? E.g., if it takes me 4x more time to write statically-typed code, and it saves me 10% fewer bugs (completely made up numbers here), is that worthwhile? Maybe, if I'm programming self-driving cars or autopilot software for aircraft. Probably not if I'm programming a web calendar for dog sitters.
  - brink 3y ago
    I don't consider it an added cost, you're just moving the cost upfront to reduce long-term pain. Ruby apps are easy to build, but are difficult to maintain. Rust apps take a bit more effort to build, but are easier to maintain.
    - the_gastropod 3y ago
      But this is where the studies come in. Lots of people think this is true. And it seems perfectly reasonable. But there's really no research to back this up.
      - worik 3y ago
        Can this be confirmed or refuted? There is so much variation confounding any study, in tasks, specifications, programmer inclination. I use both staticly typed (rust) and dynamically typed (perl) a lot. I like rust for the big projects and perl to hold everything together-job control That is my inclination, not a fact. I have used many languages over the years. Static/dynamic typing is just one dimension, and not necessarily related to quality
      - mpweiher 3y ago
        And with most of the type-related stuff it's slightly worse than "no research to back this up". There is research, and it doesn't back this up. Except for the documentation effect. For that there is apparently solid research.
  - mpweiher 3y ago
    It's even worse than that. If it saves me 10% bugs per unit of code, but I have to write 20% more code, am I actually even ahead in the bugs department?
  - wharvle 3y ago
    If you work with other people or will ever onboard someone to the project (including your future self) static types save time. Lots of it.
- bryik 3y ago
  If "type bugs are an obvious thing and happen all the time" and "static Typing reduces type related bugs" then it should be easy to demonstrate this empirically. However, "a review of all the available literature (up to 2014), show[s] that the solid research is inconclusive while the conclusive research had methodological issues."
  - deleted 3y ago
    [deleted]
  - _gabe_ 3y ago
    Why would you need an empirical study for this? It’s trivially provable. Runtime exceptions in a language like JavaScript can arise from type mismatches. That’s impossible to do in a language like Java, because the compiler catches it before you ever run the program. This eliminates an entire class of bugs. What you’re proposing here sounds like somebody saying “How do we know Rust results in less bugs than C++ without an empirical study?”. Even though, we _know_ Rust eliminates an entire class of memory related bugs. I say this as a C++ advocate too. Anytime I run into a memory bug, that’s a bug that would not have happened in Rust. Likewise, any time you run into a runtime exception due to a type mismatch (for example: expected an int not an object), that is a bug that would not have happened with a type safe language. Edit: I also want to add that the metric is important. Is it number of bugs per line of code? What does that even mean? Assembly programs consist of many more lines of code because it’s more terse, but the number of bugs in assembly will probably be greater than a higher level language. Even though the large number of lines of code would probably push the metric down and make it seem like assembly has a low number of bugs per line of code. Because of this, bugs per line of code isn’t a useful metric. The only way I could think of measuring this would be to have two feature for feature equivalent projects in two different languages and compare the number of bugs in each. But even that probably has a bunch of flaws.
    - bryik 3y ago
      I think you're right that static typing reduces bugs, but I am not convinced the reduction is significant or meaningful. If static typing has a significant effect, then why is the existing research so weak and inconclusive? I don't understand your point about metrics and measurement. Are you saying the effect of static typing is so small that it is completely dominated by other confounding factors and thus cannot be measured?
theLiminator 3y ago
Even if it doesn't reduce bugs (and how do we even measure this? in terms of bugs per loc? bugs per unit time?), it does make APIs easier to use (not even in terms of correctness, but in terms of time required to grok an API).
- _greim_ 3y ago
  "Reduce bugs" is kind of a loaded term anyway. Static typing doesn't reduce bugs in an absolute sense, but I think it does reduce bugs per unit of value delivered. That's a lot harder to measure in a formal study.
Swizec 3y ago
> Static typing lets you do more complicated things by offloading a subset of complexity-management to robots I've read some of the research on this! Yes, static typing improves documentation and helps you navigate code. It also correlates with code quality and reduces smells. Inconclusive whether that's because of static typing or because more mature teams are likelier to choose static typing. But all the research agrees: Static typing does not reduce logic bugs. You can build the wrong thing just as easily with dynamic and with static typing. The only type of bug that static typing reduces is the sort of bug you'll find by running the code. In my experience, static typing is best thought of as a way to reduce the need for manually written unit tests. Instead of writing tests that break when a function signature changes, you write types that break when you call functions wrong. You still need tests for logic. Static typing doesn't help there.
- tikhonj 3y ago
  > Static typing does not reduce logic bugs. This seems like a strong statement to make based on the research. What I've seen falls into several camps: - research that made some conclusion about logic bugs for complete beginners on small assignments, with languages that have bad type systems - research that had significant limitations making it impossible to generalize - research that failed to demonstrate that static typing reduced bugs—which is very different from demonstrating that it didn't! I haven't done a super thorough review of the literature or anything, but I have looked through a decent number of software engineering papers on the subject. The only strong conclusion I got from the research is that we can't get strong conclusions on the subject through purely empirical means. Hell, the whole question is meaningless. "Static typing" is not one thing—there's way more difference between Java and Haskell than between Java and Python, even though both Java and Haskell are statically typed and Python isn't. (This is even assuming you completely ignore Python's type annotations and gradual typing!)
- jrajav 3y ago
  > The only type of bug that static typing reduces is the sort of bug you'll find by running the code. This is a pretty solid argument in favor of static typing, then, unless you somehow have a test suite that exercises every possible code path and type variation in your codebase, and also keeps itself perfectly up to date. Because otherwise you're rarely running all of your code and verifying the result.
CyberDildonics 3y ago
offloading a subset of complexity-management to robots Are you calling compilers robots?
- bigbillheck 3y ago
  I don't see why not.
  - junon 3y ago
    Is a car a robot to you? Is a hello world program? Curious where the line is.
  - CyberDildonics 3y ago
    Are you still going to use the word software or compiler or do you plan on switching over to calling everything a robot? Is your coffee maker a robot too?
    - chihuahua 3y ago
      A coffee maker is definitely a (simple) robot. It's a machine that performs a task, just like a welding robot in a car factory or a Roomba.
      - CyberDildonics 3y ago
        Is a single assembly instruction a robot?
        worik 3y ago
        Quit pedantic, useless, nitpicking over a casual reference to "robot"
        CyberDildonics 3y ago
        If people are going to start calling compilers robots, I want to know about it.
koririn 3y ago
A literary review I made in 2020 was actually pretty conclusive about it also reducing bugs. I think we might be missing some of the later literature here.

junon 3y ago

Dynamic typing doesn't change/improve this, though, so I'm not sure what the point being made is. I'm also not sure I agree with it at its premise anyway.

brink 3y ago

I know, right? When did obvious things become hype and "need literature"? Type bugs are an obvious thing and happen all the time. Not hype, but an obvious fact: "Static Typing reduces type related bugs."

the_gastropod 3y ago
I think the important question is: at what cost? E.g., if it takes me 4x more time to write statically-typed code, and it saves me 10% fewer bugs (completely made up numbers here), is that worthwhile? Maybe, if I'm programming self-driving cars or autopilot software for aircraft. Probably not if I'm programming a web calendar for dog sitters.
- brink 3y ago
  I don't consider it an added cost, you're just moving the cost upfront to reduce long-term pain. Ruby apps are easy to build, but are difficult to maintain. Rust apps take a bit more effort to build, but are easier to maintain.
  - the_gastropod 3y ago
    But this is where the studies come in. Lots of people think this is true. And it seems perfectly reasonable. But there's really no research to back this up.
    - worik 3y ago
      Can this be confirmed or refuted? There is so much variation confounding any study, in tasks, specifications, programmer inclination. I use both staticly typed (rust) and dynamically typed (perl) a lot. I like rust for the big projects and perl to hold everything together-job control That is my inclination, not a fact. I have used many languages over the years. Static/dynamic typing is just one dimension, and not necessarily related to quality
    - mpweiher 3y ago
      And with most of the type-related stuff it's slightly worse than "no research to back this up". There is research, and it doesn't back this up. Except for the documentation effect. For that there is apparently solid research.
- mpweiher 3y ago
  It's even worse than that. If it saves me 10% bugs per unit of code, but I have to write 20% more code, am I actually even ahead in the bugs department?
- wharvle 3y ago
  If you work with other people or will ever onboard someone to the project (including your future self) static types save time. Lots of it.
bryik 3y ago
If "type bugs are an obvious thing and happen all the time" and "static Typing reduces type related bugs" then it should be easy to demonstrate this empirically. However, "a review of all the available literature (up to 2014), show[s] that the solid research is inconclusive while the conclusive research had methodological issues."
- deleted 3y ago
  [deleted]
- _gabe_ 3y ago
  Why would you need an empirical study for this? It’s trivially provable. Runtime exceptions in a language like JavaScript can arise from type mismatches. That’s impossible to do in a language like Java, because the compiler catches it before you ever run the program. This eliminates an entire class of bugs. What you’re proposing here sounds like somebody saying “How do we know Rust results in less bugs than C++ without an empirical study?”. Even though, we _know_ Rust eliminates an entire class of memory related bugs. I say this as a C++ advocate too. Anytime I run into a memory bug, that’s a bug that would not have happened in Rust. Likewise, any time you run into a runtime exception due to a type mismatch (for example: expected an int not an object), that is a bug that would not have happened with a type safe language. Edit: I also want to add that the metric is important. Is it number of bugs per line of code? What does that even mean? Assembly programs consist of many more lines of code because it’s more terse, but the number of bugs in assembly will probably be greater than a higher level language. Even though the large number of lines of code would probably push the metric down and make it seem like assembly has a low number of bugs per line of code. Because of this, bugs per line of code isn’t a useful metric. The only way I could think of measuring this would be to have two feature for feature equivalent projects in two different languages and compare the number of bugs in each. But even that probably has a bunch of flaws.
  - bryik 3y ago
    I think you're right that static typing reduces bugs, but I am not convinced the reduction is significant or meaningful. If static typing has a significant effect, then why is the existing research so weak and inconclusive? I don't understand your point about metrics and measurement. Are you saying the effect of static typing is so small that it is completely dominated by other confounding factors and thus cannot be measured?

theLiminator 3y ago

Even if it doesn't reduce bugs (and how do we even measure this? in terms of bugs per loc? bugs per unit time?), it does make APIs easier to use (not even in terms of correctness, but in terms of time required to grok an API).

_greim_ 3y ago
"Reduce bugs" is kind of a loaded term anyway. Static typing doesn't reduce bugs in an absolute sense, but I think it does reduce bugs per unit of value delivered. That's a lot harder to measure in a formal study.

Swizec 3y ago

> Static typing lets you do more complicated things by offloading a subset of complexity-management to robots I've read some of the research on this! Yes, static typing improves documentation and helps you navigate code. It also correlates with code quality and reduces smells. Inconclusive whether that's because of static typing or because more mature teams are likelier to choose static typing. But all the research agrees: Static typing does not reduce logic bugs. You can build the wrong thing just as easily with dynamic and with static typing. The only type of bug that static typing reduces is the sort of bug you'll find by running the code. In my experience, static typing is best thought of as a way to reduce the need for manually written unit tests. Instead of writing tests that break when a function signature changes, you write types that break when you call functions wrong. You still need tests for logic. Static typing doesn't help there.

tikhonj 3y ago
> Static typing does not reduce logic bugs. This seems like a strong statement to make based on the research. What I've seen falls into several camps: - research that made some conclusion about logic bugs for complete beginners on small assignments, with languages that have bad type systems - research that had significant limitations making it impossible to generalize - research that failed to demonstrate that static typing reduced bugs—which is very different from demonstrating that it didn't! I haven't done a super thorough review of the literature or anything, but I have looked through a decent number of software engineering papers on the subject. The only strong conclusion I got from the research is that we can't get strong conclusions on the subject through purely empirical means. Hell, the whole question is meaningless. "Static typing" is not one thing—there's way more difference between Java and Haskell than between Java and Python, even though both Java and Haskell are statically typed and Python isn't. (This is even assuming you completely ignore Python's type annotations and gradual typing!)
jrajav 3y ago
> The only type of bug that static typing reduces is the sort of bug you'll find by running the code. This is a pretty solid argument in favor of static typing, then, unless you somehow have a test suite that exercises every possible code path and type variation in your codebase, and also keeps itself perfectly up to date. Because otherwise you're rarely running all of your code and verifying the result.

CyberDildonics 3y ago

offloading a subset of complexity-management to robots Are you calling compilers robots?

bigbillheck 3y ago
I don't see why not.
- junon 3y ago
  Is a car a robot to you? Is a hello world program? Curious where the line is.
- CyberDildonics 3y ago
  Are you still going to use the word software or compiler or do you plan on switching over to calling everything a robot? Is your coffee maker a robot too?
  - chihuahua 3y ago
    A coffee maker is definitely a (simple) robot. It's a machine that performs a task, just like a welding robot in a car factory or a Roomba.
    - CyberDildonics 3y ago
      Is a single assembly instruction a robot?
      - worik 3y ago
        Quit pedantic, useless, nitpicking over a casual reference to "robot"
        CyberDildonics 3y ago
        If people are going to start calling compilers robots, I want to know about it.

koririn 3y ago

A literary review I made in 2020 was actually pretty conclusive about it also reducing bugs. I think we might be missing some of the later literature here.