9 ms·

I've been on teams at multiple companies that have used type annotations for Python. Developers have generally loved adopting type annotations because it (1) ma

I've been on teams at multiple companies that have used type annotations for Python. Developers have generally loved adopting type annotations because it (1) makes the code easier to understand (2) makes the code easier to refactor (3) improves integration with IDEs and (4) catches real bugs. I disagree that type errors "aren't practically a problem". NoneType errors and AttributeErrors turn up all the time in Python.

globular-toast 5y ago
Clearly didn't watch the talk then. He found that about 1-2% of errors in real world projects were type-related errors.
- joshuamorton 5y ago
  Fwiw this absolutely doesn't match my experience with pytype, both on my own work and, as I understand it, "at scale".
  - globular-toast 5y ago
    Ah yes... good old anecdotal evidence.
    - joshuamorton 5y ago
      The org I'm in regularly publishes peer reviewed papers on engineering productivity (including on this particular subject!). It's far less anecdotal than a random talk by a random dude.
      - globular-toast 5y ago
        Could you kindly link to one of those papers?
        joshuamorton 5y ago
        https://storage.googleapis.com/pub-tools-public-publication-data/pdf/b031c1c4a85542ced86e8536ec2380db72d86ab8.pdf https://storage.googleapis.com/pub-tools-public-publication-... for ane example not directly related to python (a migration from stronger than python to even stronger caught some, though a small number of errors). I've also written about this publicly before, e.g. https://mail.python.org/archives/list/typing-sig@python.org/thread/ENTSMRILZN5YERQFSTWJXLDGX7KGH5DG/ https://mail.python.org/archives/list/typing-sig@python.org/..., where I found a number of errors by tweaking pytype in a way to make it stronger. It's difficult to quantify what percentage of bugs would have been found anyway, but pretty much any time the type systems I work with improve, it uncovers new bugs.
- aflag 5y ago
  Fixing 1-2% of the errors in real world projects is nice, isn’t it? Not to mention catching the non real world errors before even running any tests.
- marcusbuffett 5y ago
  I have to really question how the study was conducted if only 1-2% of python errors were considered type errors. Even if the only type error you consider is None vs not None, you'd get more than that. Certainly those errors account for more than 1% of the errors I see in my own professional experience using Python. I'm curious what his reasoning was so I'll watch the talk, but there's something screwy there.
  - globular-toast 5y ago
    It's a good talk even if just to learn more about types. It's not even specifically about python. The result might be surprising, but those are the most interesting results! I note that, so far, nobody has countered this with any evidence, only anecdotes and "surely not!"
- nickm12 5y ago
  [40 minutes into the talk]: Looks like the study scanned 3.6 million github issues across 1.7 million repositories and looked for the number of bugs that were TypeError, AttributeError, or NameError and found it was 2.7% of issues. I'm not a fan of this kind of methodology, because there are so many different use cases for github issue tracker (including feature requests, documentation requests, etc.) that I think the denominator here is overestimated. I'd rather have a study that looked at 1000 issues in depth with a rigorous rubric than one that grepped through millions of issues. Also, looking only at the issues doesn't count the ones that were caught in development or testing. I like the notion of "moving your bugs to the left"[1] and would much rather find these type errors at build time than later. Finally as I said in my other comment, avoiding type errors is not the only benefit of type checking. Making code easier to understand and refactor (including programmatic refactoring) are other benefits that are not captured only by counting bugs. [1] https://samwho.dev/blog/move-your-bugs-to-the-left/ https://samwho.dev/blog/move-your-bugs-to-the-left/
- nickm12 5y ago
  I got around to watching the entire talk. It's not convincing to me. The presentation tacitly assumes that the only benefit to type systems is catching type errors and ignores the benefits to code readability and tooling. It also assumes that time spend developing and time spend debugging are fungible and I find that a bit simplistic; I would gladly trade debugging time for development time, even if the ratio is uneven. Finally, it doesn't even talk about the runtime performance penalty that dynamic languages typically incur. Aside: It's worth commenting more on "code readability" because it cuts both ways. It can absolutely be the case that a complex, manifest type system can be a hindrance to understandably and productivity due to the "noise" of all the types that need to be written. I think struggling with C++ and (to a lesser extent) Java typing is part of the reason the industry really embraced dynamic languages in the 2000s. Python and Ruby were like a breath of fresh air. But eventually we built large systems with these languages and discovered the pain that comes from having no type-safety. The trend I've seen in the last decade is to embrace type inference, with explicit type declarations on function signatures and otherwise only when needed for disambiguation. To me, this is the sweet spot, giving the benefits of static typing with much of the cleanliness and concision of dynamic languages.

globular-toast 5y ago

Clearly didn't watch the talk then. He found that about 1-2% of errors in real world projects were type-related errors.

joshuamorton 5y ago
Fwiw this absolutely doesn't match my experience with pytype, both on my own work and, as I understand it, "at scale".
- globular-toast 5y ago
  Ah yes... good old anecdotal evidence.
  - joshuamorton 5y ago
    The org I'm in regularly publishes peer reviewed papers on engineering productivity (including on this particular subject!). It's far less anecdotal than a random talk by a random dude.
    - globular-toast 5y ago
      Could you kindly link to one of those papers?
      - joshuamorton 5y ago
        https://storage.googleapis.com/pub-tools-public-publication-data/pdf/b031c1c4a85542ced86e8536ec2380db72d86ab8.pdf https://storage.googleapis.com/pub-tools-public-publication-... for ane example not directly related to python (a migration from stronger than python to even stronger caught some, though a small number of errors). I've also written about this publicly before, e.g. https://mail.python.org/archives/list/typing-sig@python.org/thread/ENTSMRILZN5YERQFSTWJXLDGX7KGH5DG/ https://mail.python.org/archives/list/typing-sig@python.org/..., where I found a number of errors by tweaking pytype in a way to make it stronger. It's difficult to quantify what percentage of bugs would have been found anyway, but pretty much any time the type systems I work with improve, it uncovers new bugs.
aflag 5y ago
Fixing 1-2% of the errors in real world projects is nice, isn’t it? Not to mention catching the non real world errors before even running any tests.
marcusbuffett 5y ago
I have to really question how the study was conducted if only 1-2% of python errors were considered type errors. Even if the only type error you consider is None vs not None, you'd get more than that. Certainly those errors account for more than 1% of the errors I see in my own professional experience using Python. I'm curious what his reasoning was so I'll watch the talk, but there's something screwy there.
- globular-toast 5y ago
  It's a good talk even if just to learn more about types. It's not even specifically about python. The result might be surprising, but those are the most interesting results! I note that, so far, nobody has countered this with any evidence, only anecdotes and "surely not!"
nickm12 5y ago
[40 minutes into the talk]: Looks like the study scanned 3.6 million github issues across 1.7 million repositories and looked for the number of bugs that were TypeError, AttributeError, or NameError and found it was 2.7% of issues. I'm not a fan of this kind of methodology, because there are so many different use cases for github issue tracker (including feature requests, documentation requests, etc.) that I think the denominator here is overestimated. I'd rather have a study that looked at 1000 issues in depth with a rigorous rubric than one that grepped through millions of issues. Also, looking only at the issues doesn't count the ones that were caught in development or testing. I like the notion of "moving your bugs to the left"[1] and would much rather find these type errors at build time than later. Finally as I said in my other comment, avoiding type errors is not the only benefit of type checking. Making code easier to understand and refactor (including programmatic refactoring) are other benefits that are not captured only by counting bugs. [1] https://samwho.dev/blog/move-your-bugs-to-the-left/ https://samwho.dev/blog/move-your-bugs-to-the-left/
nickm12 5y ago
I got around to watching the entire talk. It's not convincing to me. The presentation tacitly assumes that the only benefit to type systems is catching type errors and ignores the benefits to code readability and tooling. It also assumes that time spend developing and time spend debugging are fungible and I find that a bit simplistic; I would gladly trade debugging time for development time, even if the ratio is uneven. Finally, it doesn't even talk about the runtime performance penalty that dynamic languages typically incur. Aside: It's worth commenting more on "code readability" because it cuts both ways. It can absolutely be the case that a complex, manifest type system can be a hindrance to understandably and productivity due to the "noise" of all the types that need to be written. I think struggling with C++ and (to a lesser extent) Java typing is part of the reason the industry really embraced dynamic languages in the 2000s. Python and Ruby were like a breath of fresh air. But eventually we built large systems with these languages and discovered the pain that comes from having no type-safety. The trend I've seen in the last decade is to embrace type inference, with explicit type declarations on function signatures and otherwise only when needed for disambiguation. To me, this is the sweet spot, giving the benefits of static typing with much of the cleanliness and concision of dynamic languages.