4 ms·

We had dedicated QA testing. QA testing failed to create the requisite conditions. Type checking is not a replacement for QA testing. However, it is much cheap

by jchw 4y ago

We had dedicated QA testing. QA testing failed to create the requisite conditions. Type checking is not a replacement for QA testing. However, it is much cheaper and also finds bugs that might be incredibly hard to find through random chance or through structured smoke testing, because it does not depend on how common a given code path is. QA testing is even less likely to find issues than automated test suites, since at least you can qualify branch coverage with test suites (which still will not enumerate all of the possible code paths, and thus... Type checking is incredibly useful.) So while type checking really is not a replacement for QA, QA is also not a replacement for type checking. I ultimately quit Python for many reasons, but the desire to write more correct code was the big one. Not just me, but my entire team was burned out on Python. Investing so much into unit testing and increasing coverage, upgrading Python versions, and adding type checking as much as possible was still not yielding the benefits we'd hoped for despite how much effort it took. We eventually found much better success with the then-still-new Go programming language, which we used on most new projects going forward. There was still the occasional head scratcher in production, but deployments were as quiet as a church mouse. YMMV of course; there's plenty of success and failure stories with any programming language. I think we were all ready and willing to put a lot of investment into improving our robustness situation and just felt that we got more out of that effort with Go than Python for our programs. Python still has plenty of advantages too, so we didn't completely quit it; we wound up using Django and SQLalchemy here and there, but definitely most stuff moved to Go over time. (And personally, I stopped using Python for those things too, since moving on.)

ReflectedImage 4y ago
[flagged]
- Toniglandyl 4y ago
  Do you have a reference to as study that support your point ?
  - deleted 4y ago
    [deleted]
  - ReflectedImage 4y ago
    [flagged]
- mountainriver 4y ago
  Static typing is not just about code performance, it’s absolutely about correctness as well. I would really love to see that study as it flys in the face of all my experience
  - ReflectedImage 4y ago
    https://games.greggman.com/game/dynamic-typing-static-typing/ https://games.greggman.com/game/dynamic-typing-static-typing...
    - lostmsu 4y ago
      > has on average 2.5x the number of bugs This claim is not supported by the linked article. In fact, the main claim (only 2% of bugs are type errors) of the linked article also does not make any sense. It is based on the assumption, that typing errors only ever cause TypeError, AttributeError, or NameError in Python, which is, ironically, false, because Python is a dynamic language. To give a concrete example, I went out to GitHub, got into top Python repository (TensorFlow), and searched for a first closed pull request which explicitly mentioned ValueError and did not mention either of the above. It turned out to be this one: https://github.com/tensorflow/tensorflow/pull/47017 https://github.com/tensorflow/tensorflow/pull/47017 This issue is an obvious type error: in statically-typed languages the value of float can not be nil. You could probably repeat that experiment with other closed pull requests, but I can bet you you will find that the rate will be closer to 50% if not 90%.
      - ReflectedImage 4y ago
        That not even an error, it's an usability improvement.
        lostmsu 4y ago
        Firstly, usability bugs like this are still bugs. Secondly, think how that issue surfaced in the first place. Somebody used TensorFlow in their Python code, and their Python code produced unexpected result and/or crashed. Judging by the fix, it is highly unlikely to have produced one of the errors from the above list. As a ML practitioner I can also tell you it likely means somebody had to spend a significant amount of time to get to the cause once they saw the output, because numerical miscalculations are very hard to debug in the first place.
- zelphirkalt 4y ago
  Not that I am a huge static or dynamic typing fan, but I think language with very strict type system disagree. Like for example Haskell or Rust, where when it compiles, it usually works, unless you introduced a logic mistake. Python with its typing might potentially never get there, but that is not the point.
- jchw 4y ago
  This is nearly impossible to precisely study: - Because correlation != causation, for one thing: 2.5x the number of bugs in static "style" code bases (not sure what that means) does not mean that static type systems lead to more bugs. It could be that users of dynamically typed languages find themselves needing robust automated tests more often, because they can't rely on static typings. This would suggest that in a similarly productive language, your effort would go much further with types than without, even if this measurement is accurate. - Because measuring bugs is hard: you don't actually know how many bugs a given piece of code contains, you only know how many you found. Statically typed code contains more information about the intent of the code. Not only does this make it easier for machines to analyze, it makes it easier for humans to analyze, too. Ask anyone who does security audits whether they prefer code that has strict static type information or not. - Because not all bugs are the same; a reputable study would need to go through great pains to classify the different kinds of bugs and determine their severity as well as determine why they happened. Counting bugs as just a number and then comparing them is like counting SLOC. It doesn't tell you anything on it's own! > Static typing has nothing to do with code correctness, it's purely about code performance. Uhhh... Says who? Type erasure systems offer no performance benefits at all, since the types are removed before runtime. There are contradicting studies, that suggest static typing prevents bugs, but there's little point in trying to fight them. None of the studies I've seen feel robust or complete enough, and on top of that, what would really be helpful is a meta-analysis of multiple more robust studies so we can get an idea of the true takeaways. Until then, there will be some subjectivity. There's no nice way to say this: I don't trust people who treat issues like this as black and white. To me, it's a sign of immaturity as an engineer. I do of course believe that static type checking is a net win overall and that the evidence to the contrary likely has other factors (like, for example, relying on relatively primitive type checkers that catch less errors, require more handholding, and generally have trouble handling idiomatic code) but like I said, YMMV: everyone has their failure and success stories. To me, if I can find a bug before I even save the file, versus needing to wait until QA catches it, or worse, it lights prod on fire, it's probably a win. As far as this "static code style" business goes, I don't understand it. My code does not look that different when it's JS vs TS. For Python, MyPy still doesn't seem sufficient, so I would not be surprised if strictly using things MyPy supports would make code worse. For us, switching to Go had more benefits than JUST types; of course we wound up having better performance and memory usage too, and the error handling practices, while verbose, definitely put error handling and edge cases front and center.
HdS84 4y ago
I often think that may is just bad. It's type system feels bolted on and limiting. Where typescript can do many things and often teases me into a new pattern, mypy is ugly and stifling. It's a pity. I really like types but mypy is just not great. It's ok, buts that's not enough for python.

by jchw 4y ago

ReflectedImage 4y ago

[flagged]

Toniglandyl 4y ago
Do you have a reference to as study that support your point ?
- deleted 4y ago
  [deleted]
- ReflectedImage 4y ago
  [flagged]
mountainriver 4y ago
Static typing is not just about code performance, it’s absolutely about correctness as well. I would really love to see that study as it flys in the face of all my experience
- ReflectedImage 4y ago
  https://games.greggman.com/game/dynamic-typing-static-typing/ https://games.greggman.com/game/dynamic-typing-static-typing...
  - lostmsu 4y ago
    > has on average 2.5x the number of bugs This claim is not supported by the linked article. In fact, the main claim (only 2% of bugs are type errors) of the linked article also does not make any sense. It is based on the assumption, that typing errors only ever cause TypeError, AttributeError, or NameError in Python, which is, ironically, false, because Python is a dynamic language. To give a concrete example, I went out to GitHub, got into top Python repository (TensorFlow), and searched for a first closed pull request which explicitly mentioned ValueError and did not mention either of the above. It turned out to be this one: https://github.com/tensorflow/tensorflow/pull/47017 https://github.com/tensorflow/tensorflow/pull/47017 This issue is an obvious type error: in statically-typed languages the value of float can not be nil. You could probably repeat that experiment with other closed pull requests, but I can bet you you will find that the rate will be closer to 50% if not 90%.
    - ReflectedImage 4y ago
      That not even an error, it's an usability improvement.
      - lostmsu 4y ago
        Firstly, usability bugs like this are still bugs. Secondly, think how that issue surfaced in the first place. Somebody used TensorFlow in their Python code, and their Python code produced unexpected result and/or crashed. Judging by the fix, it is highly unlikely to have produced one of the errors from the above list. As a ML practitioner I can also tell you it likely means somebody had to spend a significant amount of time to get to the cause once they saw the output, because numerical miscalculations are very hard to debug in the first place.
zelphirkalt 4y ago
Not that I am a huge static or dynamic typing fan, but I think language with very strict type system disagree. Like for example Haskell or Rust, where when it compiles, it usually works, unless you introduced a logic mistake. Python with its typing might potentially never get there, but that is not the point.
jchw 4y ago
This is nearly impossible to precisely study: - Because correlation != causation, for one thing: 2.5x the number of bugs in static "style" code bases (not sure what that means) does not mean that static type systems lead to more bugs. It could be that users of dynamically typed languages find themselves needing robust automated tests more often, because they can't rely on static typings. This would suggest that in a similarly productive language, your effort would go much further with types than without, even if this measurement is accurate. - Because measuring bugs is hard: you don't actually know how many bugs a given piece of code contains, you only know how many you found. Statically typed code contains more information about the intent of the code. Not only does this make it easier for machines to analyze, it makes it easier for humans to analyze, too. Ask anyone who does security audits whether they prefer code that has strict static type information or not. - Because not all bugs are the same; a reputable study would need to go through great pains to classify the different kinds of bugs and determine their severity as well as determine why they happened. Counting bugs as just a number and then comparing them is like counting SLOC. It doesn't tell you anything on it's own! > Static typing has nothing to do with code correctness, it's purely about code performance. Uhhh... Says who? Type erasure systems offer no performance benefits at all, since the types are removed before runtime. There are contradicting studies, that suggest static typing prevents bugs, but there's little point in trying to fight them. None of the studies I've seen feel robust or complete enough, and on top of that, what would really be helpful is a meta-analysis of multiple more robust studies so we can get an idea of the true takeaways. Until then, there will be some subjectivity. There's no nice way to say this: I don't trust people who treat issues like this as black and white. To me, it's a sign of immaturity as an engineer. I do of course believe that static type checking is a net win overall and that the evidence to the contrary likely has other factors (like, for example, relying on relatively primitive type checkers that catch less errors, require more handholding, and generally have trouble handling idiomatic code) but like I said, YMMV: everyone has their failure and success stories. To me, if I can find a bug before I even save the file, versus needing to wait until QA catches it, or worse, it lights prod on fire, it's probably a win. As far as this "static code style" business goes, I don't understand it. My code does not look that different when it's JS vs TS. For Python, MyPy still doesn't seem sufficient, so I would not be surprised if strictly using things MyPy supports would make code worse. For us, switching to Go had more benefits than JUST types; of course we wound up having better performance and memory usage too, and the error handling practices, while verbose, definitely put error handling and edge cases front and center.

HdS84 4y ago

I often think that may is just bad. It's type system feels bolted on and limiting. Where typescript can do many things and often teases me into a new pattern, mypy is ugly and stifling. It's a pity. I really like types but mypy is just not great. It's ok, buts that's not enough for python.