4 ms·

Dynamic language or not, having good test coverage will be your biggest support.

weberc2 8y ago
It's definitely important, but we (Python shop) have "good" (85%) test coverage and we still see 500s in prod every day because of things a type checker could trivially catch. And this is just in the course of normal operation; this isn't even a migration. Having extensive experience with both Go and Python, I would conservatively estimate that Go requires ~30% fewer tests than untyped Python for the same confidence. That's more than 30% time savings; not only are you not writing 30% of the tests, but that's 30% fewer tests to have to maintain. Of course these aren't the only considerations--for certain tasks Python may be faster to develop with (although I think people forget about things like deployment, tooling, dependency management, performance requirements, etc when they make their estimations).
- jb3689 8y ago
  I noticed something similar with Elixir with regards to the number of tests we needed thanks to the compiler, but it also has late binding (albeit it is statically typed). A "best of both worlds" exists out there
- MattHeard 8y ago
  Line coverage is not test coverage. Having a test execute every line in your application doesn't mean your application is _covered_ or _tested_.
  - weberc2 8y ago
    Yes, we all know the metric is easily gamed but no one at our org is trying to game the metric. We are paid to build a product, not to boost the metric.
    - kbp 8y ago
      It's not about gaming the metric, it's just that the metric doesn't mean very much in the first place. Running a coverage tool during tests won't show you edge cases you forgot to handle in the code under test, it will show you code that's not tested at all. That can sometimes be useful for pointing out blind spots, but you shouldn't derive any confidence in the tests from a high coverage score, even if the people who worked on the project had the best intentions. Coverage tools could only measure quality of a test suite if you're assuming that either the code is perfect or that the existing tests cover (logically) everything about what they test. Without either of those guarantees, it doesn't tell you anything very meaningful, as you discovered.
      - weberc2 8y ago
        The metric is meaningful; I think you’re misinterpreting it. To your point, 100% coverage doesn’t mean you’ve eliminated all bugs, but it does mean that your code base almost certainly has a lower bug yield that the code base with 50% coverage (assuming no one has games the metric). If you really think that the metric is meaningless and useless for deriving confidence, then you are necessarily asserting that code bases with 100% coverage have indistinguishable bug yields compared to those with 50%, 5%, or even 0% coverage. A claim like this is too extraordinary to be believed without considerable evidence.
        kbp 8y ago
        I guess it's useful for deriving a baseline level of confidence, like a low coverage score is a red flag, and an increasing coverage score probably corresponds to increasing test coverage, but my issue is that 100% coverage doesn't mean anything about the correctness of the code in absolute terms (unless 'gaming the metric' includes not thinking of every edge case, ie, that we're assuming the existing test suite is perfect). If you're working on a poorly tested codebase, it's a useful relative metric of your progress in testing what already exists, but unless you're assuming the code is already correct, that doesn't mean anything more than that. If you wanted to derive, say, the confidence that you won't see 500's daily in production from a metric, then line coverage isn't an effective one to use for that; the tests you write that give you that kind of confidence don't really help your coverage score. The parts of the codebase that are in the most urgent need of tests for getting that kind of confidence in places will most of the time be ones that already have good coverage; think of how TDD works, even if you're not doing TDD. I could agree that a high coverage score is a prerequisite for having confidence that your test suite is comprehensive (in general), but that's such a low bar, it's like saying a full bath is a prerequisite for a nice house, just knowing that shouldn't do much to convince you it's a mansion.
- ezekg 8y ago
  Integration tests are your friend. ;)

It's definitely important, but we (Python shop) have "good" (85%) test coverage and we still see 500s in prod every day because of things a type checker could trivially catch. And this is just in the course of normal operation; this isn't even a migration. Having extensive experience with both Go and Python, I would conservatively estimate that Go requires ~30% fewer tests than untyped Python for the same confidence. That's more than 30% time savings; not only are you not writing 30% of the tests, but that's 30% fewer tests to have to maintain. Of course these aren't the only considerations--for certain tasks Python may be faster to develop with (although I think people forget about things like deployment, tooling, dependency management, performance requirements, etc when they make their estimations).

jb3689 8y ago
I noticed something similar with Elixir with regards to the number of tests we needed thanks to the compiler, but it also has late binding (albeit it is statically typed). A "best of both worlds" exists out there
MattHeard 8y ago
Line coverage is not test coverage. Having a test execute every line in your application doesn't mean your application is _covered_ or _tested_.
- weberc2 8y ago
  Yes, we all know the metric is easily gamed but no one at our org is trying to game the metric. We are paid to build a product, not to boost the metric.
  - kbp 8y ago
    It's not about gaming the metric, it's just that the metric doesn't mean very much in the first place. Running a coverage tool during tests won't show you edge cases you forgot to handle in the code under test, it will show you code that's not tested at all. That can sometimes be useful for pointing out blind spots, but you shouldn't derive any confidence in the tests from a high coverage score, even if the people who worked on the project had the best intentions. Coverage tools could only measure quality of a test suite if you're assuming that either the code is perfect or that the existing tests cover (logically) everything about what they test. Without either of those guarantees, it doesn't tell you anything very meaningful, as you discovered.
    - weberc2 8y ago
      The metric is meaningful; I think you’re misinterpreting it. To your point, 100% coverage doesn’t mean you’ve eliminated all bugs, but it does mean that your code base almost certainly has a lower bug yield that the code base with 50% coverage (assuming no one has games the metric). If you really think that the metric is meaningless and useless for deriving confidence, then you are necessarily asserting that code bases with 100% coverage have indistinguishable bug yields compared to those with 50%, 5%, or even 0% coverage. A claim like this is too extraordinary to be believed without considerable evidence.
      - kbp 8y ago
        I guess it's useful for deriving a baseline level of confidence, like a low coverage score is a red flag, and an increasing coverage score probably corresponds to increasing test coverage, but my issue is that 100% coverage doesn't mean anything about the correctness of the code in absolute terms (unless 'gaming the metric' includes not thinking of every edge case, ie, that we're assuming the existing test suite is perfect). If you're working on a poorly tested codebase, it's a useful relative metric of your progress in testing what already exists, but unless you're assuming the code is already correct, that doesn't mean anything more than that. If you wanted to derive, say, the confidence that you won't see 500's daily in production from a metric, then line coverage isn't an effective one to use for that; the tests you write that give you that kind of confidence don't really help your coverage score. The parts of the codebase that are in the most urgent need of tests for getting that kind of confidence in places will most of the time be ones that already have good coverage; think of how TDD works, even if you're not doing TDD. I could agree that a high coverage score is a prerequisite for having confidence that your test suite is comprehensive (in general), but that's such a low bar, it's like saying a full bath is a prerequisite for a nice house, just knowing that shouldn't do much to convince you it's a mansion.
ezekg 8y ago
Integration tests are your friend. ;)