4 ms·

Oh? Their tests have 100% coverage of every feature, and every meaningful, non-obvious interaction between features?

by vkou 4y ago

latchkey 4y ago
Pretty much yes, look them up. They are huge on TDD and pair programming.
- vkou 4y ago
  I'll believe it when I see their codebase. In my experience (10 years in test infra for ~50 to 300-dev products)[1], for any non-trivial product, there's always a colossal testing gap between what I would consider to be thorough coverage and actually attained coverage. The combinatoric complexity I alluded to is one of the reasons for it. And this is despite significant and on-going investments in test cases, test infra, integration testing infra, and good overall dev culture around quality. Also, pair programming has nothing to do with any of this. Mentioning it is like saying that their tests are better because they use IntelliJ instead of Eclipse. It's a very interesting fact, but irrelevant to this. [1] I will, however, point out, that there have been many times in my career that I have said 'This comment should be a test'. Tests are incredible. [2] But there is no circumstance where I would ever agree with a blanket policy of 'Every comment should be a test'. [2] It's entirely feasible, and highly desirable to get a small project to the point where just running the tests makes you feel 100% confident in the correctness of your changes. It is highly desirable to drive large products towards such a point, but those efforts can, at best, approach it asymptotically, and cover some enumerated, but limited set of use-cases and data-flows. [3] [3] And don't even get me started on verifying adherence to security best-practices through testing. It's an utter miracle that security gets two thoughts from a test-writer, and those thoughts are inevitably "These checks are annoying, how do I disable them to get my test to work?"
  - latchkey 4y ago
    Wow, sounds like you’ve consistently worked only in some really shitty environments, sorry for you. The worst companies I've ever worked for, had no testing. Seriously, PL is all about TDD, I'm not joking. PP actually does matter because two people sign off on commits that they worked on together and it is quite an achievement to convince your coworker that you're sitting next to, to not write a test. Consider it a sort of checks and balances. PL's version of pair programming is 100% of the time, it works. I have been running an open source project for a few years now with 40k downloads a month on NPM. This project has a heavy dependency on two other projects. Testing isn’t 100%, but it is enough to rely on it for releases. I also started out with TDD as well. Nothing is perfect, in fact we just had a release yesterday with a contributed fix in it, that came with a test, and still ended up with a bug... but out of 118 tags, a few here and there isn't bad. I routinely upgrade all the dependencies and never hear a peep out of people who depend on it. Most recently, I wrote some software that ran on 20k servers... it was extremely well tested because if it failed, it could have caused massive amounts of downtime. I've worked for a number of other companies where we had fantastic testing infrastructure, CI/CD... mostly because we started from day one with that. Bolting it on after the fact, never happens to the degree it should. I just saw Hashicorp release Hermes today... and was saddened to see it had zero testing in it. I just sighed and closed the browser window. It just isn't worth it.
    - vkou 4y ago
      > Consider it a sort of checks and balances. So, just like a proper code review process. > I have been running an open source project for a few years now with 40k downloads a month on NPM. This project has a heavy dependency on two other projects. Are they dependencies on libraries, or are they dependencies on services? Library dependencies are 'easy' to test. Service dependencies are very, very difficult. Tractable, but very difficult. That's just one of the difference between a small project, and a large product. Just because we have vastly more than two dependencies doesn't mean that I'm working in a shitty environment. It just means I'm working on a large problem. > I've worked for a number of other companies where we had fantastic testing infrastructure, CI/CD... mostly because we started from day one with that. Bolting it on after the fact, never happens to the degree it should. I'm not sure why you believe that all of these things weren't with us at day 1, either.
      - latchkey 4y ago
        > So, just like a proper code review process. Except it happens in real time, with a real person, sitting next to you. > Are they dependencies on libraries, or are they dependencies on services? In this case, other libraries... but I honestly don't see the difference. Libraries and services have APIs. They either work or they don't. Services have the additional complexity of network failures and runtime errors, but these are things that can be dealt with and tested in the primary project. > Just because we have vastly more than two dependencies doesn't mean that I'm working in a shitty environment. You made the wrong equation there. The environment is shitty because you said that you're in an environment where people have created products so complex that it is very difficult to achieve adequate testing. You also shrugged off Eclipse vs. IDEA... where I'd actually say that matters. You also talk about disabling tests for security... that seems absurd. > It's entirely feasible, and highly desirable to get a small project to the point where just running the tests makes you feel 100% confident in the correctness of your changes. I've built multiple companies that have achieved significant revenue on that confidence. Not small projects. I know it is possible to do. > I'm not sure why you believe that all of these things weren't with us at day 1, either. So you started off good and ended poorly? Another reason to believe that the environment is shitty.
lmm 4y ago
You don't need 100% to be better than comments.
- vkou 4y ago
  You don't need to be at 100% to be better than some comments. Mind you, no serious project is anywhere close to 100% coverage. It may have 100% unit-test coverage, but that is a far cry from 100% coverage. And the valuable comments are the ones that describe integration interactions.
  - lmm 4y ago
    No serious project is anywhere close to "100% comment coverage" either.
    - vkou 4y ago
      I'm not pitching comments as a replacement for all tests, like some are pitching tests as a replacement for all comments. I'm pitching comments as a compromise when the test that encodes their meaning is impractical.
      - lmm 4y ago
        Combining tests and comments is still not going to get you to 100% coverage though. So for comments to be worthwhile the marginal gain from using them has to outweigh the cost. IME it's usually better to put that effort into better tests/tooling/etc..
      - latchkey 4y ago
        I'm curious, who is pitching for that?

Oh? Their tests have 100% coverage of every feature, and every meaningful, non-obvious interaction between features?

by vkou 4y ago

Oh? Their tests have 100% coverage of every feature, and every meaningful, non-obvious interaction between features?

latchkey 4y ago

Pretty much yes, look them up. They are huge on TDD and pair programming.

vkou 4y ago
I'll believe it when I see their codebase. In my experience (10 years in test infra for ~50 to 300-dev products)[1], for any non-trivial product, there's always a colossal testing gap between what I would consider to be thorough coverage and actually attained coverage. The combinatoric complexity I alluded to is one of the reasons for it. And this is despite significant and on-going investments in test cases, test infra, integration testing infra, and good overall dev culture around quality. Also, pair programming has nothing to do with any of this. Mentioning it is like saying that their tests are better because they use IntelliJ instead of Eclipse. It's a very interesting fact, but irrelevant to this. [1] I will, however, point out, that there have been many times in my career that I have said 'This comment should be a test'. Tests are incredible. [2] But there is no circumstance where I would ever agree with a blanket policy of 'Every comment should be a test'. [2] It's entirely feasible, and highly desirable to get a small project to the point where just running the tests makes you feel 100% confident in the correctness of your changes. It is highly desirable to drive large products towards such a point, but those efforts can, at best, approach it asymptotically, and cover some enumerated, but limited set of use-cases and data-flows. [3] [3] And don't even get me started on verifying adherence to security best-practices through testing. It's an utter miracle that security gets two thoughts from a test-writer, and those thoughts are inevitably "These checks are annoying, how do I disable them to get my test to work?"
- latchkey 4y ago
  Wow, sounds like you’ve consistently worked only in some really shitty environments, sorry for you. The worst companies I've ever worked for, had no testing. Seriously, PL is all about TDD, I'm not joking. PP actually does matter because two people sign off on commits that they worked on together and it is quite an achievement to convince your coworker that you're sitting next to, to not write a test. Consider it a sort of checks and balances. PL's version of pair programming is 100% of the time, it works. I have been running an open source project for a few years now with 40k downloads a month on NPM. This project has a heavy dependency on two other projects. Testing isn’t 100%, but it is enough to rely on it for releases. I also started out with TDD as well. Nothing is perfect, in fact we just had a release yesterday with a contributed fix in it, that came with a test, and still ended up with a bug... but out of 118 tags, a few here and there isn't bad. I routinely upgrade all the dependencies and never hear a peep out of people who depend on it. Most recently, I wrote some software that ran on 20k servers... it was extremely well tested because if it failed, it could have caused massive amounts of downtime. I've worked for a number of other companies where we had fantastic testing infrastructure, CI/CD... mostly because we started from day one with that. Bolting it on after the fact, never happens to the degree it should. I just saw Hashicorp release Hermes today... and was saddened to see it had zero testing in it. I just sighed and closed the browser window. It just isn't worth it.
  - vkou 4y ago
    > Consider it a sort of checks and balances. So, just like a proper code review process. > I have been running an open source project for a few years now with 40k downloads a month on NPM. This project has a heavy dependency on two other projects. Are they dependencies on libraries, or are they dependencies on services? Library dependencies are 'easy' to test. Service dependencies are very, very difficult. Tractable, but very difficult. That's just one of the difference between a small project, and a large product. Just because we have vastly more than two dependencies doesn't mean that I'm working in a shitty environment. It just means I'm working on a large problem. > I've worked for a number of other companies where we had fantastic testing infrastructure, CI/CD... mostly because we started from day one with that. Bolting it on after the fact, never happens to the degree it should. I'm not sure why you believe that all of these things weren't with us at day 1, either.
    - latchkey 4y ago
      > So, just like a proper code review process. Except it happens in real time, with a real person, sitting next to you. > Are they dependencies on libraries, or are they dependencies on services? In this case, other libraries... but I honestly don't see the difference. Libraries and services have APIs. They either work or they don't. Services have the additional complexity of network failures and runtime errors, but these are things that can be dealt with and tested in the primary project. > Just because we have vastly more than two dependencies doesn't mean that I'm working in a shitty environment. You made the wrong equation there. The environment is shitty because you said that you're in an environment where people have created products so complex that it is very difficult to achieve adequate testing. You also shrugged off Eclipse vs. IDEA... where I'd actually say that matters. You also talk about disabling tests for security... that seems absurd. > It's entirely feasible, and highly desirable to get a small project to the point where just running the tests makes you feel 100% confident in the correctness of your changes. I've built multiple companies that have achieved significant revenue on that confidence. Not small projects. I know it is possible to do. > I'm not sure why you believe that all of these things weren't with us at day 1, either. So you started off good and ended poorly? Another reason to believe that the environment is shitty.

lmm 4y ago

You don't need 100% to be better than comments.

vkou 4y ago
You don't need to be at 100% to be better than some comments. Mind you, no serious project is anywhere close to 100% coverage. It may have 100% unit-test coverage, but that is a far cry from 100% coverage. And the valuable comments are the ones that describe integration interactions.
- lmm 4y ago
  No serious project is anywhere close to "100% comment coverage" either.
  - vkou 4y ago
    I'm not pitching comments as a replacement for all tests, like some are pitching tests as a replacement for all comments. I'm pitching comments as a compromise when the test that encodes their meaning is impractical.
    - lmm 4y ago
      Combining tests and comments is still not going to get you to 100% coverage though. So for comments to be worthwhile the marginal gain from using them has to outweigh the cost. IME it's usually better to put that effort into better tests/tooling/etc..
    - latchkey 4y ago
      I'm curious, who is pitching for that?