6 ms·

Are Tests Necessary in TypeScript?

31 points by capableweb 7y ago

ubertaco 7y ago
The title is not a very good one, because it's nearly a rhetorical question: no language I've ever seen -- not even Haskell -- has a type system that completely removes the need for tests. For example, even Haskell has partial functions over types like Int whose behavior can't necessarily be encoded in the type system (at least, not ergonomically/practically) and thus require actual tests to cover. That said, I think a more accurate title for this post (based on the paragraphs beyond the first one) might be "are tests necessary for a view layer that doesn't have much business logic in it beyond passing properties when that view layer is written in Typescript?" At that point, it's really just a permutation of the question "are tests necessary for the view layer of an application?", which has highly-situational answers.
- xmmrm 7y ago
  >completely removing the need for tests That would be formal verification.
  - simiones 7y ago
    Not really. You still need tests to check whether your application is actually achieving its targets in real-world usage, including feature completeness, usability, performance; and how all of these are impacted by the environment (e.g. network issues in a distributed application). Formal verification only tells you that the application matches its spec, given a set of assumptions. Only testing can tell you whether the spec matches the business case, and whether the spec assumptions hold. Paraphrasing Knuth, code that has only been proven correct but not actually tested should always be approached with caution. Edit: Knuth, not Dijkstra
    - seanmcdirmid 7y ago
      Theoretically, formal verification can only tell you as much as can be encoded in a formal spec@.The problem is that no such kind of specs are known to exist (yet, if ever). As a thought experiment, if the informal specs written in English/Chinese or trapped in our head were as incomplete as a formal spec, we wouldn’t even be able to reason about what to test in the first place. @ Then there is the limitation of verification tech of course.
  - hinkley 7y ago
    Which from what I’ve seen is way harder and way less accessible than testing. And one of the things I say to anxious coworkers who are struggling with testing is that it’s a lot harder than you think.
  - tom_mellior 7y ago
    When I was working with CompCert, I had no problems producing a formally verified compiler that miscompiled its (meager) test suite. Tests are necessary for "formally verified" systems! You might mess up parts of the specs or the model you are verifying against. You can only catch these problems through testing.
- steveklabnik 7y ago
  The title is a good one, because everyone has to learn this lesson sometime. How will they learn? Well, they will probably type "Are tests necessary in typescript?" and come across this post, which will explain it to them. Nobody is typing "are tests necessary for a view layer that doesn't have much business logic in it beyond passing properties when that view layer is written in Typescript?" into Google.
sebazzz 7y ago
Yes, but not to validate your syntax and any references to undefined stuff (except on the boundary where you might interface to external - not in Typescript written - libraries). Still, tests are meant for logic testing - which is something the Typescript compiler won't help you with (or any compiler probably).
- asib 7y ago
  > ...which is something the Typescript compiler won't help you with (or any compiler probably) Not so - there're languages out there in which you're able to encode your business logic within your program's types, at least to a certain extent, e.g. dependently typed languages. This can definitely replace some of your logic tests.
ecmascript 7y ago
> We ported our React frontend from JavaScript to TypeScript, then did the same for our Ruby backend. Next year it will be an article on how they ported everything to rust and webassembly and how that is just so much better than everything. Of course they come to the conclusion that types and tests aren't the same thing. Otherwise you would never need tests, not even in dynamically typed languages.
- noelwelsh 7y ago
  I agree that the answer is obviously yes, as types and tests do different work. However "types" in dynamically typed languages are not the same thing as types in statically typed languages. One difference is that one causes an error at compile time and one causes an error at run time. Types in statically typed languages can replace tests, but types in dynamically typed languages cannot (unless you believe program crashes are a desired response to code errors.)
- steveklabnik 7y ago
  Gary has already checked out Rust and didn't particularly like it, so I doubt that will be happening. Regardless, it was not about being some sort of hipster. It solved real problems: https://www.executeprogram.com/blog/porting-to-typescript-solved-our-api-woes https://www.executeprogram.com/blog/porting-to-typescript-so... (this is linked from the part you quoted, of course.)
- tln 7y ago
  That's not really fair, they ported their stack so front end and backend share type definitions. It was not a fashion driven decision. Wrt to types and tests, "Types" means some verifiable annotations on the code. If a type system can encode enough information in them, perhaps with preconditions or other contracts, then would tests be superfluous? If so, I don't get your assertion about tests and dynamically typed languages. On the other hand, since typescript is unsound, shouldn't you really test like a dynamically typed language?
nelsonic 7y ago
Yes. Types are a great way to communicate with the compiler and fellow humans, but they do not replace tests.
- hinkley 7y ago
  I liken a few different things to Zeno’s paradox. Types might save you from 10% of your failure modes. And the next tool, and every one after, fix 10% of what’s left. But you never arrive. Or alternatively, every efficient method of code verification just results in more code. Each product is “worth” so many man-years and so anything that reduces effort just increases scope. Which may also explain program bloat; adding three libraries gives me time to... add two more libraries.
  - nelsonic 7y ago
    Good points. Thanks for the reminder of Zeon's Pradoxes. https://en.wikipedia.org/wiki/Zeno%27s_paradoxes https://en.wikipedia.org/wiki/Zeno%27s_paradoxes +1
makach 7y ago
Yes.
idoby 7y ago
Yes. You're welcome.
tantalor 7y ago
> We don't want to go down the esoteric and labor-intensive path of automated image capture and image diffing Suck it up. Screenshot diffing solves a specific problem you can't address with unit/integration tests.
holtalanm 7y ago
Short answer: yes. Long answer: yesssssssssssssssssssssssssssssssssss
alkonaut 7y ago
The only person who would (incorrectly) answer "no" to the title question I assume is someone who only ever wrote javascript, and used tests as a way of preventing screwups with types, such as using undefined data or passing the wrong type of data to a function. Which is of course a pretty useless way of spending your day.
thomascgalvin 7y ago
> In Execute Program, we always write tests around subsystems that either (1) are critical to the product or (2) contain a lot of details that can't be statically checked. This is one of those things that sounds good when you're in college, and sounds, well, horrifying is too strong a word, but it will definitely set off the warning bells for anyone that's been programming for a decade or more. My team has a simple standard: every single user-facing requirement is verified by automated testing. We don't have 100% coverage, but it's damn close, somewhere like 98%. The remaining untested code is generally weird branch paths that are hard to trigger, and "shouldn't" be seen in production. But this goes beyond code coverage or TDD: firing up the debugger and executing a unit test is the simplest way to add a feature, period. I cannot fathom building and starting our application so that I can hand test every time I change a line of code. Unit and integration tests aren't punishment inflicted on inferior developers: they'd an extremely useful tool that makes your life easier and your team more productive.
- noelwelsh 7y ago
  What language do you use? My experience in Scala is that tests can be replaced with types in a number of cases (but not all!) In these cases it is often not worth implementing a test, IMO. Depending on the code it is possible to construct the types in such a way that an incorrect implementation is very hard. This requires a modern type system and the knowledge to use it effectively. Type systems like in Java and Go are not up the task for the most part.
  - lern_too_spel 7y ago
    In Scala, you should test that the type system makes the guarantees you expect with compile time errors. Several frameworks provide tooling for this.
- onei 7y ago
  > Unit and integration tests aren't punishment inflicted on inferior developers: they'd an extremely useful tool that makes your life easier and your team more productive. I can't agree with this more and also find the reverse is true: not having tests will punish you repeatedly. If we measure inferiority by unproductivity, adding tests also enables you to become the opposite of inferior (ferior?)
hexa00 7y ago
I think this is somewhat funny since to think of this question at all really highlights the time people waste in JS/python etc.. Testing things that can be automatically checked by a simple compiler. I do find that as some of these people migrate to TypeScript they write too much tests, as they are just not used to have some checks already done for them. But to others that come from compiled/typed languages that's a question you'd never ask :)
samtheprogram 7y ago
Didn’t know Gary Bernhardt was working on something outside of DAS. Quality content as always.
29athrowaway 7y ago
Take a non-trivial program written in a statically typed language, with 100% unit test coverage, pass it to American Fuzzy Lop and see what happens.
hinkley 7y ago
To quote Mr Babbage: “I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”
awinter-py 7y ago
> Types can't (usually) tell us when we accidentally put a "!" in front of a conditional critical point in any 'types vs tests' discussion. strongly typed langs can tell you that your program will run w/out crashing but not that they'll be right (even the 'run w/out crashing' promise turns out to be not always true) I wonder what tools will evolve in the next 10 years to measure if code is 'semantically correct'. Some sort of function that maps from code to docstrings might be able to check this.
- agentultra 7y ago
  Semantically correct is important! What kind of semantics? I prefer denotational. Many programmers think in operational semantics. And how does one verify their code? I'm trying to push our review guidelines to ensure developers submit their code/change with an argument as to why it is correct. I got this idea from E.W. Dijkstra who was having a hard time keeping up with his graduate students and colleagues. In order to save his time and sanity he asked them to submit an argument as to why their change was correct. Instead of validating their algorithms, Dijkstra was able to validate their argument and check their proof. One can use unit tests to make this argument. The level of rigor depends on the task on how difficult it would be to convince the reviewer that your change is correct. One can go beyond unit tests and write property tests, make arguments based on the type information, embed proofs in the types, or make more formal arguments by proof if they feel it's necessary. Making unit tests required is a good minimum case I think but is often not sufficient.
  - awinter-py 7y ago
    I've certainly talked myself into 'correcting' unit tests that didn't agree with bad code
_greim_ 7y ago
The "ideology" video linked at the end of the article was also instructive for me, by framing the issue in other terms. Both tests and types are proof of correctness; tests are proof by example, types are proof by category. The architectural analogy in my head is that types are vertical and horizontal beams, while tests are diagonal bracing. The interlocking of the two is what makes the overall structure rigid.
k__ 7y ago
Different quesion, what kinds of tests does TypeScript eliminate? Also, does ReasonML eliminate other kinds of tests than TypeScript?
stickfigure 7y ago
We're running an experiment on this with our small company. Our core web application consists of a Java backend (40k LOC, plus 12K LOC of tests) and a Typescript/React SPA frontend (12k LOC, zero tests). LOC #s are 'real code' via CLOC. Not a simple CRUD app. It's fine. I can't say that this would be fine for everyone. We have a very small team (2 fullstack engineers) with good communication; we keep as much logic as possible in the backend; we have a robust test harness for the backend. Still, it works very well for us. Even without testing the frontend, it is very rare for bugs to hit users. Our reasoning for skipping frontend tests: Most of the frontend code is visual. Tests can't tell you if the result is aesthetically satisfying, so you have to look at it no matter what. Since the backend diligently enforces data sanity, the frontend can't really get into too much trouble, and a page reload fixes almost anything. Frontend testing seems to be high-effort low-reward. I don't think this would be possible without Typescript. TSX helps too since the compiler (really the IDE) catches what would otherwise be template errors. Large refactorings are pretty painless. I would do this again. I have also reverse engineered about a dozen other company's web applications recently (to automate them), and noticed that there are other app design philosophies that would make this approach difficult. Some teams go with a "smart frontend dumb backend" approach that can produce all sorts of nasty data corruption issues if the frontend screws up. I'm not a fan of this approach but if you take it, I don't think Typescript alone will save you.
- latchkey 7y ago
  > Our reasoning for skipping frontend tests: Most of the frontend code is visual. Tests can't tell you if the result is aesthetically satisfying, so you have to look at it no matter what. Since the backend diligently enforces data sanity, the frontend can't really get into too much trouble, and a page reload fixes almost anything. Frontend testing seems to be high-effort low-reward. This is an example front-end react component project that I wrote that benefits heavily from tests: https://github.com/lookfirst/mui-rff https://github.com/lookfirst/mui-rff Why? Because it is a wrapper around two other frameworks (React Final Form and Material-UI). Any time those two frameworks change something in a way that I didn't expect, I want test failures.

31 points by capableweb 7y ago

The title is not a very good one, because it's nearly a rhetorical question: no language I've ever seen -- not even Haskell -- has a type system that completely removes the need for tests. For example, even Haskell has partial functions over types like Int whose behavior can't necessarily be encoded in the type system (at least, not ergonomically/practically) and thus require actual tests to cover. That said, I think a more accurate title for this post (based on the paragraphs beyond the first one) might be "are tests necessary for a view layer that doesn't have much business logic in it beyond passing properties when that view layer is written in Typescript?" At that point, it's really just a permutation of the question "are tests necessary for the view layer of an application?", which has highly-situational answers.

xmmrm 7y ago
>completely removing the need for tests That would be formal verification.
- simiones 7y ago
  Not really. You still need tests to check whether your application is actually achieving its targets in real-world usage, including feature completeness, usability, performance; and how all of these are impacted by the environment (e.g. network issues in a distributed application). Formal verification only tells you that the application matches its spec, given a set of assumptions. Only testing can tell you whether the spec matches the business case, and whether the spec assumptions hold. Paraphrasing Knuth, code that has only been proven correct but not actually tested should always be approached with caution. Edit: Knuth, not Dijkstra
  - seanmcdirmid 7y ago
    Theoretically, formal verification can only tell you as much as can be encoded in a formal spec@.The problem is that no such kind of specs are known to exist (yet, if ever). As a thought experiment, if the informal specs written in English/Chinese or trapped in our head were as incomplete as a formal spec, we wouldn’t even be able to reason about what to test in the first place. @ Then there is the limitation of verification tech of course.
- hinkley 7y ago
  Which from what I’ve seen is way harder and way less accessible than testing. And one of the things I say to anxious coworkers who are struggling with testing is that it’s a lot harder than you think.
- tom_mellior 7y ago
  When I was working with CompCert, I had no problems producing a formally verified compiler that miscompiled its (meager) test suite. Tests are necessary for "formally verified" systems! You might mess up parts of the specs or the model you are verifying against. You can only catch these problems through testing.
steveklabnik 7y ago
The title is a good one, because everyone has to learn this lesson sometime. How will they learn? Well, they will probably type "Are tests necessary in typescript?" and come across this post, which will explain it to them. Nobody is typing "are tests necessary for a view layer that doesn't have much business logic in it beyond passing properties when that view layer is written in Typescript?" into Google.

sebazzz 7y ago

Yes, but not to validate your syntax and any references to undefined stuff (except on the boundary where you might interface to external - not in Typescript written - libraries). Still, tests are meant for logic testing - which is something the Typescript compiler won't help you with (or any compiler probably).

asib 7y ago
> ...which is something the Typescript compiler won't help you with (or any compiler probably) Not so - there're languages out there in which you're able to encode your business logic within your program's types, at least to a certain extent, e.g. dependently typed languages. This can definitely replace some of your logic tests.

ecmascript 7y ago

> We ported our React frontend from JavaScript to TypeScript, then did the same for our Ruby backend. Next year it will be an article on how they ported everything to rust and webassembly and how that is just so much better than everything. Of course they come to the conclusion that types and tests aren't the same thing. Otherwise you would never need tests, not even in dynamically typed languages.

noelwelsh 7y ago
I agree that the answer is obviously yes, as types and tests do different work. However "types" in dynamically typed languages are not the same thing as types in statically typed languages. One difference is that one causes an error at compile time and one causes an error at run time. Types in statically typed languages can replace tests, but types in dynamically typed languages cannot (unless you believe program crashes are a desired response to code errors.)
steveklabnik 7y ago
Gary has already checked out Rust and didn't particularly like it, so I doubt that will be happening. Regardless, it was not about being some sort of hipster. It solved real problems: https://www.executeprogram.com/blog/porting-to-typescript-solved-our-api-woes https://www.executeprogram.com/blog/porting-to-typescript-so... (this is linked from the part you quoted, of course.)
tln 7y ago
That's not really fair, they ported their stack so front end and backend share type definitions. It was not a fashion driven decision. Wrt to types and tests, "Types" means some verifiable annotations on the code. If a type system can encode enough information in them, perhaps with preconditions or other contracts, then would tests be superfluous? If so, I don't get your assertion about tests and dynamically typed languages. On the other hand, since typescript is unsound, shouldn't you really test like a dynamically typed language?

nelsonic 7y ago

Yes. Types are a great way to communicate with the compiler and fellow humans, but they do not replace tests.

hinkley 7y ago
I liken a few different things to Zeno’s paradox. Types might save you from 10% of your failure modes. And the next tool, and every one after, fix 10% of what’s left. But you never arrive. Or alternatively, every efficient method of code verification just results in more code. Each product is “worth” so many man-years and so anything that reduces effort just increases scope. Which may also explain program bloat; adding three libraries gives me time to... add two more libraries.
- nelsonic 7y ago
  Good points. Thanks for the reminder of Zeon's Pradoxes. https://en.wikipedia.org/wiki/Zeno%27s_paradoxes https://en.wikipedia.org/wiki/Zeno%27s_paradoxes +1

makach 7y ago

Yes.

idoby 7y ago

Yes. You're welcome.

tantalor 7y ago

> We don't want to go down the esoteric and labor-intensive path of automated image capture and image diffing Suck it up. Screenshot diffing solves a specific problem you can't address with unit/integration tests.

holtalanm 7y ago

Short answer: yes. Long answer: yesssssssssssssssssssssssssssssssssss

alkonaut 7y ago

The only person who would (incorrectly) answer "no" to the title question I assume is someone who only ever wrote javascript, and used tests as a way of preventing screwups with types, such as using undefined data or passing the wrong type of data to a function. Which is of course a pretty useless way of spending your day.

thomascgalvin 7y ago

> In Execute Program, we always write tests around subsystems that either (1) are critical to the product or (2) contain a lot of details that can't be statically checked. This is one of those things that sounds good when you're in college, and sounds, well, horrifying is too strong a word, but it will definitely set off the warning bells for anyone that's been programming for a decade or more. My team has a simple standard: every single user-facing requirement is verified by automated testing. We don't have 100% coverage, but it's damn close, somewhere like 98%. The remaining untested code is generally weird branch paths that are hard to trigger, and "shouldn't" be seen in production. But this goes beyond code coverage or TDD: firing up the debugger and executing a unit test is the simplest way to add a feature, period. I cannot fathom building and starting our application so that I can hand test every time I change a line of code. Unit and integration tests aren't punishment inflicted on inferior developers: they'd an extremely useful tool that makes your life easier and your team more productive.

noelwelsh 7y ago
What language do you use? My experience in Scala is that tests can be replaced with types in a number of cases (but not all!) In these cases it is often not worth implementing a test, IMO. Depending on the code it is possible to construct the types in such a way that an incorrect implementation is very hard. This requires a modern type system and the knowledge to use it effectively. Type systems like in Java and Go are not up the task for the most part.
- lern_too_spel 7y ago
  In Scala, you should test that the type system makes the guarantees you expect with compile time errors. Several frameworks provide tooling for this.
onei 7y ago
> Unit and integration tests aren't punishment inflicted on inferior developers: they'd an extremely useful tool that makes your life easier and your team more productive. I can't agree with this more and also find the reverse is true: not having tests will punish you repeatedly. If we measure inferiority by unproductivity, adding tests also enables you to become the opposite of inferior (ferior?)

hexa00 7y ago

I think this is somewhat funny since to think of this question at all really highlights the time people waste in JS/python etc.. Testing things that can be automatically checked by a simple compiler. I do find that as some of these people migrate to TypeScript they write too much tests, as they are just not used to have some checks already done for them. But to others that come from compiled/typed languages that's a question you'd never ask :)

samtheprogram 7y ago

Didn’t know Gary Bernhardt was working on something outside of DAS. Quality content as always.

29athrowaway 7y ago

Take a non-trivial program written in a statically typed language, with 100% unit test coverage, pass it to American Fuzzy Lop and see what happens.

hinkley 7y ago

To quote Mr Babbage: “I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”

awinter-py 7y ago

> Types can't (usually) tell us when we accidentally put a "!" in front of a conditional critical point in any 'types vs tests' discussion. strongly typed langs can tell you that your program will run w/out crashing but not that they'll be right (even the 'run w/out crashing' promise turns out to be not always true) I wonder what tools will evolve in the next 10 years to measure if code is 'semantically correct'. Some sort of function that maps from code to docstrings might be able to check this.

agentultra 7y ago
Semantically correct is important! What kind of semantics? I prefer denotational. Many programmers think in operational semantics. And how does one verify their code? I'm trying to push our review guidelines to ensure developers submit their code/change with an argument as to why it is correct. I got this idea from E.W. Dijkstra who was having a hard time keeping up with his graduate students and colleagues. In order to save his time and sanity he asked them to submit an argument as to why their change was correct. Instead of validating their algorithms, Dijkstra was able to validate their argument and check their proof. One can use unit tests to make this argument. The level of rigor depends on the task on how difficult it would be to convince the reviewer that your change is correct. One can go beyond unit tests and write property tests, make arguments based on the type information, embed proofs in the types, or make more formal arguments by proof if they feel it's necessary. Making unit tests required is a good minimum case I think but is often not sufficient.
- awinter-py 7y ago
  I've certainly talked myself into 'correcting' unit tests that didn't agree with bad code

_greim_ 7y ago

The "ideology" video linked at the end of the article was also instructive for me, by framing the issue in other terms. Both tests and types are proof of correctness; tests are proof by example, types are proof by category. The architectural analogy in my head is that types are vertical and horizontal beams, while tests are diagonal bracing. The interlocking of the two is what makes the overall structure rigid.

k__ 7y ago

Different quesion, what kinds of tests does TypeScript eliminate? Also, does ReasonML eliminate other kinds of tests than TypeScript?

stickfigure 7y ago

We're running an experiment on this with our small company. Our core web application consists of a Java backend (40k LOC, plus 12K LOC of tests) and a Typescript/React SPA frontend (12k LOC, zero tests). LOC #s are 'real code' via CLOC. Not a simple CRUD app. It's fine. I can't say that this would be fine for everyone. We have a very small team (2 fullstack engineers) with good communication; we keep as much logic as possible in the backend; we have a robust test harness for the backend. Still, it works very well for us. Even without testing the frontend, it is very rare for bugs to hit users. Our reasoning for skipping frontend tests: Most of the frontend code is visual. Tests can't tell you if the result is aesthetically satisfying, so you have to look at it no matter what. Since the backend diligently enforces data sanity, the frontend can't really get into too much trouble, and a page reload fixes almost anything. Frontend testing seems to be high-effort low-reward. I don't think this would be possible without Typescript. TSX helps too since the compiler (really the IDE) catches what would otherwise be template errors. Large refactorings are pretty painless. I would do this again. I have also reverse engineered about a dozen other company's web applications recently (to automate them), and noticed that there are other app design philosophies that would make this approach difficult. Some teams go with a "smart frontend dumb backend" approach that can produce all sorts of nasty data corruption issues if the frontend screws up. I'm not a fan of this approach but if you take it, I don't think Typescript alone will save you.

latchkey 7y ago
> Our reasoning for skipping frontend tests: Most of the frontend code is visual. Tests can't tell you if the result is aesthetically satisfying, so you have to look at it no matter what. Since the backend diligently enforces data sanity, the frontend can't really get into too much trouble, and a page reload fixes almost anything. Frontend testing seems to be high-effort low-reward. This is an example front-end react component project that I wrote that benefits heavily from tests: https://github.com/lookfirst/mui-rff https://github.com/lookfirst/mui-rff Why? Because it is a wrapper around two other frameworks (React Final Form and Material-UI). Any time those two frameworks change something in a way that I didn't expect, I want test failures.