4 ms·

What about invalid arguments? What if someone calls: (random-elements -23 nil)? Should I write tests for those cases? The function already handles negatives by

by lgas 7y ago

What about invalid arguments? What if someone calls: (random-elements -23 nil)? Should I write tests for those cases? The function already handles negatives by returning an empty list for any count less than one. This isn’t tested; but the code is pretty clear. In the case of the nil, an exception will be thrown. That’s OK with me. This is a dynamically typed language. You get exceptions when you mess up the types. Is that a risk? Sure, but only if some other part of the system was written without tests. If you call this function from a module that you wrote with tests, using something as effective as the TDD discipline, you won’t be passing nil or negative numbers, or any other form of invalid arguments. So I’m not going to worry about that. It’s on you. This attitude coming from an experienced software developer is hard to fathom. Sure, if you're an experienced and meticulous developer that fully embraces TDD and you only work with other experienced and meticulous developers that full embrace TDD on the same type of software handling the same types of problems over and over, then sure, you might be able dodge a pretty high percentage of bugs a pretty high percentage of the time. But most teams in the real world have a mix of inexperienced and experienced developers, meticulous and careless developers, developers that embrace TDD to varying degrees, and work on a variety of problems in a mix of domains with which the team members have varying degrees of experience. So now your choices are: 1) code each and every function in a way that gives you high confidence that it works for the happy path and rely on the discipline of your team to ensure that no unhappy paths are hit or, for approximately the same level of effort: 2) code each and every function in a way that gives you high confidence that it works for the happy path and which you have an extremely high degree of confidence that no unhappy paths even exist at all It's just not a tough decision for me.

deleted 7y ago
[deleted]
JackFr 7y ago
It’s worse than that though — he believes an exception will be thrown, but doesn’t have a test showing it. Indeed in many dynamically typed languages an argument of the ‘wrong’ type is silently coerced in possibly unexpected ways.
- je42 7y ago
  in a unit-test scenario, you are writing tests based on the language and its properties. he is applying risk based assessment which tests are required to provide a high quality implementation. If a system misbehaves you will catch in higher level tests in the test pyramid eventually. The bug was found regarding the null pointer, he can and should add a dedicated unittest to reproduce the issue when he fixes the bug, since there is evidence that his assessment was wrong. This is not bad but usually quite efficient if this doesn't occur often or has a large impact on the system's user base. In a software solution you can't prove 100% correctness - neither through "standard" type-systems nor through tests. If you want to show formal correctness of a program have a look at https://cs.stackexchange.com/questions/13785/formal-program-verification-in-practice https://cs.stackexchange.com/questions/13785/formal-program-...
  - lkitching 7y ago
    > If a system misbehaves you will catch in higher level tests in the test pyramid eventually. You might catch it eventually, or you might not. That's the problem with hoping you write enough tests. > he can and should add a dedicated unittest to reproduce the issue when he fixes the bug His argument is that dynamic language don't 'really' require writing more exhaustive tests than in an expressive static language. In this scenario, not only would he need to write an extra test but he also needs to spend time after the fact tracking down an error thrown in a higher-level function. > In a software solution you can't prove 100% correctness This isn't about proving 100% correctness, it's about what happens when you pass a non-list into a list processing function. More expressive type systems allow you to reduce the number of nonsensical questions you have to answer.
antonvs 7y ago
Just adding another hard to fathom quote: > ... even using a dynamically typed language I don’t have to write the test for nil because I know that nil will never be passed. I don’t need to make states unrepresentable, if I know those states will never be represented. It would be interesting to actually study and measure how much time even teams doing TDD spend on dealing with these kinds of errors. In my experience, they're pretty common.
- AnimalMuppet 7y ago
  "My caller will never mess up" is... let's be charitable and call it "more optimistic than justifiable from experience".
hinkley 7y ago
In the 90’s we had a large rush of new people writing libraries for the first or second time. One common source of frustration was instability in the exceptions thrown due to bad states, from bad inputs to networking issues to disk permissions. You’d get a new point version and suddenly have to rewrite your error handling code so the user still got a reasonable message and/or call to action. Lesson? Errors are part of your public interface. Write at least enough tests to pin the behavior. It also helps with false positives for tests. Because if you introduce a null pointer exception for negative numbers while trying to add features or improve performance, that’s probably not what you expected. Certainly not what your users expect. Test your shit. It takes less energy than the mental gymnastics of justifying why you aren’t.
greenyouse 7y ago
I don't quite agree... I feel like, theoretically, yes we should feed bad data to all the functions and use property-based tests to ensure the features hold. However, in the real-world if you're using a language like Clojure for day to day development your team is probably skilled enough to write basic functions without meticulously checking every little thing for errors. There has to be a trade off between 99.9999% confidence of your functions and fast development velocity. For most products the application code matters a lot more than tests. There should still be strong checks on the external interfaces to your system with bad data tests, schemas, property based testing, etc. For a normal function inside your system it's mostly overkill to be testing those things. Integration tests should validate that your functions call each other correctly and some acceptance tests could generate data from one interface and check that it flows through the system well. There could be corrupted types that you use for testing with Clojure's spec library that either return the expected type or a random corrupted type. Those combinations should adequately exercise the functions to find most of the errors.
mikeash 7y ago
What stands out for me is the casual acceptance of this behavior for negative parameters. It returns an empty list so it’s all good? No way! Asking for a list of negative length is a conceptual error. This should at least be an assert. That ensures nobody accidentally starts relying on bad behavior by prohibiting it. Of course, if you can use the type system to do this by using an unsigned type (and one without implicit non-failing conversions from signed types such as C has) then so much the better. I think the author has the foundation of a good idea: don’t write tests for things that aren’t supposed to happen. (This is not the same as writing tests for error handling: errors are supposed to happen!) But he builds the wrong thing on top. You don’t just ignore things that aren’t supposed to happen and hope that they don’t. You make sure they don’t happen, either by asserting at runtime or making them logically impossible at compile time. Once you reach that point, then you see that a type is just an assert that’s nicer to write, checked earlier, and is more comprehensive. The tradeoff is that they are less expressive, so you still need traditional asserts sometimes.
- jghn 7y ago
  > You make sure they don’t happen, either by asserting at runtime or making them logically impossible at compile time. One thing I find common with arguments from people like the author is that they don't consider types can be more powerful than the ones they're used to dealing with in the languages they use. It seems impossible to them that one could simply encode the parameter as being a type which does not allow for negative values or `nil`. As you say, asserting at compile time that the program is correct.
  - mikeash 7y ago
    I guess this is a specific example of the Blub Paradox. I saw this a lot when Swift came out. Suddenly a lot of Objective-C programmers were learning a new language with a fairly different type system, and it was tough. The Optional type was an especially strong pain point. What I found particularly interesting is that C (and Objective-C) has optional types. All pointer types are optionals. What it lacks is optional non-pointer types, and non-optional pointer types. Because optionality is an inherent part of pointer types in the language, you never learn to think about optionality separately. It's like a fish in water: it's so common you don't even see it. I suppose it would show up even more strongly with someone used to a language like Ruby. If C programmers have a hard time really grasping the idea of optionality because the language doesn't make it explicit, then a Ruby programmer might have trouble with static types at all. (And, to be fair, the same no doubt applies in the other direction: someone coming from a strong statically typed language probably has some hurdles to understanding dynamic type systems.)
dwenzek 7y ago
The author claims that the number of tests required are only those tests that are necessary to describe the correct behavior of the system and that this is true whatever the programming language is dynamic or statically typed. But with these assertions you highlighted What about invalid arguments? What if someone calls: (random-elements -23 nil)?, the author shows that he don't give any value to static typing at all. He simply believes that what is tested with types don't matter: So I’m not going to worry about that. It’s on you.. I think he misses the fact that types are a cheap syntactic way to bring confidence on very diverse aspects of a system: compatibility of values or binary representations but also correctness of dynamic behaviors (see how Rust controls memory with scope type variables) or even algorithmic complexity (see A typed, algebraic approach to parsing https://www.cl.cam.ac.uk/~nk480/parsing.pdf https://www.cl.cam.ac.uk/~nk480/parsing.pdf).

by lgas 7y ago

deleted 7y ago

[deleted]

JackFr 7y ago

It’s worse than that though — he believes an exception will be thrown, but doesn’t have a test showing it. Indeed in many dynamically typed languages an argument of the ‘wrong’ type is silently coerced in possibly unexpected ways.

je42 7y ago
in a unit-test scenario, you are writing tests based on the language and its properties. he is applying risk based assessment which tests are required to provide a high quality implementation. If a system misbehaves you will catch in higher level tests in the test pyramid eventually. The bug was found regarding the null pointer, he can and should add a dedicated unittest to reproduce the issue when he fixes the bug, since there is evidence that his assessment was wrong. This is not bad but usually quite efficient if this doesn't occur often or has a large impact on the system's user base. In a software solution you can't prove 100% correctness - neither through "standard" type-systems nor through tests. If you want to show formal correctness of a program have a look at https://cs.stackexchange.com/questions/13785/formal-program-verification-in-practice https://cs.stackexchange.com/questions/13785/formal-program-...
- lkitching 7y ago
  > If a system misbehaves you will catch in higher level tests in the test pyramid eventually. You might catch it eventually, or you might not. That's the problem with hoping you write enough tests. > he can and should add a dedicated unittest to reproduce the issue when he fixes the bug His argument is that dynamic language don't 'really' require writing more exhaustive tests than in an expressive static language. In this scenario, not only would he need to write an extra test but he also needs to spend time after the fact tracking down an error thrown in a higher-level function. > In a software solution you can't prove 100% correctness This isn't about proving 100% correctness, it's about what happens when you pass a non-list into a list processing function. More expressive type systems allow you to reduce the number of nonsensical questions you have to answer.

antonvs 7y ago

Just adding another hard to fathom quote: > ... even using a dynamically typed language I don’t have to write the test for nil because I know that nil will never be passed. I don’t need to make states unrepresentable, if I know those states will never be represented. It would be interesting to actually study and measure how much time even teams doing TDD spend on dealing with these kinds of errors. In my experience, they're pretty common.

AnimalMuppet 7y ago
"My caller will never mess up" is... let's be charitable and call it "more optimistic than justifiable from experience".

hinkley 7y ago

In the 90’s we had a large rush of new people writing libraries for the first or second time. One common source of frustration was instability in the exceptions thrown due to bad states, from bad inputs to networking issues to disk permissions. You’d get a new point version and suddenly have to rewrite your error handling code so the user still got a reasonable message and/or call to action. Lesson? Errors are part of your public interface. Write at least enough tests to pin the behavior. It also helps with false positives for tests. Because if you introduce a null pointer exception for negative numbers while trying to add features or improve performance, that’s probably not what you expected. Certainly not what your users expect. Test your shit. It takes less energy than the mental gymnastics of justifying why you aren’t.

greenyouse 7y ago

I don't quite agree... I feel like, theoretically, yes we should feed bad data to all the functions and use property-based tests to ensure the features hold. However, in the real-world if you're using a language like Clojure for day to day development your team is probably skilled enough to write basic functions without meticulously checking every little thing for errors. There has to be a trade off between 99.9999% confidence of your functions and fast development velocity. For most products the application code matters a lot more than tests. There should still be strong checks on the external interfaces to your system with bad data tests, schemas, property based testing, etc. For a normal function inside your system it's mostly overkill to be testing those things. Integration tests should validate that your functions call each other correctly and some acceptance tests could generate data from one interface and check that it flows through the system well. There could be corrupted types that you use for testing with Clojure's spec library that either return the expected type or a random corrupted type. Those combinations should adequately exercise the functions to find most of the errors.

mikeash 7y ago

What stands out for me is the casual acceptance of this behavior for negative parameters. It returns an empty list so it’s all good? No way! Asking for a list of negative length is a conceptual error. This should at least be an assert. That ensures nobody accidentally starts relying on bad behavior by prohibiting it. Of course, if you can use the type system to do this by using an unsigned type (and one without implicit non-failing conversions from signed types such as C has) then so much the better. I think the author has the foundation of a good idea: don’t write tests for things that aren’t supposed to happen. (This is not the same as writing tests for error handling: errors are supposed to happen!) But he builds the wrong thing on top. You don’t just ignore things that aren’t supposed to happen and hope that they don’t. You make sure they don’t happen, either by asserting at runtime or making them logically impossible at compile time. Once you reach that point, then you see that a type is just an assert that’s nicer to write, checked earlier, and is more comprehensive. The tradeoff is that they are less expressive, so you still need traditional asserts sometimes.

jghn 7y ago
> You make sure they don’t happen, either by asserting at runtime or making them logically impossible at compile time. One thing I find common with arguments from people like the author is that they don't consider types can be more powerful than the ones they're used to dealing with in the languages they use. It seems impossible to them that one could simply encode the parameter as being a type which does not allow for negative values or `nil`. As you say, asserting at compile time that the program is correct.
- mikeash 7y ago
  I guess this is a specific example of the Blub Paradox. I saw this a lot when Swift came out. Suddenly a lot of Objective-C programmers were learning a new language with a fairly different type system, and it was tough. The Optional type was an especially strong pain point. What I found particularly interesting is that C (and Objective-C) has optional types. All pointer types are optionals. What it lacks is optional non-pointer types, and non-optional pointer types. Because optionality is an inherent part of pointer types in the language, you never learn to think about optionality separately. It's like a fish in water: it's so common you don't even see it. I suppose it would show up even more strongly with someone used to a language like Ruby. If C programmers have a hard time really grasping the idea of optionality because the language doesn't make it explicit, then a Ruby programmer might have trouble with static types at all. (And, to be fair, the same no doubt applies in the other direction: someone coming from a strong statically typed language probably has some hurdles to understanding dynamic type systems.)

dwenzek 7y ago

The author claims that the number of tests required are only those tests that are necessary to describe the correct behavior of the system and that this is true whatever the programming language is dynamic or statically typed. But with these assertions you highlighted What about invalid arguments? What if someone calls: (random-elements -23 nil)?, the author shows that he don't give any value to static typing at all. He simply believes that what is tested with types don't matter: So I’m not going to worry about that. It’s on you.. I think he misses the fact that types are a cheap syntactic way to bring confidence on very diverse aspects of a system: compatibility of values or binary representations but also correctness of dynamic behaviors (see how Rust controls memory with scope type variables) or even algorithmic complexity (see A typed, algebraic approach to parsing https://www.cl.cam.ac.uk/~nk480/parsing.pdf https://www.cl.cam.ac.uk/~nk480/parsing.pdf).