5 ms·

I'm just talking null-less FP languages such as Haskell and Elm, not a full proof system such Lean and Agda or a formal specification language such as TLA+. I'

by 1-more 1y ago

I'm just talking null-less FP languages such as Haskell and Elm, not a full proof system such Lean and Agda or a formal specification language such as TLA+. I'm not sure I agree with your prior that "your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot." Invalid input is at the very edge of the program where it goes through a parser. If I parse a string value into a type with no room for null, that eliminates a whole class of errors throughout the program. I do need to test that my parser does what I assume it will, sure. But once I have a type that cannot represent illegal data, I can focus my testing away from playing defense on every function.

9rx 1y ago
Assume that there was room for null. What is your concrete example of where null becomes a problem in production, but goes unnoticed by your tests?
- 1-more 1y ago
  In a world where I am writing a language with null and have half-implemented a grown up type system by checking for null everywhere and writing tests that try to call functions with null (EDIT: and I remembered to do all of that), I guess we could say that I'm at the same place I am right now. But right now I don't have to write defensive tests that include null. You're asking about a circumstance that's just very very different from the one I'm in.
  - 9rx 1y ago
    > But right now I don't have to write defensive tests that include null. You seem to misunderstand. The question was centred around the fact that unexpected null cases end up being tested by virtue of you covering normal test cases due to the constraints on execution. Explicitly testing for null is something else. Something else I suggest unnecessary — at least where null is not actually part of the contract, which for the purposes of our discussion is the case. Again, we're specifically talking about the testing that is necessary to the extent of covering what is missing in the type system when you don't have a formal proof language in hand. > You're asking about a circumstance that's just very very different from the one I'm in. But one you've clearly had trouble with in the past given your claims that you weren't able to deal with it. Otherwise, how would you know? Perhaps I can phrase my request in another way: If null isn't expected in your codebase, and your testing over where the type system is lacking has covered your bases to know that the behaviour is as documented, where are the errant nulls going to magically appear from?
    - 1-more 1y ago
      Oh the errant nulls only ever came up when I was writing dynamically typed languages. Is that what you're asking about?
      - 9rx 1y ago
        Not specifically, but if that is where your example comes from, that’s fine. It’s all the same. What have you got?
        1-more 1y ago
        On a PHP project I ran into cases where I changed the shape of an object (associative array) I was passing around and forgot about one of the places I was using it. Didn’t turn into a production bug but still was the kind of thing I would rather be a squiggly line rather than remembering to rerun all the paths through the code. Didn’t help that we were testing by hand. Same thing on the front end in JS: change the shape of some record that has more places using it than I could remember. Better tests would have caught these. A compiler would be even better. FWIW I’ve written a lot of tests of code written in all of the languages I like. You absolutely need tests, you just don’t need them to be as paranoid when writing them.
        9rx 1y ago
        > A compiler would be even better. Right, but the condition here is that the languages that are expressive enough to negate the need for testing are, shall we say, unusable. In the real world people are going to be using, at best, languages with gimped type systems that still require testing to fill in the gaps. Given that, we're trying to understand your rejection of the premise that the tests you will write to fill in those gaps will also catch things like null exceptions in the due course of execution. It remains unclear where you think these errant nulls are magically coming from. I'm not convinced "Didn’t help that we were testing by hand.", "Better tests would have caught these." is that rejection. Those assertions, while no doubt applicable to your circumstances, is kind of like saying that static type systems don't help either because you can write something like this: data MaybeString = Null | Str String len :: MaybeString -> Int len (Str s) = length s main :: IO () main = do print (len Null) -- exception thrown But just because you can doesn't mean you should. There is a necessary assumption here that you know what you are doing and aren't tossing complete garbage at the screen. With that assumption in force, it remains uncertain how these null cases are manifesting even if we assume a compiler that cannot determine null exception cases. What is a concrete example that we can run with to better understand your rejection?
- default-kramer 1y ago
  How many people actually write exhaustive tests for everything that could possibly be null? No one I've ever met in my mostly-C# career. I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I insist on writing new C# code with null-checking enabled which mostly solves the problem, but there's still plenty of code in the wild crashing on null bugs all the time.
  - 9rx 1y ago
    > How many people actually write exhaustive tests for everything that could possibly be null? Of those who are concerned about type theory? 99%. With a delusional 1% thinking that a gimped type system (read: insufficient for formal proofs) is some kind of magic that negates the need to write tests, somehow not noticing that many of the lessons on how to write good tests come from those language ecosystems (e.g. Haskell). > I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I don't think I've ever seen a null exception (or closest language analog) occur in production, and I spent a lot of years involved in projects that used dynamically typed languages even. I'd still love for someone to show actual code and associated tests to see how they ended up in that situation. The other commenter, despite being adamant, has become avoidant when it comes down to it.
    - mdaniel 1y ago
      > to see how they ended up in that situation The "how" is almost always lack of discipline (or as I sometimes couch it, "imagination") but usually shit like https://github.com/microsoft/SynapseML/issues/405#:~:text=clearly%20something%20went%20wrong https://github.com/microsoft/SynapseML/issues/405#:~:text=cl... I've had to learn in my career not to open other people's projects in a real IDE because of all the screaming it does about "value can be null"
      - 9rx 1y ago
        That's a tough bouncing ball to follow as it appears that the resolution was to upgrade to a newer version of a dependency, but if we look at that dependency, the fix seems to be found somewhere in this https://github.com/microsoft/LightGBM/compare/v2.2.1...v2.2.2 https://github.com/microsoft/LightGBM/compare/v2.2.1...v2.2.... There is admittedly a lot in the update and I may have simply missed it, but I don't see any modifications, even additions, to tests to denote recognition of the problem in the earlier version. Which, while not knowing much about the project, makes me think that there really isn't any meaningful testing going on. That maybe be interesting for what it is, I suppose, but not really in the vein of the discussion here about where one is using type systems and testing to overcome their personal limitations.
- mdaniel 1y ago
  Relevant: Google Cloud Incident Report – 2025-06-13 - https://news.ycombinator.com/item?id=44274563 https://news.ycombinator.com/item?id=44274563 - June, 2025 (220 comments)
  - 9rx 1y ago
    Not exactly as, in that case, they chose not to test the error condition for correct behaviour at all. The problem wasn't just the null pointer condition, but also that the coded behaviour under that state was just plain wrong from top to bottom. More careful use of the type system might have caught the null pointer case specifically, but the compiler still wouldn't have caught that they were doing the wrong thing beyond the null pointer error. In other words, the failure would have still occurred due to the next problem down the line from a problem that is impossible to catch with a partial type system. While a developer full of hubris who thinks they can do no wrong may let that slide, our discussion is specifically about a developer who fully understands his personal limitations. He recognizes that he needs the machine to hold his hand. While that includes leveraging types, he understands that the type system in any language he will use in the real world isn't expressive enough to cover all of the conditions necessary. Thus he will also write tests to fill in the gaps. Now, the premise proposed was that once you write the tests to cover that behaviour in order to fill in the gaps where the type system isn't expressive enough, you naturally also ensure that you don't end up with things like null pointer cases by way of the constraints of test execution. The parent rejected that notion, but it remains unclear why, and we don't yet have a concrete example showing how errant nulls "magically" appear under those conditions. "I don't need testing" isn't the same thing, and has nothing to do with the discussion taking place here.

by 1-more 1y ago

9rx 1y ago

Assume that there was room for null. What is your concrete example of where null becomes a problem in production, but goes unnoticed by your tests?

1-more 1y ago
In a world where I am writing a language with null and have half-implemented a grown up type system by checking for null everywhere and writing tests that try to call functions with null (EDIT: and I remembered to do all of that), I guess we could say that I'm at the same place I am right now. But right now I don't have to write defensive tests that include null. You're asking about a circumstance that's just very very different from the one I'm in.
- 9rx 1y ago
  > But right now I don't have to write defensive tests that include null. You seem to misunderstand. The question was centred around the fact that unexpected null cases end up being tested by virtue of you covering normal test cases due to the constraints on execution. Explicitly testing for null is something else. Something else I suggest unnecessary — at least where null is not actually part of the contract, which for the purposes of our discussion is the case. Again, we're specifically talking about the testing that is necessary to the extent of covering what is missing in the type system when you don't have a formal proof language in hand. > You're asking about a circumstance that's just very very different from the one I'm in. But one you've clearly had trouble with in the past given your claims that you weren't able to deal with it. Otherwise, how would you know? Perhaps I can phrase my request in another way: If null isn't expected in your codebase, and your testing over where the type system is lacking has covered your bases to know that the behaviour is as documented, where are the errant nulls going to magically appear from?
  - 1-more 1y ago
    Oh the errant nulls only ever came up when I was writing dynamically typed languages. Is that what you're asking about?
    - 9rx 1y ago
      Not specifically, but if that is where your example comes from, that’s fine. It’s all the same. What have you got?
      - 1-more 1y ago
        On a PHP project I ran into cases where I changed the shape of an object (associative array) I was passing around and forgot about one of the places I was using it. Didn’t turn into a production bug but still was the kind of thing I would rather be a squiggly line rather than remembering to rerun all the paths through the code. Didn’t help that we were testing by hand. Same thing on the front end in JS: change the shape of some record that has more places using it than I could remember. Better tests would have caught these. A compiler would be even better. FWIW I’ve written a lot of tests of code written in all of the languages I like. You absolutely need tests, you just don’t need them to be as paranoid when writing them.
        9rx 1y ago
        > A compiler would be even better. Right, but the condition here is that the languages that are expressive enough to negate the need for testing are, shall we say, unusable. In the real world people are going to be using, at best, languages with gimped type systems that still require testing to fill in the gaps. Given that, we're trying to understand your rejection of the premise that the tests you will write to fill in those gaps will also catch things like null exceptions in the due course of execution. It remains unclear where you think these errant nulls are magically coming from. I'm not convinced "Didn’t help that we were testing by hand.", "Better tests would have caught these." is that rejection. Those assertions, while no doubt applicable to your circumstances, is kind of like saying that static type systems don't help either because you can write something like this: data MaybeString = Null | Str String len :: MaybeString -> Int len (Str s) = length s main :: IO () main = do print (len Null) -- exception thrown But just because you can doesn't mean you should. There is a necessary assumption here that you know what you are doing and aren't tossing complete garbage at the screen. With that assumption in force, it remains uncertain how these null cases are manifesting even if we assume a compiler that cannot determine null exception cases. What is a concrete example that we can run with to better understand your rejection?
default-kramer 1y ago
How many people actually write exhaustive tests for everything that could possibly be null? No one I've ever met in my mostly-C# career. I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I insist on writing new C# code with null-checking enabled which mostly solves the problem, but there's still plenty of code in the wild crashing on null bugs all the time.
- 9rx 1y ago
  > How many people actually write exhaustive tests for everything that could possibly be null? Of those who are concerned about type theory? 99%. With a delusional 1% thinking that a gimped type system (read: insufficient for formal proofs) is some kind of magic that negates the need to write tests, somehow not noticing that many of the lessons on how to write good tests come from those language ecosystems (e.g. Haskell). > I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I don't think I've ever seen a null exception (or closest language analog) occur in production, and I spent a lot of years involved in projects that used dynamically typed languages even. I'd still love for someone to show actual code and associated tests to see how they ended up in that situation. The other commenter, despite being adamant, has become avoidant when it comes down to it.
  - mdaniel 1y ago
    > to see how they ended up in that situation The "how" is almost always lack of discipline (or as I sometimes couch it, "imagination") but usually shit like https://github.com/microsoft/SynapseML/issues/405#:~:text=clearly%20something%20went%20wrong https://github.com/microsoft/SynapseML/issues/405#:~:text=cl... I've had to learn in my career not to open other people's projects in a real IDE because of all the screaming it does about "value can be null"
    - 9rx 1y ago
      That's a tough bouncing ball to follow as it appears that the resolution was to upgrade to a newer version of a dependency, but if we look at that dependency, the fix seems to be found somewhere in this https://github.com/microsoft/LightGBM/compare/v2.2.1...v2.2.2 https://github.com/microsoft/LightGBM/compare/v2.2.1...v2.2.... There is admittedly a lot in the update and I may have simply missed it, but I don't see any modifications, even additions, to tests to denote recognition of the problem in the earlier version. Which, while not knowing much about the project, makes me think that there really isn't any meaningful testing going on. That maybe be interesting for what it is, I suppose, but not really in the vein of the discussion here about where one is using type systems and testing to overcome their personal limitations.
mdaniel 1y ago
Relevant: Google Cloud Incident Report – 2025-06-13 - https://news.ycombinator.com/item?id=44274563 https://news.ycombinator.com/item?id=44274563 - June, 2025 (220 comments)
- 9rx 1y ago
  Not exactly as, in that case, they chose not to test the error condition for correct behaviour at all. The problem wasn't just the null pointer condition, but also that the coded behaviour under that state was just plain wrong from top to bottom. More careful use of the type system might have caught the null pointer case specifically, but the compiler still wouldn't have caught that they were doing the wrong thing beyond the null pointer error. In other words, the failure would have still occurred due to the next problem down the line from a problem that is impossible to catch with a partial type system. While a developer full of hubris who thinks they can do no wrong may let that slide, our discussion is specifically about a developer who fully understands his personal limitations. He recognizes that he needs the machine to hold his hand. While that includes leveraging types, he understands that the type system in any language he will use in the real world isn't expressive enough to cover all of the conditions necessary. Thus he will also write tests to fill in the gaps. Now, the premise proposed was that once you write the tests to cover that behaviour in order to fill in the gaps where the type system isn't expressive enough, you naturally also ensure that you don't end up with things like null pointer cases by way of the constraints of test execution. The parent rejected that notion, but it remains unclear why, and we don't yet have a concrete example showing how errant nulls "magically" appear under those conditions. "I don't need testing" isn't the same thing, and has nothing to do with the discussion taking place here.