4 ms·

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under variou

Which is funny as the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions. It is documentation. If you have to document your documentation, you might be missing something fundamental in how you are writing your first order documentation. Not to mention that in doing so you defeat the reason for writing your documentation in an executable form (to be able to automatically validate that the documentation is true).

hulitu 4y ago
The test plan is the documentation. That people are cutting corners is unfortunate. A test must be reproduceable. If it is not, is not a test.
vincnetas 4y ago
So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable document where intentions and expectations are specified it's much easier to come to common (working) solution. Not all languages have capability to express complex intentions so code as documentation does not work most of the time.
- alpaca128 4y ago
  Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking. Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice that's the kind of doc I tend to have the best experience with. For example the Rust stdlib docs are auto-generated but the language also supports notes and (automatically unit-tested) examples in docstrings which means the API docs are filled with explanations & examples and mentions what assumptions are made about inputs.
  - hitchstory 4y ago
    I built this framework coz while I didn't believe in "code as documentation" I did believe that example based specifications, tests and documentation were all sort of the same thing (triality): https://hitchdev.com/hitchstory/ https://hitchdev.com/hitchstory/ The difference between this and behave/cucumber is that the A) specification language allows for more complex representations and B) there's a templating step to generate readable documentation.
  - eevilspock 4y ago
    I'm not sure if you're saying that rust stdlib docs do this but documentation where all the examples are themselves runnable as tests and included in the CI test suite solves so many problems.
    - steveklabnik 4y ago
      Not just the stdlib, the documentation tool itself supports this, so it's the case for any Rust package that writes documentation.
- magicalhippo 4y ago
  Code is documentation, but it only tells you a part of the story. Good comments can explain why, but without writing comment essays it's usually not sufficient. And, as you note, when integrating systems you need more than just the code and comments, since the code might not even be written with the other system in mind.
- kapep 4y ago
  I think they mean "test code is documentation". For example if there's a unit test that expects an error for a certain input, it serves as documentation that this kind of input is not allowed. It not always feasible to document every little edge case in natural language and keep it in sync with your code. If you "document" edge cases as tests, they _have_ to be in sync with your code. It shouldn't replace traditional documentation though and is better suited for internal components and not for public API.
- randomdata 4y ago
  No. Your documentation is your documentation. It documents your code. It is not your code. If the documentation can also be interpreted by machine to validate what it claims is true you have a nice side benefit, but not the reason for writing your documentation.
- pjerem 4y ago
  Same. I’m sick of people escaping writing documentation by saying that "code is the doc" and in the meantime, writing unreadable code abstracted over dozens of code files. They almost convinced me somewhere in my career. But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. If your "code is doc", it should be extremely easy to add a little sentence above your method to explain what it does. And no, doc doesn’t stale. If your documentation isn’t up to what your function does, it’s probably because you should have written a brand new function instead of changing a function’s behavior.
  - arkh 4y ago
    > I’m sick of people escaping writing documentation First level of documentation would be specifications. Which can then be used to write tests. But in many, many shops "Agile" means "we don't do specs anymore, woohoo!".
  - rightbyte 4y ago
    Not writing docs is a sort of hazing. Also job security.
  - saiya-jin 4y ago
    > But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. I completely agree with you, ie right now doing bunch of data migration code that is awful 200 lines on first look, but does quite clever transformations, handles various data corner cases, manages lots of threads, is already quite optimized (had 30x speed increase just over last week's state and not yet done with it) etc. and... is full of little green one-liners explaining why certain logic is happening, why at given place, and not elsewhere, and how it helps later in the code. Its even one-off migration, and its mostly for me only. But I still put comments in, have enough experience to know I will keep using those comments in further optimizations, and I know by heart that many one-off efforts end up being re-used later. Code dense with logic shouldn't require you to re-read it all to have constant full mental model of it and all its branching and possibilities just because you want to tweak it a bit. The important point is to evolve those comments with code, otherwise they become worse than no comments at all. This is where most folks hit the wall - they are simply too lazy or undisciplined for that.
charcircuit 4y ago
>the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions No, the point of automated testing is to verify that what is under test behaves correctly and to be able to scale this verification cheaper than having humans do it. Documenting what it verifies and under what conditions is just a side effect.
- randomdata 4y ago
  That is a common falsehood. Testing does not verify that the code under test behaves correctly. It only verifies that the what the documentation asserts correctly matches what the code does. Indeed, enabling the machine to verify that the documentation is true is cheaper than having humans do it. Also less error prone. Humans are notoriously bad at keeping documentation properly up to date.
jmcomets 4y ago
Disregarding the "code is doc" position, it's still common to have an overview or index for documentation, which points readers in the right direction instead of dumping pages of detailed docs on them. Now, you could also have a well organized test suite that goes from most obvious to most detailed, split into sections for each use-case, but this sounds a lot more tedious than "write a one-line comment describing the unit test".

hulitu 4y ago

The test plan is the documentation. That people are cutting corners is unfortunate. A test must be reproduceable. If it is not, is not a test.

vincnetas 4y ago

So i understand correctly, that your position is "code is the documentation"? Over time im inclined to value human written documentation. Especially when things involve integrations of multiple systems. I had real cases, where two parties point at code and say their code is correct. And in isolation code looks correct. But when time comes to integrate these systems. It breaks. And then if you have human readable document where intentions and expectations are specified it's much easier to come to common (working) solution. Not all languages have capability to express complex intentions so code as documentation does not work most of the time.

alpaca128 4y ago
Code as documentation feels like a good idea because code is the only reliable source of truth. But it also assumes that code can comprehensively express all assumptions and other info, which sounds more like wishful thinking. Auto-generated API docs combined with handwritten documentation that covers what can't be expressed in code and includes some useful examples seems like the right approach to me. In practice that's the kind of doc I tend to have the best experience with. For example the Rust stdlib docs are auto-generated but the language also supports notes and (automatically unit-tested) examples in docstrings which means the API docs are filled with explanations & examples and mentions what assumptions are made about inputs.
- hitchstory 4y ago
  I built this framework coz while I didn't believe in "code as documentation" I did believe that example based specifications, tests and documentation were all sort of the same thing (triality): https://hitchdev.com/hitchstory/ https://hitchdev.com/hitchstory/ The difference between this and behave/cucumber is that the A) specification language allows for more complex representations and B) there's a templating step to generate readable documentation.
- eevilspock 4y ago
  I'm not sure if you're saying that rust stdlib docs do this but documentation where all the examples are themselves runnable as tests and included in the CI test suite solves so many problems.
  - steveklabnik 4y ago
    Not just the stdlib, the documentation tool itself supports this, so it's the case for any Rust package that writes documentation.
magicalhippo 4y ago
Code is documentation, but it only tells you a part of the story. Good comments can explain why, but without writing comment essays it's usually not sufficient. And, as you note, when integrating systems you need more than just the code and comments, since the code might not even be written with the other system in mind.
kapep 4y ago
I think they mean "test code is documentation". For example if there's a unit test that expects an error for a certain input, it serves as documentation that this kind of input is not allowed. It not always feasible to document every little edge case in natural language and keep it in sync with your code. If you "document" edge cases as tests, they _have_ to be in sync with your code. It shouldn't replace traditional documentation though and is better suited for internal components and not for public API.
randomdata 4y ago
No. Your documentation is your documentation. It documents your code. It is not your code. If the documentation can also be interpreted by machine to validate what it claims is true you have a nice side benefit, but not the reason for writing your documentation.
pjerem 4y ago
Same. I’m sick of people escaping writing documentation by saying that "code is the doc" and in the meantime, writing unreadable code abstracted over dozens of code files. They almost convinced me somewhere in my career. But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. If your "code is doc", it should be extremely easy to add a little sentence above your method to explain what it does. And no, doc doesn’t stale. If your documentation isn’t up to what your function does, it’s probably because you should have written a brand new function instead of changing a function’s behavior.
- arkh 4y ago
  > I’m sick of people escaping writing documentation First level of documentation would be specifications. Which can then be used to write tests. But in many, many shops "Agile" means "we don't do specs anymore, woohoo!".
- rightbyte 4y ago
  Not writing docs is a sort of hazing. Also job security.
- saiya-jin 4y ago
  > But the hard truth I learnt is that most people are saying this because they aren’t capable of verbalizing what they are programming. I completely agree with you, ie right now doing bunch of data migration code that is awful 200 lines on first look, but does quite clever transformations, handles various data corner cases, manages lots of threads, is already quite optimized (had 30x speed increase just over last week's state and not yet done with it) etc. and... is full of little green one-liners explaining why certain logic is happening, why at given place, and not elsewhere, and how it helps later in the code. Its even one-off migration, and its mostly for me only. But I still put comments in, have enough experience to know I will keep using those comments in further optimizations, and I know by heart that many one-off efforts end up being re-used later. Code dense with logic shouldn't require you to re-read it all to have constant full mental model of it and all its branching and possibilities just because you want to tweak it a bit. The important point is to evolve those comments with code, otherwise they become worse than no comments at all. This is where most folks hit the wall - they are simply too lazy or undisciplined for that.

charcircuit 4y ago

>the purpose of testing is to explain to other other developers what the code under test assumes and what should be expected of it under various conditions No, the point of automated testing is to verify that what is under test behaves correctly and to be able to scale this verification cheaper than having humans do it. Documenting what it verifies and under what conditions is just a side effect.

randomdata 4y ago
That is a common falsehood. Testing does not verify that the code under test behaves correctly. It only verifies that the what the documentation asserts correctly matches what the code does. Indeed, enabling the machine to verify that the documentation is true is cheaper than having humans do it. Also less error prone. Humans are notoriously bad at keeping documentation properly up to date.

jmcomets 4y ago

Disregarding the "code is doc" position, it's still common to have an overview or index for documentation, which points readers in the right direction instead of dumping pages of detailed docs on them. Now, you could also have a well organized test suite that goes from most obvious to most detailed, split into sections for each use-case, but this sounds a lot more tedious than "write a one-line comment describing the unit test".