5 ms·

Your first paragraph totally resonated. I've been thinking about this problem for several years. However, my approach is diametrically opposed to yours. I think

Your first paragraph totally resonated. I've been thinking about this problem for several years. However, my approach is diametrically opposed to yours. I think our problems in software all stem from focussing on the code as the tangible artifact to maintain control over. We should instead be focusing on the space of possible inputs that the code is intended to work for. This is something you can't deduce from the code (and automatically using the computer to deduce it, well forget about it), it requires cooperation with the original author to present things in a way that makes the state space more explicit. This is why I love projects with lots of tests. I can't be bothered to analyze static code structure, either manually (what people call 'reading'[1]) or automatically. Just show me how the program is supposed to run in all the different situations that you've considered. Let me change it and rerun the tests to find out if I broke something. Modern programming practice emphasizes tests, which is great. However, not all kinds of tests can be written so far. So we end up doing manual certification work everytime we release or publish a new version of software, for performance, fault tolerance, etc. I want to make it all automatic. Some links about my project in case you'd like to learn more: http://akkartik.name/about; http://akkartik.name/about; http://github.com/akkartik/mu#readme http://github.com/akkartik/mu#readme. I'd love to hear your thoughts, either here or over email (address in profile). [1] http://akkartik.name/post/readable-bad http://akkartik.name/post/readable-bad

reledi 11y ago
Tests significantly help with understanding a codebase and safely making modifications. Unit tests describe the behavior and usage of the individual systems, while integration tests describe business use cases in core workflows (usually just the happy paths). Integration tests should ideally be written in a gherkin style with feature description and acceptance criteria clearly outlined. Tests should be optimized for readability foremost. For example, most people try and make their tests DRY (which is an abuse of DRY because it should only apply to concepts, not code, but I digress) while they should instead be making them DAMP. Each test should be able tell a story without jumping up and down around the file and outside the file. It's a lot more dangerous to misunderstand a test than it is to have a little bit of code duplication in your tests. When I come across a project that has no documentation, I look at the tests. This isn't an excuse not to write proper documentation of course.
- crdoconnor 11y ago
  >Tests should be optimized for readability foremost. For example, most people try and make their tests DRY (which is an abuse of DRY because it should only apply to concepts, not code, but I digress) That's wrong on so many levels. DRY applies to code. >while they should instead be making them DAMP. Each test should be able tell a story without jumping up and down around the file and outside the file. It's a lot more dangerous to misunderstand a test than it is to have a little bit of code duplication in your tests. This is a problem with Gherkin. Gherkin is not particularly well suited to making tests that are both DRY and readable due to its syntax.
  - akkartik 11y ago
    > which is an abuse of DRY because it should only apply to concepts, not code As a 'copyista', I would change this to "DRY should only apply to production code, not tests." It wasn't clear from your reply if you agree or disagree. A couple of great recent links about this IMO under-discussed topic: http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstractio... http://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to http://programmingisterrible.com/post/139222674273/write-cod... http://bravenewgeek.com/abstraction-considered-harmful http://bravenewgeek.com/abstraction-considered-harmful
crdoconnor 11y ago
>Modern programming practice emphasizes tests, which is great. However, not all kinds of tests can be written so far. I don't think this is true in practice. It's just that a lot of scenarios people need tests for aren't easy to replicate and most people end up not bothering and relying upon manual checking instead.
- akkartik 11y ago
  Are you aware of any projects where this is not true? Perhaps Sqlite3 (http://www.sqlite.org/testing.html http://www.sqlite.org/testing.html). But in my personal experience it's not "most people" but "all people". I'd love to hear contradicting data or anecdotes.
  - crdoconnor 11y ago
    I've worked on projects before which initially had no tests, then some tests, then progressively more and more sophisticated tests. Each time the harness became capable enough to replicate a certain type of scenario it didn't take long before those kinds of bugs dried up. The bugs then always nearly always migrated to the areas the test harness couldn't easily create scenarios for - whether that was interactions with crazy external APIs, odd timezones or weird quirky browsers or whatever. Continually modifying the test harness to be made capable of testing bugs reported from production was often a huge amount of work but it paid off handsomely.
jacques_chester 11y ago
> So we end up doing manual certification work everytime we release or publish a new version of software, for performance, fault tolerance, etc. What can humans do, sitting at a computer, that software can't be written to do? Before other nitpickers derail this question, I'm talking about validating a software product, not hard AI.
- akkartik 11y ago
  We don't need to think about AI. I'd rephrase your question like this: What can the author do, sitting at a computer, that later readers can't undo? When phrased that way the answer is obvious: a lot. Like multiply two large numbers together, or obfuscate a program to make its meaning less clear. Both are in principle possible to undo, but the "load factor" can be arbitrarily large, making the effort for the 'reader' entirely uneconomic. If even a human can't undo some things that the author did, how can we expect the computer to do so?
  - jacques_chester 11y ago
    I'm afraid I don't follow you at all.
    - akkartik 11y ago
      Sorry, I misunderstood what you were replying to. I'd change my answer to: nothing at all in the context of manual testing. And yet in practice the answer so far seems different from our in-principle argument. I'm trying to make principle and practice line up better.

reledi 11y ago

Tests significantly help with understanding a codebase and safely making modifications. Unit tests describe the behavior and usage of the individual systems, while integration tests describe business use cases in core workflows (usually just the happy paths). Integration tests should ideally be written in a gherkin style with feature description and acceptance criteria clearly outlined. Tests should be optimized for readability foremost. For example, most people try and make their tests DRY (which is an abuse of DRY because it should only apply to concepts, not code, but I digress) while they should instead be making them DAMP. Each test should be able tell a story without jumping up and down around the file and outside the file. It's a lot more dangerous to misunderstand a test than it is to have a little bit of code duplication in your tests. When I come across a project that has no documentation, I look at the tests. This isn't an excuse not to write proper documentation of course.

crdoconnor 11y ago
>Tests should be optimized for readability foremost. For example, most people try and make their tests DRY (which is an abuse of DRY because it should only apply to concepts, not code, but I digress) That's wrong on so many levels. DRY applies to code. >while they should instead be making them DAMP. Each test should be able tell a story without jumping up and down around the file and outside the file. It's a lot more dangerous to misunderstand a test than it is to have a little bit of code duplication in your tests. This is a problem with Gherkin. Gherkin is not particularly well suited to making tests that are both DRY and readable due to its syntax.
- akkartik 11y ago
  > which is an abuse of DRY because it should only apply to concepts, not code As a 'copyista', I would change this to "DRY should only apply to production code, not tests." It wasn't clear from your reply if you agree or disagree. A couple of great recent links about this IMO under-discussed topic: http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstractio... http://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to http://programmingisterrible.com/post/139222674273/write-cod... http://bravenewgeek.com/abstraction-considered-harmful http://bravenewgeek.com/abstraction-considered-harmful

crdoconnor 11y ago

>Modern programming practice emphasizes tests, which is great. However, not all kinds of tests can be written so far. I don't think this is true in practice. It's just that a lot of scenarios people need tests for aren't easy to replicate and most people end up not bothering and relying upon manual checking instead.

akkartik 11y ago
Are you aware of any projects where this is not true? Perhaps Sqlite3 (http://www.sqlite.org/testing.html http://www.sqlite.org/testing.html). But in my personal experience it's not "most people" but "all people". I'd love to hear contradicting data or anecdotes.
- crdoconnor 11y ago
  I've worked on projects before which initially had no tests, then some tests, then progressively more and more sophisticated tests. Each time the harness became capable enough to replicate a certain type of scenario it didn't take long before those kinds of bugs dried up. The bugs then always nearly always migrated to the areas the test harness couldn't easily create scenarios for - whether that was interactions with crazy external APIs, odd timezones or weird quirky browsers or whatever. Continually modifying the test harness to be made capable of testing bugs reported from production was often a huge amount of work but it paid off handsomely.

jacques_chester 11y ago

> So we end up doing manual certification work everytime we release or publish a new version of software, for performance, fault tolerance, etc. What can humans do, sitting at a computer, that software can't be written to do? Before other nitpickers derail this question, I'm talking about validating a software product, not hard AI.

akkartik 11y ago
We don't need to think about AI. I'd rephrase your question like this: What can the author do, sitting at a computer, that later readers can't undo? When phrased that way the answer is obvious: a lot. Like multiply two large numbers together, or obfuscate a program to make its meaning less clear. Both are in principle possible to undo, but the "load factor" can be arbitrarily large, making the effort for the 'reader' entirely uneconomic. If even a human can't undo some things that the author did, how can we expect the computer to do so?
- jacques_chester 11y ago
  I'm afraid I don't follow you at all.
  - akkartik 11y ago
    Sorry, I misunderstood what you were replying to. I'd change my answer to: nothing at all in the context of manual testing. And yet in practice the answer so far seems different from our in-principle argument. I'm trying to make principle and practice line up better.