4 ms·

> When most people start as programmers, they write a big 1000 line file that should completely solve the problem. Then they run it, realize it was very wrong,

> When most people start as programmers, they write a big 1000 line file that should completely solve the problem. Then they run it, realize it was very wrong, and spend hours debugging it. No, I start line by line, and tend to be thorough. When I start out there is zero complexity, everything I add to it I don't add blindly, with a "rough idea" of what it does. I know exactly what it does. I know where user input and other code ends, and my code begins, at the least, and how I normalize things that cross the threshold. Yes, I'm talking about a solo dev making little things for their own needs, sure. I have the luxury of being as thorough and slow as I want, too. It's not lazyness, the idea of tests actually excited me when I first heard about it. But for example, when I write a particle system, how would I test that? Do I test all combinations of parameters and then compare them with images, which takes a million years? How do I generate the images, with another particle system? How do I test that, in turn? It really does strike me as a "now you have two problems" kind of deal in that situation. In others, the code is just too trivial and unchanging to write a test. What's a "real" application that has good and complete tests? Something like the GIMP, or Blender, or an audio editor, or a complex game? I wouldn't know where to look, and the tests I saw when browsing random github repos often seemed too simple to be useful, often just covering some "token input". But what if your "escape string" function has a bug that kicks in at exactly 15 characters? What if your "get user ID from username" has a bug that ONLY happens when the string "whoops" is in the username? Fuzz everything? Who really does that for applications without serious security implications?

voodootrucker 8y ago
I'd love to know how to test graphics and audio programs, frankly. TDD works very well for things with a defined input and output, including: 1. compilers 2. databases 3. all web apps (output is the DOM) In regards to the escape string, if there was a bug at exactly 15 characters, there must be a branch somewhere that isn't being hit, and it should be pretty easy using code coverage reports to find it, and generate the correct input sequence to exercise it. When you get to fuzzing, you're hitting the edge of state-of-the-art. There is good research going on at automating some of that: http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/
- mdpopescu 8y ago
  What I do for graphics (never had to test audio programs) is to make the actual graphics code as simple as possible - no logic, very few lines: just take this thing and put it on screen. This way, I can inspect it visually, test it manually once and be reasonably certain that it will work. (Which can be incorrect, of course, but it's relatively rare. Lost a contract because of it once, so it can also be painful :D) Put all the logic in a class that can be tested and that returns something that can be used in the "get this, put it on screen" part mentioned above.
bcbrown 8y ago
> But for example, when I write a particle system, how would I test that? Do I test all combinations of parameters and then compare them with images, which takes a million years? Look up pairwise-independent combinatorial testing: https://github.com/Microsoft/pict https://github.com/Microsoft/pict > But what if your "escape string" function has a bug that kicks in at exactly 15 characters? Equivalence classes: Analyze which groups of inputs all exercise the same codepath, and test each group once. Boundary value testing: Anywhere there's a boundary between two different types of behavior, test the boundary, one less than the boundary, and one more than the boundary.
jacques_chester 8y ago
> No, I start line by line, and tend to be thorough. When I start out there is zero complexity, everything I add to it I don't add blindly, with a "rough idea" of what it does. I know exactly what it does. I know where user input and other code ends, and my code begins, at the least, and how I normalize things that cross the threshold. TDD isn't explosively incompatible with being careful. > What's a "real" application that has good and complete tests? Cloud Foundry has 4 or 5 million lines of Golang, about a million lines or Ruby and I think half a million lines of Java. Pretty much all test-driven. It's been a while since I checked the SLOC figures. Kubernetes is of similar (now greater and definitely faster-growing) magnitude and the community has tooling to track coverage (I think the target is > 90%), though TDD is not the norm. I've personally worked on maybe two dozen systems where TDD was the ordinary practice, both in Cloud Foundry and for clients. But it's a rare practice. Just on the numbers, almost nobody practices TDD. I'm aware that this argument is a kissing cousin to No True Scotsman, but there it is.
je42 8y ago
> But for example, when I write a particle system, how would I test that? Do I test all combinations of parameters and then compare them with images, which takes a million years? A couple of assumptions: - your particle system is not one monolithic function, right ? - you also didn't just write it in a single pass and it had all features. So: (i am applying the test pyramid here ( https://martinfowler.com/bliki/TestPyramid.html https://martinfowler.com/bliki/TestPyramid.html ) - you start with a very simple (system) test that describes the minimal particle system. something like: given a particle system with particle count 1 and an emitter with probability 1 and velocity (1,0,0) and random seed 0; then after 1 step, there is a exactly one particle with velocity 1,0,0 at position 0. - and you make this test compile and run and eventually pass. - eventually you split out different functions; you develop also a test first. and then write the new function. and then change the original large function of the particle system to use the smaller functions. :) - when it comes to rendering, you probably never want to write a system test that renders a full blown particle system with random paratemers ( because the test will likely be unstable unless you have a good idea about to keep it stable ). - what is easier is to take the rendering functions and let them render aspects separately, i.e it renders a single particle at a known position. and you check the rendered image if it is correct; this because pretty simple with the image resolution is for example is just a pixel or 9 and that you can compare with a single static value. color animation can be tested that way as well. etc. - thus in general even a particle system can be tested; break the system into small functions and test them separate and fully. - then integration test the function in concert. use stable test scenarios. - test manually; and when you find a bug (and you will). write first a test for the simple buggy function to expose the issue on unittest level. and then you decide if you want to add an extra scenario on the high levels.

voodootrucker 8y ago

I'd love to know how to test graphics and audio programs, frankly. TDD works very well for things with a defined input and output, including: 1. compilers 2. databases 3. all web apps (output is the DOM) In regards to the escape string, if there was a bug at exactly 15 characters, there must be a branch somewhere that isn't being hit, and it should be pretty easy using code coverage reports to find it, and generate the correct input sequence to exercise it. When you get to fuzzing, you're hitting the edge of state-of-the-art. There is good research going on at automating some of that: http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/

mdpopescu 8y ago
What I do for graphics (never had to test audio programs) is to make the actual graphics code as simple as possible - no logic, very few lines: just take this thing and put it on screen. This way, I can inspect it visually, test it manually once and be reasonably certain that it will work. (Which can be incorrect, of course, but it's relatively rare. Lost a contract because of it once, so it can also be painful :D) Put all the logic in a class that can be tested and that returns something that can be used in the "get this, put it on screen" part mentioned above.

bcbrown 8y ago

> But for example, when I write a particle system, how would I test that? Do I test all combinations of parameters and then compare them with images, which takes a million years? Look up pairwise-independent combinatorial testing: https://github.com/Microsoft/pict https://github.com/Microsoft/pict > But what if your "escape string" function has a bug that kicks in at exactly 15 characters? Equivalence classes: Analyze which groups of inputs all exercise the same codepath, and test each group once. Boundary value testing: Anywhere there's a boundary between two different types of behavior, test the boundary, one less than the boundary, and one more than the boundary.

jacques_chester 8y ago

> No, I start line by line, and tend to be thorough. When I start out there is zero complexity, everything I add to it I don't add blindly, with a "rough idea" of what it does. I know exactly what it does. I know where user input and other code ends, and my code begins, at the least, and how I normalize things that cross the threshold. TDD isn't explosively incompatible with being careful. > What's a "real" application that has good and complete tests? Cloud Foundry has 4 or 5 million lines of Golang, about a million lines or Ruby and I think half a million lines of Java. Pretty much all test-driven. It's been a while since I checked the SLOC figures. Kubernetes is of similar (now greater and definitely faster-growing) magnitude and the community has tooling to track coverage (I think the target is > 90%), though TDD is not the norm. I've personally worked on maybe two dozen systems where TDD was the ordinary practice, both in Cloud Foundry and for clients. But it's a rare practice. Just on the numbers, almost nobody practices TDD. I'm aware that this argument is a kissing cousin to No True Scotsman, but there it is.

je42 8y ago

> But for example, when I write a particle system, how would I test that? Do I test all combinations of parameters and then compare them with images, which takes a million years? A couple of assumptions: - your particle system is not one monolithic function, right ? - you also didn't just write it in a single pass and it had all features. So: (i am applying the test pyramid here ( https://martinfowler.com/bliki/TestPyramid.html https://martinfowler.com/bliki/TestPyramid.html ) - you start with a very simple (system) test that describes the minimal particle system. something like: given a particle system with particle count 1 and an emitter with probability 1 and velocity (1,0,0) and random seed 0; then after 1 step, there is a exactly one particle with velocity 1,0,0 at position 0. - and you make this test compile and run and eventually pass. - eventually you split out different functions; you develop also a test first. and then write the new function. and then change the original large function of the particle system to use the smaller functions. :) - when it comes to rendering, you probably never want to write a system test that renders a full blown particle system with random paratemers ( because the test will likely be unstable unless you have a good idea about to keep it stable ). - what is easier is to take the rendering functions and let them render aspects separately, i.e it renders a single particle at a known position. and you check the rendered image if it is correct; this because pretty simple with the image resolution is for example is just a pixel or 9 and that you can compare with a single static value. color animation can be tested that way as well. etc. - thus in general even a particle system can be tested; break the system into small functions and test them separate and fully. - then integration test the function in concert. use stable test scenarios. - test manually; and when you find a bug (and you will). write first a test for the simple buggy function to expose the issue on unittest level. and then you decide if you want to add an extra scenario on the high levels.