4 ms·

Until you mentioned him I had never heard of this Jonathan Blow character. The thing is both of these people seem to do game development. If you write code that

by cjfd 7y ago

Until you mentioned him I had never heard of this Jonathan Blow character. The thing is both of these people seem to do game development. If you write code that is close to the hardware and needs every ounce of performance there might be a point to this. It would make applying TDD both harder to pull off and less beneficial. When one is writing complex logic that is supposed to receive both frequent new features and keep working correctly there is nothing that beats TDD. Let me attempt to compare this to bowling even though I know nothing about bowling. If you just occasionally like to go out bowling you can use whatever technique you like and you will get better if you just bowl a bit more often. If you want to play bowling on the national level you may at some point have to drop the technique that you made up yourself when you started out. You need to go through the uncomfortable stage of learning a new an better technique and temporarily be worse at it. I think the people not doing TDD are just refusing to go through this because there is not much of an objective standard of quality for developers. We are in a field where code that lacks any sort of quality is standard and where breaking features three times a week is standard. TDD will fix these things.

3fe9a03ccd14ca5 7y ago
TDD works great when the requirements are known and there’s little mystery. Waterfall methods oddly enough lends themselves well for TDD, which is kind of associated with hip styles like agile. The reality is most projects start as exploration. What you’re thinking might not even be possible. Can you even approximate the classes you’ll need yet? For these scenarios TDD fits in awkwardly. I prefer to do a little back and forth. A little dev. When things are looking stable, a little resting. Towards the end a lot of testing.
- cjfd 7y ago
  There is some point to your comment but also something that seem to me to be misunderstanding. Also, part of it seems self-contradictory to me. There can indeed be exploration stages where TDD does not help that much. E.g, if you are going to interface with a piece of hardware you have never seen before and you need to get some sort of working communication going there is much point to what you say. As soon as you know what sequence of commands makes the piece of hardware work one has arrived in the territory where TDD is highly beneficial. You can then write a test where you require that your software indeed produces the correct sequence of commands and after that you can refactor your exploratory code while having to check that it still works together with the hardware much less often then otherwise would be needed. The remark about TDD and waterfall seems much less true to me. In my original post I wrote 'both frequent new features and keep working correctly'. That does not describe waterfall at all. In fact, and this is where you messages seems to get a bit self-contradictory, your post actually sounds much more waterfally to me than mine. It contains the words 'Towards the end'. It is waterfall projects that have an end that was envisioned right from the start. Whatever 'agile' means, it seems to have the feature that we never stop adding features and that stuff is supposed to be in a working state in between adding these features. So, if one is actually doing that there are dragons in the sentence 'Towards the end a lot of testing' because it actually turns into 'always lots of testing' which costs lots of time. In case of the hardware example it would also require that every developer has a species of that hardware on his desk next to his computer. As I happen to work in a field where we interface with hardware that is three stories high, we should realize that this is not always practical. One misunderstanding that might be going on is that perhaps you are assuming that in TDD one would test just one class at a time. I generally prefer to test a set of classes, i.e., a subsystem of a whole program. In that case the test are about properties of the program that, in many cases, the user could recognize as beneficial. There should be less churn at that level. I suppose some people would call this behaviour driven development instead but I am not so very married to terminology and will just keep calling it TDD. Testing individual classes can also be helpful but if they contain rather simple logic I am not sure writing tests for that is so very helpful. At some point would just be testing whether the processor really is capable of copying values which is not that interesting.
- ellius 7y ago
  I've found the opposite. When I start to write a test, I often realize I don't know exactly what I need to do because the requirements are unclear or somehow conflict with some other assumption of the system. That realization prompts me to go talk to my product owner, which forces them to clarify their requirements and target a more coherent idea of what the system should be.
gfodor 7y ago
I think as a first order approximation the level to which you can deploy TDD is inversely proportional to how difficult your problem is with regards to computational complexity. The most TDD’able code is a freshman level CS 100 programming assignment. The least TDD’able are things like real time simulations, numerical computing, machine learning, etc. In the latter case, you can TDD the boring parts (reading inputs, simple pure functions, etc) but TDD breaks down quickly if you have a huge state space, side effects, complex outputs, etc.
- cjfd 7y ago
  That depends. A problem may be quite difficult in computational complexity but if you can find simpler less computational complex version of it that can be used for automated tests that can be quick. E.g., test your neural network training procedure with a smaller network which will give a result that performs worse and after that use the thus obtained program on a with a bigger neural net.
  - closeparen 7y ago
    It's not about runtime performance. It's about how well you can actually assess your code by matching small and well-defined inputs to small and well-defined outputs. LeetCode / CS101 problems are almost always like that. In the real world, it's also useful to separate out any tricky "policy" logic into nicely testable pure functions. But you are still going to have a bunch of side-effect munging left over, and unit testing is terrible at that. Tests that only assess their own mocks are a waste of time.
    - cjfd 7y ago
      Could you give an example of situations where there would not be small and well-defined inputs and outputs because this certainly does not sound like a common situation to me? And what do you mean by 'Tests that only asses their own mocks are a waste of time.' Certainly, many side effects can very effecitively be checked by mocks. Now, it could certainly get much more difficult if the side effects were very large and hard to define as you said in the paragraph before that, but I have some difficulty picturing that as a common situation.
dakom 7y ago
> If you write code that is close to the hardware and needs every ounce of performance there might be a point to this. It would make applying TDD both harder to pull off and less beneficial. It's not really about being close to the hardware[0] - there's many other things that are unrelated to low-level performance tweaks that make TDD or even unit tests a waste of time for experienced game developers. When someone like Blow or Carmack sees an automated playtest fail, they have a honed intuition for what caused it. This intuition was built up over many years of hard-earned experience. They simply don't need the lower level unit test that says "this matrix multiplication math is wrong" - they'll see a collision fail or a weird scaling thing and _know_ right off the bat the candidates in the codebase where things went wonky and fix it before you can even say "look at the logs" :) Since they get the same results without having to write unit tests everywhere, and especially without writing tests first, it's simply an unnecessary overhead to make them write/maintain it. I don't think anyone would argue TDD and/or unit tests are valueless, it's just a cost/benefit question. Pretty sure they'd use any technique where the benefit outweighs the cost- and for them, for many reasons in many varied scenarios, it just doesn't. [0] actually, if anything, I'd bet that the stuff that's really close to the hardware like the core engine stuff (math, memory management, etc.) is the one place they would consider having TDD and/or more unit tests!

by cjfd 7y ago

3fe9a03ccd14ca5 7y ago

TDD works great when the requirements are known and there’s little mystery. Waterfall methods oddly enough lends themselves well for TDD, which is kind of associated with hip styles like agile. The reality is most projects start as exploration. What you’re thinking might not even be possible. Can you even approximate the classes you’ll need yet? For these scenarios TDD fits in awkwardly. I prefer to do a little back and forth. A little dev. When things are looking stable, a little resting. Towards the end a lot of testing.

cjfd 7y ago
There is some point to your comment but also something that seem to me to be misunderstanding. Also, part of it seems self-contradictory to me. There can indeed be exploration stages where TDD does not help that much. E.g, if you are going to interface with a piece of hardware you have never seen before and you need to get some sort of working communication going there is much point to what you say. As soon as you know what sequence of commands makes the piece of hardware work one has arrived in the territory where TDD is highly beneficial. You can then write a test where you require that your software indeed produces the correct sequence of commands and after that you can refactor your exploratory code while having to check that it still works together with the hardware much less often then otherwise would be needed. The remark about TDD and waterfall seems much less true to me. In my original post I wrote 'both frequent new features and keep working correctly'. That does not describe waterfall at all. In fact, and this is where you messages seems to get a bit self-contradictory, your post actually sounds much more waterfally to me than mine. It contains the words 'Towards the end'. It is waterfall projects that have an end that was envisioned right from the start. Whatever 'agile' means, it seems to have the feature that we never stop adding features and that stuff is supposed to be in a working state in between adding these features. So, if one is actually doing that there are dragons in the sentence 'Towards the end a lot of testing' because it actually turns into 'always lots of testing' which costs lots of time. In case of the hardware example it would also require that every developer has a species of that hardware on his desk next to his computer. As I happen to work in a field where we interface with hardware that is three stories high, we should realize that this is not always practical. One misunderstanding that might be going on is that perhaps you are assuming that in TDD one would test just one class at a time. I generally prefer to test a set of classes, i.e., a subsystem of a whole program. In that case the test are about properties of the program that, in many cases, the user could recognize as beneficial. There should be less churn at that level. I suppose some people would call this behaviour driven development instead but I am not so very married to terminology and will just keep calling it TDD. Testing individual classes can also be helpful but if they contain rather simple logic I am not sure writing tests for that is so very helpful. At some point would just be testing whether the processor really is capable of copying values which is not that interesting.
ellius 7y ago
I've found the opposite. When I start to write a test, I often realize I don't know exactly what I need to do because the requirements are unclear or somehow conflict with some other assumption of the system. That realization prompts me to go talk to my product owner, which forces them to clarify their requirements and target a more coherent idea of what the system should be.

gfodor 7y ago

I think as a first order approximation the level to which you can deploy TDD is inversely proportional to how difficult your problem is with regards to computational complexity. The most TDD’able code is a freshman level CS 100 programming assignment. The least TDD’able are things like real time simulations, numerical computing, machine learning, etc. In the latter case, you can TDD the boring parts (reading inputs, simple pure functions, etc) but TDD breaks down quickly if you have a huge state space, side effects, complex outputs, etc.

cjfd 7y ago
That depends. A problem may be quite difficult in computational complexity but if you can find simpler less computational complex version of it that can be used for automated tests that can be quick. E.g., test your neural network training procedure with a smaller network which will give a result that performs worse and after that use the thus obtained program on a with a bigger neural net.
- closeparen 7y ago
  It's not about runtime performance. It's about how well you can actually assess your code by matching small and well-defined inputs to small and well-defined outputs. LeetCode / CS101 problems are almost always like that. In the real world, it's also useful to separate out any tricky "policy" logic into nicely testable pure functions. But you are still going to have a bunch of side-effect munging left over, and unit testing is terrible at that. Tests that only assess their own mocks are a waste of time.
  - cjfd 7y ago
    Could you give an example of situations where there would not be small and well-defined inputs and outputs because this certainly does not sound like a common situation to me? And what do you mean by 'Tests that only asses their own mocks are a waste of time.' Certainly, many side effects can very effecitively be checked by mocks. Now, it could certainly get much more difficult if the side effects were very large and hard to define as you said in the paragraph before that, but I have some difficulty picturing that as a common situation.

dakom 7y ago

> If you write code that is close to the hardware and needs every ounce of performance there might be a point to this. It would make applying TDD both harder to pull off and less beneficial. It's not really about being close to the hardware[0] - there's many other things that are unrelated to low-level performance tweaks that make TDD or even unit tests a waste of time for experienced game developers. When someone like Blow or Carmack sees an automated playtest fail, they have a honed intuition for what caused it. This intuition was built up over many years of hard-earned experience. They simply don't need the lower level unit test that says "this matrix multiplication math is wrong" - they'll see a collision fail or a weird scaling thing and _know_ right off the bat the candidates in the codebase where things went wonky and fix it before you can even say "look at the logs" :) Since they get the same results without having to write unit tests everywhere, and especially without writing tests first, it's simply an unnecessary overhead to make them write/maintain it. I don't think anyone would argue TDD and/or unit tests are valueless, it's just a cost/benefit question. Pretty sure they'd use any technique where the benefit outweighs the cost- and for them, for many reasons in many varied scenarios, it just doesn't. [0] actually, if anything, I'd bet that the stuff that's really close to the hardware like the core engine stuff (math, memory management, etc.) is the one place they would consider having TDD and/or more unit tests!