3 ms·

>Writing code to verify code is so much harder than just writing the code It's not supposed to be easy. Doing it well is really hard and way underappreciated.

>Writing code to verify code is so much harder than just writing the code It's not supposed to be easy. Doing it well is really hard and way underappreciated. >Testing side-effecting code is hard. This is well established. It's also convoluted, complex, generally brittle. It's not necessarily brittle when done right but it's hard, it's convoluted and it's complex. >Before the test, create the input AND go to the filesystem, prepare the input and the spot where output is expected. After the test, check the output AND go to the filesystem, read the files from there and check their contents. Everything is intertwined: the prep, the implementation of the code under test, and the checks at the end. It's specific to my filesystem. And it's slow. No way can I run more than a few of these each build. This doesn't sound like it would take more than a few hundred milliseconds. That's not slow. Furthermore, you could definitely run it on multiple filesystems by firing up a virtual machine for each filesystem and running the test. That would be slow, but you don't need to do it often.

mikeash 11y ago
I generally want all tests in total to complete in a few seconds at most. Longer than that and they become a serious impediment to the edit build test cycle, and you either start screwing around with only running some tests routinely, or only run the tests occasionally, neither of which is good. If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context.
- Alupis 11y ago
  There's nothing wrong with having a full test suite that runs as part of the nightly build. Sometimes it's just not practical to impose a hard limit of "a few seconds at most" on your test suite, especially for larger projects. (some projects have test suites that take hours to run, and simply cannot be any quicker unless they were less thorough).
  - mikeash 11y ago
    I agree, but you want to avoid it when you can, and being able to put tests into the fast ones you run all the time is good when you can. So I'd say that a test which takes a few hundred milliseconds easily qualifies as "slow" and is something you want to avoid if you can, even if it's not the end of the world if you can't.
- crdoconnor 11y ago
  >I generally want all tests in total to complete in a few seconds at most. Longer than that and they become a serious impediment to the edit build test cycle, and you either start screwing around with only running some tests routinely, or only run the tests occasionally, neither of which is good. I don't think that's a problem at all. I usually run one functional test at a time while developing (ideally runtime < 30 seconds) which covers the code that I'm changing. 90% of the bugs that I introduce accidentally will get picked up by that test while I'm refactoring. In the background (often on a different machine) I will have the full set of tests running constantly. Those might take hours to complete (sometimes a whole night). They will pick up the 5-10% of bugs that slipped through the net eventually, though - which is ok. This is way better than having a unit test suite that doesn't even come close to mimicing real life usage, and is a PITA to maintain because it's naturally tightly coupled to your architecture (this bites you back when you refactor). >If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context. One 500 millisecond test for this particular use case will pick up 90% of bugs. No, that is not going to interrupt your development cycle. Yes, a full suite of tests that fires up a VM for every different possible kind of filesystem will take hours. That's perfectly ok. You can go get a coffee while it runs.
- vosper 11y ago
  I don't want there to be an edit/build/test cycle - I want my tests to be validated continuously. I want my IDE to point out instantly that I have made a change that breaks 3 of my 10 unit tests, and I want a panel that shows exactly what the output used to be for each test, and what it is now.
Dewie3 11y ago
> It's not supposed to be easy. > [...] run it on multiple filesystems by firing up a virtual machine for each filesystem Yeah... I can see what you mean by not supposed to be easy.

mikeash 11y ago

I generally want all tests in total to complete in a few seconds at most. Longer than that and they become a serious impediment to the edit build test cycle, and you either start screwing around with only running some tests routinely, or only run the tests occasionally, neither of which is good. If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context.

Alupis 11y ago
There's nothing wrong with having a full test suite that runs as part of the nightly build. Sometimes it's just not practical to impose a hard limit of "a few seconds at most" on your test suite, especially for larger projects. (some projects have test suites that take hours to run, and simply cannot be any quicker unless they were less thorough).
- mikeash 11y ago
  I agree, but you want to avoid it when you can, and being able to put tests into the fast ones you run all the time is good when you can. So I'd say that a test which takes a few hundred milliseconds easily qualifies as "slow" and is something you want to avoid if you can, even if it's not the end of the world if you can't.
crdoconnor 11y ago
>I generally want all tests in total to complete in a few seconds at most. Longer than that and they become a serious impediment to the edit build test cycle, and you either start screwing around with only running some tests routinely, or only run the tests occasionally, neither of which is good. I don't think that's a problem at all. I usually run one functional test at a time while developing (ideally runtime < 30 seconds) which covers the code that I'm changing. 90% of the bugs that I introduce accidentally will get picked up by that test while I'm refactoring. In the background (often on a different machine) I will have the full set of tests running constantly. Those might take hours to complete (sometimes a whole night). They will pick up the 5-10% of bugs that slipped through the net eventually, though - which is ok. This is way better than having a unit test suite that doesn't even come close to mimicing real life usage, and is a PITA to maintain because it's naturally tightly coupled to your architecture (this bites you back when you refactor). >If a single test takes a few hundred milliseconds then that means you can only run a dozen or so such tests before you hit that problem, which isn't very many. I'd say that a few hundred milliseconds is quite slow in this context. One 500 millisecond test for this particular use case will pick up 90% of bugs. No, that is not going to interrupt your development cycle. Yes, a full suite of tests that fires up a VM for every different possible kind of filesystem will take hours. That's perfectly ok. You can go get a coffee while it runs.
vosper 11y ago
I don't want there to be an edit/build/test cycle - I want my tests to be validated continuously. I want my IDE to point out instantly that I have made a change that breaks 3 of my 10 unit tests, and I want a panel that shows exactly what the output used to be for each test, and what it is now.

Dewie3 11y ago

> It's not supposed to be easy. > [...] run it on multiple filesystems by firing up a virtual machine for each filesystem Yeah... I can see what you mean by not supposed to be easy.