4 ms·

> Partly because we had no unit tests (a whole 'nother discussion) I think that the is the primary point. If you have good test coverage at each level, it's ac

by mbell 11y ago

> Partly because we had no unit tests (a whole 'nother discussion) I think that the is the primary point. If you have good test coverage at each level, it's actually really easy to make large sweeping changes to a codebase. If you don't have those tests, it's hell. This is a big part of why I think unit tests are ok, but not nearly as important as good E2E and perf tests, as painful as they can be to write and wait for to run some times. Good high level tests let you tear the heart out of the codebase, replace it and still be confident it works.

shoo 11y ago
I agree that having good test coverage across different levels/purposes makes sweeping changes far quicker, easier, and even feasible to consider doing. Building and shipping the software is perhaps half the work - another half of the work is growing and maintaining the scaffolding to rapidly, automatically and reliably measure the properties of the software that you wish it to have. It seems no-one here is arguing that YAGNI should be applied to justify not building automated tests.
krupan 11y ago
The one thing we did have was good end-to-end tests. When they failed, it was the unit tests I was wishing for.
- lmm 11y ago
  Interesting. I've found myself working almost entirely without unit tests (at least when I have an advanced type system) - making small incremental changes means that if an end-to-end test starts failing there's only a very small number of places that could have caused that failure. What kind of problems did you face? Why were they hard to diagnose?
  - krupan 11y ago
    I definitely did not have an advanced type system helping me out. This was a SystemVerilog testbench. SystemVerilog has a type system akin to Java's or C's. Actually it's probably worse than either of those. It's a real Frankenstein's monster of a language. The general idea with SystemVerilog is you have a low-level model of the hardware made up of language primitives like wires and logic functions. Then you wrap that in a few layers of testbench code. Each layer ups the abstraction level until at your top level you can write tests in terms of high level transactions (in this case it was a flash controller, so you could do simple commands like write, read, erase). The layers all take care of converting those high-level transactions into (eventually at the bottom layer) wiggling the input wires of the hardware at just the right times. It goes the other way too, monitoring the output wires of the hardware and converting those wiggles into high-level response transactions (such as, "here's the read data you requested"). The net result is you should be able to write succinct tests at a high-level of abstraction that exercise a lot of the hardware (wiggles a lot of the wires a lot of different ways). There were actually two testbenches that shared some of this code and among them at least three different code paths from high-level to low-level and back again that gave me trouble. If I had unit tests isolating each of those layers it would have been a lot easier to find which layer was dropping a byte of data here or adding extra padding there. EDIT: I just noticed the "small incremental changes" part of your question. That's the other thing about not having unit tests. I had to get the whole thing to compile and run before I could test any change, and that meant changing every layer to handle the variable sized data. Part way through I did stop to write unit tests for the trickiest (lowest-level) layer.

by mbell 11y ago

shoo 11y ago

I agree that having good test coverage across different levels/purposes makes sweeping changes far quicker, easier, and even feasible to consider doing. Building and shipping the software is perhaps half the work - another half of the work is growing and maintaining the scaffolding to rapidly, automatically and reliably measure the properties of the software that you wish it to have. It seems no-one here is arguing that YAGNI should be applied to justify not building automated tests.

krupan 11y ago

The one thing we did have was good end-to-end tests. When they failed, it was the unit tests I was wishing for.

lmm 11y ago
Interesting. I've found myself working almost entirely without unit tests (at least when I have an advanced type system) - making small incremental changes means that if an end-to-end test starts failing there's only a very small number of places that could have caused that failure. What kind of problems did you face? Why were they hard to diagnose?
- krupan 11y ago
  I definitely did not have an advanced type system helping me out. This was a SystemVerilog testbench. SystemVerilog has a type system akin to Java's or C's. Actually it's probably worse than either of those. It's a real Frankenstein's monster of a language. The general idea with SystemVerilog is you have a low-level model of the hardware made up of language primitives like wires and logic functions. Then you wrap that in a few layers of testbench code. Each layer ups the abstraction level until at your top level you can write tests in terms of high level transactions (in this case it was a flash controller, so you could do simple commands like write, read, erase). The layers all take care of converting those high-level transactions into (eventually at the bottom layer) wiggling the input wires of the hardware at just the right times. It goes the other way too, monitoring the output wires of the hardware and converting those wiggles into high-level response transactions (such as, "here's the read data you requested"). The net result is you should be able to write succinct tests at a high-level of abstraction that exercise a lot of the hardware (wiggles a lot of the wires a lot of different ways). There were actually two testbenches that shared some of this code and among them at least three different code paths from high-level to low-level and back again that gave me trouble. If I had unit tests isolating each of those layers it would have been a lot easier to find which layer was dropping a byte of data here or adding extra padding there. EDIT: I just noticed the "small incremental changes" part of your question. That's the other thing about not having unit tests. I had to get the whole thing to compile and run before I could test any change, and that meant changing every layer to handle the variable sized data. Part way through I did stop to write unit tests for the trickiest (lowest-level) layer.