3 ms·

Unit testing is generally not useful for bare-metal code, since it depends more on the actual device behavior than the behavior documented in the datasheet. Cre

Unit testing is generally not useful for bare-metal code, since it depends more on the actual device behavior than the behavior documented in the datasheet. Creating an accurate mock is, sadly, often just not possible with the documentation available. And with the quality of mocks being bad, you end up testing the datasheets against the devices instead of testing the kernel code against the devices.

nerdponx 5y ago
Serious question: how many CVEs are due to bugs in code that falls into this hard-to-test category?
rurban 5y ago
That's exactly the mindset why baremetal code is so horribly buggy. Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification. Without mock you can just test on actual HW a few selected test scenarios you think of. Emulation: The lack of qemu support for arm chips is mindboggling. avr/atmel is quite good in contrast. renode is far better.
- blueflow 5y ago
  > Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification. You did not understand what i wrote. With hardware, its just damn difficult to get accurate mocks. If the mock is wrong, all tests relying on it are not meaningful. If you ask me, QEMU doesn't even properly support emulating x86. Its just "good enough" to run common operating systems. You could try improving the support, maybe then you will realize how just damn difficult and undocumented this shit is, and you would understand why nobody wants to hear your crappy unit test evangelism.
  - rurban 5y ago
    Well, my company is happy to test baremetal fw on sw, as other better companies do also. Ditching tests is a lousy argument, only coming from people who have no idea how to test difficult targets. Of course you have to patch qemu, gdb, renode and valgrind if something is missing. Eg the recent apple M1. But this is only needed for emulation, and we are talking simulation here.
    - deleted 5y ago
      [deleted]
staticassertion 5y ago
> since it depends more on the actual device behavior than the behavior documented in the datasheet. That doesn't sound true at all tbh. I would imagine a huge amount of code is just generalized data structures and algorithms. I doubt many CVEs (<1%) rely on an accurate model of the underlying hardware.
- Too 5y ago
  Spectre and meltdown can be seen as caused by the underlying hardware. But those are in that rare 1% bucket and I doubt a more accurate model used by unit testing would have stopped these anyway.

nerdponx 5y ago

Serious question: how many CVEs are due to bugs in code that falls into this hard-to-test category?

rurban 5y ago

That's exactly the mindset why baremetal code is so horribly buggy. Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification. Without mock you can just test on actual HW a few selected test scenarios you think of. Emulation: The lack of qemu support for arm chips is mindboggling. avr/atmel is quite good in contrast. renode is far better.

blueflow 5y ago
> Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification. You did not understand what i wrote. With hardware, its just damn difficult to get accurate mocks. If the mock is wrong, all tests relying on it are not meaningful. If you ask me, QEMU doesn't even properly support emulating x86. Its just "good enough" to run common operating systems. You could try improving the support, maybe then you will realize how just damn difficult and undocumented this shit is, and you would understand why nobody wants to hear your crappy unit test evangelism.
- rurban 5y ago
  Well, my company is happy to test baremetal fw on sw, as other better companies do also. Ditching tests is a lousy argument, only coming from people who have no idea how to test difficult targets. Of course you have to patch qemu, gdb, renode and valgrind if something is missing. Eg the recent apple M1. But this is only needed for emulation, and we are talking simulation here.
  - deleted 5y ago
    [deleted]

staticassertion 5y ago

> since it depends more on the actual device behavior than the behavior documented in the datasheet. That doesn't sound true at all tbh. I would imagine a huge amount of code is just generalized data structures and algorithms. I doubt many CVEs (<1%) rely on an accurate model of the underlying hardware.

Too 5y ago
Spectre and meltdown can be seen as caused by the underlying hardware. But those are in that rare 1% bucket and I doubt a more accurate model used by unit testing would have stopped these anyway.