6 ms·

Notice how “unit testing coverage” is not on this map.

• ylyn 5y ago
  Unit tests are not the only way, or the best way, to test software.
  • Dylan16807 5y ago
    Sure? The note is that it is not a way here.
    • egberts 5y ago
      Because a kernel is not a user-land?
      • zibzab 5y ago
        I agree with parent that unit testing privileged bare metal code is not always possible, at least not without spending 10x-100x dev time on writing tests. Example: anything that directly accesses MMU or caches.
        • rurban 5y ago
          On the contrary, simulating MMU or caches are trivial. You can create a memory subsystem mock in a few hours. IO, assembly and interrupts are much harder.
          • zibzab 5y ago
            How would you create a mock object for something that modifies the memory layout of the unit test while it is running?
            • Dylan16807 5y ago
              > How would you create a mock object for something that modifies the memory layout of the unit test while it is running? There's two ways I can interpret "the unit test". 1) "How would you create a mock object for modifying the memory layout of the code being tested?" That's just a routine that shuffles around some bytes. Or marks parts of the address space to fault so it can simulate the contents. 2) "How would you create a mock object for modifying the memory layout of the code performing the test?" That wouldn't be a test. The code performing the test isn't inside its own simulation.
            • rurban 5y ago
              You create your mmap's on init. The MMU is just a simple 3way array lookup, as a Unicode table.
              • yencabulator 5y ago
                Unicode?
  • Too 5y ago
    Perhaps there was an /s missing in parent post? Implying that unit testing is often prioritized more by management, over checking security, configuration and tools like this one.
• deleted 5y ago
  [deleted]
• phendrenad2 5y ago
  I can just imagine the technical and political difficulties in adding proper unit tests to the kernel now. The C model is "everything just imports and calls whatever it wants". It's great for speed, but for good unit testing you actually want some of that "AbstractFactoryFactory" stuff we make fun of in Java. For instance, take any module of the Linux kernel and try to see what it would take to test it without importing any other kernel code. Then, if you have a plan for that, now you need to stub out the rest of the kernel. Since every module imports dozens of files from other modules, and calls dozens of functions from said modules, it's not very pretty.
• blueflow 5y ago
  Unit testing is generally not useful for bare-metal code, since it depends more on the actual device behavior than the behavior documented in the datasheet. Creating an accurate mock is, sadly, often just not possible with the documentation available. And with the quality of mocks being bad, you end up testing the datasheets against the devices instead of testing the kernel code against the devices.
  • nerdponx 5y ago
    Serious question: how many CVEs are due to bugs in code that falls into this hard-to-test category?
  • rurban 5y ago
    That's exactly the mindset why baremetal code is so horribly buggy. Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification. Without mock you can just test on actual HW a few selected test scenarios you think of. Emulation: The lack of qemu support for arm chips is mindboggling. avr/atmel is quite good in contrast. renode is far better.
    • blueflow 5y ago
      > Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification. You did not understand what i wrote. With hardware, its just damn difficult to get accurate mocks. If the mock is wrong, all tests relying on it are not meaningful. If you ask me, QEMU doesn't even properly support emulating x86. Its just "good enough" to run common operating systems. You could try improving the support, maybe then you will realize how just damn difficult and undocumented this shit is, and you would understand why nobody wants to hear your crappy unit test evangelism.
      • rurban 5y ago
        Well, my company is happy to test baremetal fw on sw, as other better companies do also. Ditching tests is a lousy argument, only coming from people who have no idea how to test difficult targets. Of course you have to patch qemu, gdb, renode and valgrind if something is missing. Eg the recent apple M1. But this is only needed for emulation, and we are talking simulation here.
• GuB-42 5y ago
  Unit testing coverage is a development practice, not really a defense. Testing, coverage, static analysis, methodologies, etc... are a way to make prevent things from going wrong in the first place, defenses are here to protect you if things go wrong. We need both. The article focuses only on the "defense" part, the development practices could make another article. Note that testing and defenses can go together: test to make sure all paths are taken before shipping, and see if defenses trigger. Fuzzing in particular takes that approach.