3 ms·

How can you test against an unknown bug?

by egeozcan 9y ago

maltalex 9y ago
> How can you test against an unknown bug? The point of testing is to make the unknown bugs, known.
- naasking 9y ago
  The point of testing is actually to ensure certain classes of known bugs aren't present. You can't test for bugs of which you have no knowledge.
  - deleted 9y ago
    [deleted]
  - IncRnd 9y ago
    In this case, however, an OS should have some sort of test in place that process and user isolation is being enforced. That is certainly a known situation that can be tested. It is also a selling point of the OS.
    - ksk 9y ago
      Indeed, this bug is a yowzer.
stefan_ 9y ago
This one apparently meant that something intentionally mapped for kernel access only was now accessible from userland. That is exactly the tedious, boring issue someone manually testing a bunch of applications will never find, but unit tests are designed to.
IncRnd 9y ago
It wasn't an unknown bug. This was a fix for a known bug. The point is that there was no test in place for memory isolation between users or processes. FYI, security testers call this type of testing negative testing, which is different from functional testing that is done to test an app works "properly". However, for an OS this test is not negative testing but functional testing if the OS is designed to enforce user and process isolation.
asdsa5325 9y ago
Testing to make sure you can't read from somewhere you are not supposed to read from seems like a pretty obvious test for an OS.
- tedunangst 9y ago
  There are lots of places you're not supposed to read from. Does any operating system have 100% test coverage of addresses that aren't supposed to be readable?
  - PeterisP 9y ago
    IDK, there's a quite narrow whitelisted known range of addresses that should be readable by your process; you could (and should) certainly have an simple automated test that simply tries to read everything with the expectation that it should succeed only in known cases.

How can you test against an unknown bug?

maltalex 9y ago

> How can you test against an unknown bug? The point of testing is to make the unknown bugs, known.

naasking 9y ago
The point of testing is actually to ensure certain classes of known bugs aren't present. You can't test for bugs of which you have no knowledge.
- deleted 9y ago
  [deleted]
- IncRnd 9y ago
  In this case, however, an OS should have some sort of test in place that process and user isolation is being enforced. That is certainly a known situation that can be tested. It is also a selling point of the OS.
  - ksk 9y ago
    Indeed, this bug is a yowzer.

stefan_ 9y ago

This one apparently meant that something intentionally mapped for kernel access only was now accessible from userland. That is exactly the tedious, boring issue someone manually testing a bunch of applications will never find, but unit tests are designed to.

IncRnd 9y ago

It wasn't an unknown bug. This was a fix for a known bug. The point is that there was no test in place for memory isolation between users or processes. FYI, security testers call this type of testing negative testing, which is different from functional testing that is done to test an app works "properly". However, for an OS this test is not negative testing but functional testing if the OS is designed to enforce user and process isolation.

asdsa5325 9y ago

Testing to make sure you can't read from somewhere you are not supposed to read from seems like a pretty obvious test for an OS.

tedunangst 9y ago
There are lots of places you're not supposed to read from. Does any operating system have 100% test coverage of addresses that aren't supposed to be readable?
- PeterisP 9y ago
  IDK, there's a quite narrow whitelisted known range of addresses that should be readable by your process; you could (and should) certainly have an simple automated test that simply tries to read everything with the expectation that it should succeed only in known cases.