4 ms·

They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-on

They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-one times to trigger the bug? It's deeply unsatisfying to just say "the problem space is so vast that it's hard to even know what to try beyond the basics that every QA person knows" but I don't personally have a better answer than that. Maybe those that study the space do, though?

jacobush 9y ago
Fuzzing inputs
anton_gogolev 9y ago
Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.
- ghostly_s 9y ago
  I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?
- fooey 9y ago
  Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.
  - qubex 9y ago
    And fascinatingly wasn’t broken until very recently...
- qubex 9y ago
  Following and furthering your logic, what the hell could they have been doing in the codebase to revert a control mechanism that was effective up to and including 10.12.6, but unsafe as of 10.13.0 onwards???
  - CiPHPerCoder 9y ago
    A graceful upgrade mechanism towards a new password hashing algorithm.
    - glhaynes 9y ago
      Change is so, so risky.
      - CiPHPerCoder 9y ago
        There's a lot that can go wrong, which is why writing testable code and then /actually/ testing it, matters so much.
- geofft 9y ago
  The code was not supposed to enable the root user in the first place, so I'd be surprised if someone added an assertion that anything about the root user's account was set. It was only supposed to check the root user's password. I suppose you could write an assertion that the code didn't enable the root user, but I'm pretty sure that no password-validation routine anywhere in the history of the world has ever had a test case to make sure it didn't modify the account while validating it. These are unknown unknowns. If you knew enough to write the right test, you wouldn't have written the bug in the first place.
netsharc 9y ago
But if the 2nd attempt does something different, there must be some sort of logic happening, and whoever built this magic (maybe it's an if-block like: (if loginAttempts++ > 0) ... ) should've tested it. We cam see that the function f(a, b) { return a+b; } should return the same value for the same a and b, so testing that once is enough. But if there's something in there that reads a global variable, maybe check who changes that global variable, and what happens with different values of that global variable?
- eridius 9y ago
  The "logic" in this case is the first attempt failed because the root user was disabled, but as a side-effect it enabled the root user. The second attempt worked because the root user was now enabled. There's no explicit logic in here that says the second attempt should have behaved differently.
- ben_bai 9y ago
  Oh it's never that easy. It happens 20 layers deep, with configuration set a certain way... only every Friday the 13th. I.e. MacOS is based on UNIX/BSD, root is all powerfull. So root is disabled form login with <star> password, but it's still there. Then there is the Apple autentication framework on top of that, here root is disabled. But if you try to log in with root, it gets enabled but can't login because password is still <star> an disabled. Then somebody decides new password hashes are safer and we should update old hashes whenever possible. So now when a disabled root account gets enabled, the update routine mishandles <star> password and updates the root password hash with <empty string>.

jacobush 9y ago

Fuzzing inputs

anton_gogolev 9y ago

Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.

ghostly_s 9y ago
I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?
fooey 9y ago
Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.
- qubex 9y ago
  And fascinatingly wasn’t broken until very recently...
qubex 9y ago
Following and furthering your logic, what the hell could they have been doing in the codebase to revert a control mechanism that was effective up to and including 10.12.6, but unsafe as of 10.13.0 onwards???
- CiPHPerCoder 9y ago
  A graceful upgrade mechanism towards a new password hashing algorithm.
  - glhaynes 9y ago
    Change is so, so risky.
    - CiPHPerCoder 9y ago
      There's a lot that can go wrong, which is why writing testable code and then /actually/ testing it, matters so much.
geofft 9y ago
The code was not supposed to enable the root user in the first place, so I'd be surprised if someone added an assertion that anything about the root user's account was set. It was only supposed to check the root user's password. I suppose you could write an assertion that the code didn't enable the root user, but I'm pretty sure that no password-validation routine anywhere in the history of the world has ever had a test case to make sure it didn't modify the account while validating it. These are unknown unknowns. If you knew enough to write the right test, you wouldn't have written the bug in the first place.

netsharc 9y ago

But if the 2nd attempt does something different, there must be some sort of logic happening, and whoever built this magic (maybe it's an if-block like: (if loginAttempts++ > 0) ... ) should've tested it. We cam see that the function f(a, b) { return a+b; } should return the same value for the same a and b, so testing that once is enough. But if there's something in there that reads a global variable, maybe check who changes that global variable, and what happens with different values of that global variable?

eridius 9y ago
The "logic" in this case is the first attempt failed because the root user was disabled, but as a side-effect it enabled the root user. The second attempt worked because the root user was now enabled. There's no explicit logic in here that says the second attempt should have behaved differently.
ben_bai 9y ago
Oh it's never that easy. It happens 20 layers deep, with configuration set a certain way... only every Friday the 13th. I.e. MacOS is based on UNIX/BSD, root is all powerfull. So root is disabled form login with <star> password, but it's still there. Then there is the Apple autentication framework on top of that, here root is disabled. But if you try to log in with root, it gets enabled but can't login because password is still <star> an disabled. Then somebody decides new password hashes are safer and we should update old hashes whenever possible. So now when a disabled root account gets enabled, the update routine mishandles <star> password and updates the root password hash with <empty string>.