5 ms·

I don't know how you could expect QA to be able to have a rigorous process to catch security problems of this type. It's one thing to audit the strength of cryp

I don't know how you could expect QA to be able to have a rigorous process to catch security problems of this type. It's one thing to audit the strength of crypto protocols, quite another to rigorously test every conceivable attack surface for privilege escalation. That space is vast.

blinkingled 9y ago
Actually entering blank passwords and automated password entry should be Test Cases #0 and #1 for any thing that has a login. OS and other critical infrastructure vendors should go beyond that and explore the vast space to make sure nothing like this ever happens.
- glhaynes 9y ago
  They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-one times to trigger the bug? It's deeply unsatisfying to just say "the problem space is so vast that it's hard to even know what to try beyond the basics that every QA person knows" but I don't personally have a better answer than that. Maybe those that study the space do, though?
  - jacobush 9y ago
    Fuzzing inputs
  - anton_gogolev 9y ago
    Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.
    - ghostly_s 9y ago
      I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?
    - fooey 9y ago
      Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.
      - qubex 9y ago
        And fascinatingly wasn’t broken until very recently...
    - qubex 9y ago
      Following and furthering your logic, what the hell could they have been doing in the codebase to revert a control mechanism that was effective up to and including 10.12.6, but unsafe as of 10.13.0 onwards???
      - CiPHPerCoder 9y ago
        A graceful upgrade mechanism towards a new password hashing algorithm.
        glhaynes 9y ago
        Change is so, so risky.
        CiPHPerCoder 9y ago
        There's a lot that can go wrong, which is why writing testable code and then /actually/ testing it, matters so much.
    - geofft 9y ago
      The code was not supposed to enable the root user in the first place, so I'd be surprised if someone added an assertion that anything about the root user's account was set. It was only supposed to check the root user's password. I suppose you could write an assertion that the code didn't enable the root user, but I'm pretty sure that no password-validation routine anywhere in the history of the world has ever had a test case to make sure it didn't modify the account while validating it. These are unknown unknowns. If you knew enough to write the right test, you wouldn't have written the bug in the first place.
  - netsharc 9y ago
    But if the 2nd attempt does something different, there must be some sort of logic happening, and whoever built this magic (maybe it's an if-block like: (if loginAttempts++ > 0) ... ) should've tested it. We cam see that the function f(a, b) { return a+b; } should return the same value for the same a and b, so testing that once is enough. But if there's something in there that reads a global variable, maybe check who changes that global variable, and what happens with different values of that global variable?
    - eridius 9y ago
      The "logic" in this case is the first attempt failed because the root user was disabled, but as a side-effect it enabled the root user. The second attempt worked because the root user was now enabled. There's no explicit logic in here that says the second attempt should have behaved differently.
    - ben_bai 9y ago
      Oh it's never that easy. It happens 20 layers deep, with configuration set a certain way... only every Friday the 13th. I.e. MacOS is based on UNIX/BSD, root is all powerfull. So root is disabled form login with <star> password, but it's still there. Then there is the Apple autentication framework on top of that, here root is disabled. But if you try to log in with root, it gets enabled but can't login because password is still <star> an disabled. Then somebody decides new password hashes are safer and we should update old hashes whenever possible. So now when a disabled root account gets enabled, the update routine mishandles <star> password and updates the root password hash with <empty string>.
- vinceguidry 9y ago
  And I'm sure they've had privilege escalation vulns before and a workflow already in place for catching a lot of them. I mean, it's not like Apple's totally asleep at the wheel here. OSX may not be a front-burner project anymore, but they're still supporting it better than Microsoft manages to support Windows. But that's the nasty thing about InfoSec. It's a never-ending process. There's always realms you haven't considered yet. I'd bet that they had a process in place that they thought could catch these, but it managed to fail somehow. You're always in a hindsight-is-20-20 situation.
  - blinkingled 9y ago
    > but they're still supporting it better than Microsoft manages to support Windows. I don't know what metric if any you're using to make that statement but any person working Enterprise/Corporate IT will tell you what MS manages to pull year over year with Windows, Office and bunch of other stuff is fairly monumental given the demands of their operating space. Only way Apple can survive there with their current Engineering setup/culture is if they only did simple stuff and told the customers no interop and you only get to do what we think is best for you!
    - vinceguidry 9y ago
      I bet Microsoft really does do a great job supporting corporate users. Apple does a miles-better job supporting consumers.
  - qubex 9y ago
    (I am not a developer, nor a sysadmin, but a casual programmer and intensive power user, and I get the instant bleah factor when I come across consumers or enterprises using it.) I’m a longtime NeXT-then-MacOSX user with an experience in commercial UNICES ante-2000 and a predilection for Linux and DragonFlyBSD for my server needs. That said, I am gaining a begrudging respect for Microsoft. They make you pay for your nose, and their stuff looks and feels clunky, but the solidity and backwards-compatibility is utterly amazing.
  - Spooky23 9y ago
hawski 9y ago
I worked on mobile games few years back. Our tester always did everything that was sane and most things that were not sane. For every version. He tried to break the game. He was randomly and wildly tapping the screen. He was repeatedly going through menus. The most things he did us developers thought: who would do that? We caught lots of bugs this way. It was a company that you never heard of. It's a normal thing for me to require such things from QA. I also worked in big corporation that you heard about, developing embedded software. Same thing happened. Testing is doing things that most users do. But most of all testing is monkey bashing trying to break the thing. That's the mindset that QA must have. EDIT: typos, thanks
- Sohcahtoa82 9y ago
  I've frequently seen people use "break" when they mean "brake", but I don't think I've ever seen someone get it the wrong the other way around.
- brown9-2 9y ago
  Do you realize for all the unexpected bugs your manual tester caught, an equal number were simply never found?
  - hawski 9y ago
    No one said that you can't do automatic tests. Manual tests do not exclude automatic tests. But I dare to say that manual tests are essential, because that's how the software will really be used. Especially for release candidates. Apple has resources to do both repeatedly. In this big corporation I mentioned tests were also done automatically. A device receives via infrared programmed set of commands in a loop for 24h. The test simulates a user with a remote. Even so called gravitational tests are useful. A test where "only" gravitation acts on a device for 24h. Those may seem dumb, but bugs can be hunted this way, that no automatic test could find. Because real hardware and release environment is messy.
- vinceguidry 9y ago
  That approach works for behavior and UX, but is woefully inadequate for security, sadly. The space for shenanigans is exponentially greater.
elliotec 9y ago
What do you mean? That is literally the job of QA. Even the worst QAs I've worked with test every input field into oblivion.

blinkingled 9y ago

Actually entering blank passwords and automated password entry should be Test Cases #0 and #1 for any thing that has a login. OS and other critical infrastructure vendors should go beyond that and explore the vast space to make sure nothing like this ever happens.

glhaynes 9y ago
They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-one times to trigger the bug? It's deeply unsatisfying to just say "the problem space is so vast that it's hard to even know what to try beyond the basics that every QA person knows" but I don't personally have a better answer than that. Maybe those that study the space do, though?
- jacobush 9y ago
  Fuzzing inputs
- anton_gogolev 9y ago
  Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.
  - ghostly_s 9y ago
    I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?
  - fooey 9y ago
    Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.
    - qubex 9y ago
      And fascinatingly wasn’t broken until very recently...
  - qubex 9y ago
    Following and furthering your logic, what the hell could they have been doing in the codebase to revert a control mechanism that was effective up to and including 10.12.6, but unsafe as of 10.13.0 onwards???
    - CiPHPerCoder 9y ago
      A graceful upgrade mechanism towards a new password hashing algorithm.
      - glhaynes 9y ago
        Change is so, so risky.
        CiPHPerCoder 9y ago
        There's a lot that can go wrong, which is why writing testable code and then /actually/ testing it, matters so much.
  - geofft 9y ago
    The code was not supposed to enable the root user in the first place, so I'd be surprised if someone added an assertion that anything about the root user's account was set. It was only supposed to check the root user's password. I suppose you could write an assertion that the code didn't enable the root user, but I'm pretty sure that no password-validation routine anywhere in the history of the world has ever had a test case to make sure it didn't modify the account while validating it. These are unknown unknowns. If you knew enough to write the right test, you wouldn't have written the bug in the first place.
- netsharc 9y ago
  But if the 2nd attempt does something different, there must be some sort of logic happening, and whoever built this magic (maybe it's an if-block like: (if loginAttempts++ > 0) ... ) should've tested it. We cam see that the function f(a, b) { return a+b; } should return the same value for the same a and b, so testing that once is enough. But if there's something in there that reads a global variable, maybe check who changes that global variable, and what happens with different values of that global variable?
  - eridius 9y ago
    The "logic" in this case is the first attempt failed because the root user was disabled, but as a side-effect it enabled the root user. The second attempt worked because the root user was now enabled. There's no explicit logic in here that says the second attempt should have behaved differently.
  - ben_bai 9y ago
    Oh it's never that easy. It happens 20 layers deep, with configuration set a certain way... only every Friday the 13th. I.e. MacOS is based on UNIX/BSD, root is all powerfull. So root is disabled form login with <star> password, but it's still there. Then there is the Apple autentication framework on top of that, here root is disabled. But if you try to log in with root, it gets enabled but can't login because password is still <star> an disabled. Then somebody decides new password hashes are safer and we should update old hashes whenever possible. So now when a disabled root account gets enabled, the update routine mishandles <star> password and updates the root password hash with <empty string>.
vinceguidry 9y ago
And I'm sure they've had privilege escalation vulns before and a workflow already in place for catching a lot of them. I mean, it's not like Apple's totally asleep at the wheel here. OSX may not be a front-burner project anymore, but they're still supporting it better than Microsoft manages to support Windows. But that's the nasty thing about InfoSec. It's a never-ending process. There's always realms you haven't considered yet. I'd bet that they had a process in place that they thought could catch these, but it managed to fail somehow. You're always in a hindsight-is-20-20 situation.
- blinkingled 9y ago
  > but they're still supporting it better than Microsoft manages to support Windows. I don't know what metric if any you're using to make that statement but any person working Enterprise/Corporate IT will tell you what MS manages to pull year over year with Windows, Office and bunch of other stuff is fairly monumental given the demands of their operating space. Only way Apple can survive there with their current Engineering setup/culture is if they only did simple stuff and told the customers no interop and you only get to do what we think is best for you!
  - vinceguidry 9y ago
    I bet Microsoft really does do a great job supporting corporate users. Apple does a miles-better job supporting consumers.
- qubex 9y ago
  (I am not a developer, nor a sysadmin, but a casual programmer and intensive power user, and I get the instant bleah factor when I come across consumers or enterprises using it.) I’m a longtime NeXT-then-MacOSX user with an experience in commercial UNICES ante-2000 and a predilection for Linux and DragonFlyBSD for my server needs. That said, I am gaining a begrudging respect for Microsoft. They make you pay for your nose, and their stuff looks and feels clunky, but the solidity and backwards-compatibility is utterly amazing.
- Spooky23 9y ago

hawski 9y ago

I worked on mobile games few years back. Our tester always did everything that was sane and most things that were not sane. For every version. He tried to break the game. He was randomly and wildly tapping the screen. He was repeatedly going through menus. The most things he did us developers thought: who would do that? We caught lots of bugs this way. It was a company that you never heard of. It's a normal thing for me to require such things from QA. I also worked in big corporation that you heard about, developing embedded software. Same thing happened. Testing is doing things that most users do. But most of all testing is monkey bashing trying to break the thing. That's the mindset that QA must have. EDIT: typos, thanks

Sohcahtoa82 9y ago
I've frequently seen people use "break" when they mean "brake", but I don't think I've ever seen someone get it the wrong the other way around.
brown9-2 9y ago
Do you realize for all the unexpected bugs your manual tester caught, an equal number were simply never found?
- hawski 9y ago
  No one said that you can't do automatic tests. Manual tests do not exclude automatic tests. But I dare to say that manual tests are essential, because that's how the software will really be used. Especially for release candidates. Apple has resources to do both repeatedly. In this big corporation I mentioned tests were also done automatically. A device receives via infrared programmed set of commands in a loop for 24h. The test simulates a user with a remote. Even so called gravitational tests are useful. A test where "only" gravitation acts on a device for 24h. Those may seem dumb, but bugs can be hunted this way, that no automatic test could find. Because real hardware and release environment is messy.
vinceguidry 9y ago
That approach works for behavior and UX, but is woefully inadequate for security, sadly. The space for shenanigans is exponentially greater.

elliotec 9y ago

What do you mean? That is literally the job of QA. Even the worst QAs I've worked with test every input field into oblivion.