6 ms·

> We are auditing our development processes to help prevent this from happening again. That's great to hear even if it took multiple stumbles for them to final

> We are auditing our development processes to help prevent this from happening again. That's great to hear even if it took multiple stumbles for them to finally admit - but surely they should be also audit their QA/testing processes? Or does development in AppleSpeak mean everything?

Klathmon 9y ago
It sound to me like the "development process" points to the whole soup-to-nuts system. From "brainstorming" and new feature development, to development, to testing, to QA, to deployment.
- blinkingled 9y ago
  If so, that's going to be a big undertaking :)
  - zokier 9y ago
    Or a big lip service.
vinceguidry 9y ago
I don't know how you could expect QA to be able to have a rigorous process to catch security problems of this type. It's one thing to audit the strength of crypto protocols, quite another to rigorously test every conceivable attack surface for privilege escalation. That space is vast.
- blinkingled 9y ago
  Actually entering blank passwords and automated password entry should be Test Cases #0 and #1 for any thing that has a login. OS and other critical infrastructure vendors should go beyond that and explore the vast space to make sure nothing like this ever happens.
  - glhaynes 9y ago
    They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-one times to trigger the bug? It's deeply unsatisfying to just say "the problem space is so vast that it's hard to even know what to try beyond the basics that every QA person knows" but I don't personally have a better answer than that. Maybe those that study the space do, though?
    - jacobush 9y ago
      Fuzzing inputs
    - anton_gogolev 9y ago
      Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.
      - ghostly_s 9y ago
        I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?
      - fooey 9y ago
        Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.
        qubex 9y ago
        And fascinatingly wasn’t broken until very recently...
      - qubex 9y ago
- hawski 9y ago
  I worked on mobile games few years back. Our tester always did everything that was sane and most things that were not sane. For every version. He tried to break the game. He was randomly and wildly tapping the screen. He was repeatedly going through menus. The most things he did us developers thought: who would do that? We caught lots of bugs this way. It was a company that you never heard of. It's a normal thing for me to require such things from QA. I also worked in big corporation that you heard about, developing embedded software. Same thing happened. Testing is doing things that most users do. But most of all testing is monkey bashing trying to break the thing. That's the mindset that QA must have. EDIT: typos, thanks
  - Sohcahtoa82 9y ago
    I've frequently seen people use "break" when they mean "brake", but I don't think I've ever seen someone get it the wrong the other way around.
  - brown9-2 9y ago
    Do you realize for all the unexpected bugs your manual tester caught, an equal number were simply never found?
    - hawski 9y ago
      No one said that you can't do automatic tests. Manual tests do not exclude automatic tests. But I dare to say that manual tests are essential, because that's how the software will really be used. Especially for release candidates. Apple has resources to do both repeatedly. In this big corporation I mentioned tests were also done automatically. A device receives via infrared programmed set of commands in a loop for 24h. The test simulates a user with a remote. Even so called gravitational tests are useful. A test where "only" gravitation acts on a device for 24h. Those may seem dumb, but bugs can be hunted this way, that no automatic test could find. Because real hardware and release environment is messy.
  - vinceguidry 9y ago
    That approach works for behavior and UX, but is woefully inadequate for security, sadly. The space for shenanigans is exponentially greater.
- elliotec 9y ago
  What do you mean? That is literally the job of QA. Even the worst QAs I've worked with test every input field into oblivion.
teamhappy 9y ago
> Or does development in AppleSpeak mean everything? It's a press release. It doesn't mean anything.
DougBTX 9y ago
Top google search result for "software development processes" says: What are the steps in software development? 1. Requirement gathering and analysis. 2. Design. 3. Implementation or coding. 4. Testing. 5. Deployment. 6. Maintenance. So, yes, it is step 4, QA & testing.

Klathmon 9y ago

It sound to me like the "development process" points to the whole soup-to-nuts system. From "brainstorming" and new feature development, to development, to testing, to QA, to deployment.

blinkingled 9y ago
If so, that's going to be a big undertaking :)
- zokier 9y ago
  Or a big lip service.

vinceguidry 9y ago

I don't know how you could expect QA to be able to have a rigorous process to catch security problems of this type. It's one thing to audit the strength of crypto protocols, quite another to rigorously test every conceivable attack surface for privilege escalation. That space is vast.

blinkingled 9y ago
Actually entering blank passwords and automated password entry should be Test Cases #0 and #1 for any thing that has a login. OS and other critical infrastructure vendors should go beyond that and explore the vast space to make sure nothing like this ever happens.
- glhaynes 9y ago
  They may have even had a test case that entered "root" and no password and hit OK once, but that wouldn't have caught it. What if you'd had to hit "OK" forty-one times to trigger the bug? It's deeply unsatisfying to just say "the problem space is so vast that it's hard to even know what to try beyond the basics that every QA person knows" but I don't personally have a better answer than that. Maybe those that study the space do, though?
  - jacobush 9y ago
    Fuzzing inputs
  - anton_gogolev 9y ago
    Where are all the unit/integration tests for the APIs that this damned button is calling? Hindsight being 20/20, but I cannot imagine not asserting that a newly created/re-enabled root user has a non-empty password.
    - ghostly_s 9y ago
      I don't think empty password on root is the issue here; enabling root when the user did not intend to and did not have permission to is. Isn't empty password for root a default state for systems where direct root login is not enabled? And isn't enabling root with no password the intended behavior when booting into Single User Mode?
    - fooey 9y ago
      Exactly, testing the UI may be hard, but whatever API that UI is calling is amazingly horribly broken.
      - qubex 9y ago
        And fascinatingly wasn’t broken until very recently...
    - qubex 9y ago
hawski 9y ago
I worked on mobile games few years back. Our tester always did everything that was sane and most things that were not sane. For every version. He tried to break the game. He was randomly and wildly tapping the screen. He was repeatedly going through menus. The most things he did us developers thought: who would do that? We caught lots of bugs this way. It was a company that you never heard of. It's a normal thing for me to require such things from QA. I also worked in big corporation that you heard about, developing embedded software. Same thing happened. Testing is doing things that most users do. But most of all testing is monkey bashing trying to break the thing. That's the mindset that QA must have. EDIT: typos, thanks
- Sohcahtoa82 9y ago
  I've frequently seen people use "break" when they mean "brake", but I don't think I've ever seen someone get it the wrong the other way around.
- brown9-2 9y ago
  Do you realize for all the unexpected bugs your manual tester caught, an equal number were simply never found?
  - hawski 9y ago
    No one said that you can't do automatic tests. Manual tests do not exclude automatic tests. But I dare to say that manual tests are essential, because that's how the software will really be used. Especially for release candidates. Apple has resources to do both repeatedly. In this big corporation I mentioned tests were also done automatically. A device receives via infrared programmed set of commands in a loop for 24h. The test simulates a user with a remote. Even so called gravitational tests are useful. A test where "only" gravitation acts on a device for 24h. Those may seem dumb, but bugs can be hunted this way, that no automatic test could find. Because real hardware and release environment is messy.
- vinceguidry 9y ago
  That approach works for behavior and UX, but is woefully inadequate for security, sadly. The space for shenanigans is exponentially greater.
elliotec 9y ago
What do you mean? That is literally the job of QA. Even the worst QAs I've worked with test every input field into oblivion.

teamhappy 9y ago

> Or does development in AppleSpeak mean everything? It's a press release. It doesn't mean anything.

DougBTX 9y ago

Top google search result for "software development processes" says: What are the steps in software development? 1. Requirement gathering and analysis. 2. Design. 3. Implementation or coding. 4. Testing. 5. Deployment. 6. Maintenance. So, yes, it is step 4, QA & testing.