9 ms·

I wonder what is going on with software quality and testing at Apple. It feels like recently there have been quite a few issues like this (the FileVault passwor

I wonder what is going on with software quality and testing at Apple. It feels like recently there have been quite a few issues like this (the FileVault password bug, numerous issues with iOS 11, the issue that totally broke iOS Safari a couple of years ago) which should have been fairly easily caught, especially given the limited range of devices their software runs on. I know testing is hard, but a company with Apple’s resources shouldn’t be making slip ups like this. It suggests some real issues such as lack of unit/automated tests and/or sufficient release testing, which pretty urgently need addressing. Anyone got any inside scoop?

bangonkeyboard 9y ago
macOS and iOS updates at Apple are now inextricably tied to new iPhone releases. There is a strict yearly deadline that the teams sprint toward, a timeline imposed by marketing rather than readiness. This affects prioritization of which features are pursued, where they lie in the stack, and how polished they get. Insufficient testing at today's Apple is not limited to software. They bragged about their extensive input testing lab [0] when the new line of Magic accessories was released, but the Magic Keyboard with Numeric Keypad launched last summer had all of its inventory pulled from the channel last month because users discovered that the model was so thin that its midsection bowed over time. [0]: https://medium.com/backchannel/what-i-saw-inside-apple-s-top-secret-input-lab-6637e2e5492e https://medium.com/backchannel/what-i-saw-inside-apple-s-top...
- warent 9y ago
  Everything ends in tears when managers mix up targets/estimates with commitments
- nikofeyn 9y ago
  it is also that they pursue features just for the sake of it. things get moved arund in the iPad from release to release for no good reason, often going backwards in usability. every release i have to relearn simple things like how to manage the screen brightness. i really wonder what they are thinking internally other than “we need to shake things up to make it appear we’re doing something with stale products”.
  - maxxxxx 9y ago
    It seems phones and tablets have reached the stage where laptops were maybe 15 years ago. All the major features are done and innovation is pretty much over. So they have to make a lot of cosmetic changes that look like activity.
- kifleswing 9y ago
  Where did you hear that the keyboard was pulled from the channel? It just seems to be out of stock for me.
- komali2 9y ago
  Haven't deadlines at Apple always been driven by marketing? I'm looking for a source but I remember a story where the product director for iPod was told by steve jobs "make it simple, fast, beautiful, and have it done by Christmas."
  - oneeyedpigeon 9y ago
    That's sure to send shivers down the spine of anyone reading it here but, to be fair to jobs, he managed to get exactly what he wanted on that occasion.
    - amildie 9y ago
      Asking for stuff requires no talent, vision, discipline or effort whatsoever. Pretty much anyone can do it. If you don't actually deliver, you don't actually matter.
wonderous 9y ago
Apple has always had QA issues, the difference now is that they’re increasingly tested by the users, hackers, etc.
- ben_w 9y ago
  Difference? MacOS userbase hasn’t changed much since 2011, I thought?
  - eric_h 9y ago
    I've no information on how good this site's data is, but https://www.statista.com/statistics/218089/global-market-share-of-windows-7/ https://www.statista.com/statistics/218089/global-market-sha... seems to show that from 2013-2017 global macOS market share has increased from 7.95% to 11.3%.
cm2187 9y ago
iTunes had QA problems for more than 10 years, only the early versions were really solid. I am not sure that it is a recent problem.
- tomc1985 9y ago
  Subjectively it feels like Apple bugs have become larger and more prevalent, over the last few years. That and IMO clean OSX/iOS installs don't quite feel as polished as they used to. (I stopped using Apple products, except for a MBP, for a few years and recently started using them again, and the MBP still runs 10.10 for precisely this reason) The last solid OSX release was Snow Leopard
  - Cthulhu_ 9y ago
    They've added features over the years without removing or polishing them; there's Launchpad which was added in a period where OSX seemed to lean towards becoming touch-friendly, but it didn't replace any existing feature (iirc) and just feels off. Might just be me though. Notification center? Don't use it.
mholt 9y ago
> Anyone got any inside scoop? I have a feeling that anyone who does would get fired for commenting here about it.
- ianmcgowan 9y ago
  You have to think that whoever was responsible for testing this is going to get fired. This is egregiously bad...
  - fmihaila 9y ago
    This is a management issue. It shouldn't be possible for such a mistake to slip into production code. It has happened more than once in recent times.
princetontiger 9y ago
I'm going back to Windows after a 10 year hiatus. Mac has definitely declined.
rogaha 9y ago
I've experienced lots of issues wiht IOS and High Sierra as well. Apple definitely reduced their software quality since IOS 11.
2trill2spill 9y ago
It seems apple's software has been trending down in quality since Snow Leopard.
- thanatropism 9y ago
  I'll agree that Snow Leopard is the high water-mark.
  - sanityUnbounded 9y ago
    I have been in the apple ecosystem for about 10 years. For a company that has been priding itself on end user security, the bugs that have been creeping their way into the OS are just... disappointing. What is the point of paying a premium for a well polished hardware/software bundle if the OS is malfunctioning in a non trivial manner. Design? Right now when I use my calculator app on my iPhone and do 2+2+2 I get 24. That's a pretty awful design. Actually, it's a lie.
  - mikeokner 9y ago
    3rded. I wish time stopped at 10.6.8.
  - e5an 9y ago
    It can't be a coincidence that this was the last major release before Steve Jobs died.
    - mercutio2 9y ago
      It can’t be a coincidence because it’s not true. Lion was released well before SJ’s death. Unless you’re arguing Lion wasn’t major because you didn’t like it, but that’s an argument that proves to much, methinks.
  - IBM 9y ago
    It's common among a small group of Mac users to hold Snow Leopard up as the peak of software quality, but only because of rose-colored glasses [1]. [1] https://www.computerworld.com/article/2528936/mac-os-x/snow-leopard-bug-deletes-all-user-data.html https://www.computerworld.com/article/2528936/mac-os-x/snow-...
umanwizard 9y ago
Anecdotal but I started noticing a decline in quality after Steve Jobs died.
- bsaul 9y ago
  i'm still wondering if that impression ( which i share) is real or not. At the minimum, i'd say i feel apple release less innovating os versions while producing at least the same number of bugs.
mikestew 9y ago
Take this for the anecdata that it is. I interviewed at Apple, referred by old Microsoft friends that worked there. As I was trying to get a feel for things before the interview, I asked about the software testing. I was told, "don't expect what you're used to at Microsoft". The reference there is from when Microsoft often had more testers on a team than devs (ah, the good ol' days). The summary of what I was told by friends, and the questions I asked during the interview, is that testers at Apple aren't the testers that Microsoft used to have. Microsoft had testers working in MS Research, researching ways to better test software. Apple, from the impressions I got, is doing good to have testers than can write "hello, world". This was from the app side of things, not OS; I don't know if it's any different on the OS side. But since I don't work there, I have no good inside info. But just from gut feel, I don't think my anecdata is too far off the mark. Based just on the bugs made public, I just don't get the impression that there are testers at Apple whose sole reason for being there is to tear into a piece of software and break it. There was a bug a few weeks ago posted to HN that I commented on. I don't have a link without digging through my comments, but it was something along the lines of "how could a tester not find this in five minutes of exploratory testing?" This bug is similar. It would take more than five minutes, but were this my area to test I'd pick at it once in a while when I had a few minutes. As I pick at it, I wouldn't expect to find anything, but I've got a minute between builds, so instead of randomly clicking Facebook I'll randomly click this dialog. What did the dev forget? What weird state was not accounted for? Some kind of state overflow if I click the button enough times? Shove some Unicode in there, that didn't find anything; meh, maybe I ought to move o...hey, wait a minute. Did that thing just log me in as root? But my gut says that Apple doesn't employ a lot of testers like that.
- yodsanklai 9y ago
  > But since I don't work there, I have no good inside info Actually, I've been wondering why I hear less about people working at Apple than at other big tech companies. It seems everyone and their mother work at Google or Facebook, but no so much at Apple. Do they have less software engineers, or their employees are required to be more discrete?
  - OJFord 9y ago
draw_down 9y ago
No clues from me, but the problem is undeniable. Their stuff just does not work well anymore, and it has been so for a while now.
spamizbad 9y ago
The loss of people like Avie Tevanian and Bertrand Serlet took its toll. Are there any "(tech) household name" engineers doing system-level work on iOS/macOS these days? It seems like Google and Facebook have a slew of them.
- LeoNatan25 9y ago
  In the case of Facebook, all the “household named” developers do nothing to improve the quality of their output, be it their user-facing software or developer-facing open source.
- danieldk 9y ago
  Dominic Giampaolo of BeOS/BeFS fame. He now works on APFS at Apple. Their work is really - impressive APFS was announced in June 2016 and rolled out on iOS devices in March 2017. Given that the APFS roll-out was relatively uneventful and how they tested it [1], it seems that they can still do low-level engineering and proper testing. Of course, until recently they had Chris Lattner as well. [1] For some iOS releases, they converted HFS to APFS in-place, report the results back to Apple, but did not write the APFS 'superblock' to keep the filesystem HFS+. It's quite a smart idea, because they got reports from millions of devices without actually switching them to APFS.
  - 72deluxe 9y ago
    Whilst APFS is an improvement and I fondly remember using BeOS back in the day, I am not sure APFS is that impressive feature-wise compared to NTFS. It's still miles behind NTFS, which is now ancient.
adamio 9y ago
Repressive collusion to fix salaries and restrict industry movement, doesn't really inspire your employees to try their best
- deleted 9y ago
  [deleted]
zaidf 9y ago
Unrelated to Mac OS but I used to wonder all the time why iTunes connect was so shoddy. I got my answer when I learned Apple had outsourced a ton of backend work including iTunes Connect, App Store backend to Infosys in India. They’re now retreating from that strategy: https://factordaily.com/apple-to-pull-back-development-work-in-house-from-infosys-cognizant/ https://factordaily.com/apple-to-pull-back-development-work-...
- wingworks 9y ago
  well that explains allot, iTunes store/ App Store have always been painfully slow, compared to doing anything in Safari.
- api 9y ago
  Everyone retreats from that strategy.
socalnate1 9y ago
A [?] don't know what you are talking about.
k3a 9y ago
And seeing this I am wondering why people still trust closed-source software. My long term dream is using 100% free software on a HW with minimum binary blobs.
- pasta 9y ago
  This also can happen in open software. So I don't think your comment is valid. Open software enables people to take a look inside to what is going on. It isn't a cure for bug free development.
  - fstephany 9y ago
    It reminds me the KMail bug: https://www.ctrl.blog/entry/kmail-cve-2017-9604-openpgp https://www.ctrl.blog/entry/kmail-cve-2017-9604-openpgp Some security bugs exist in the Linux/BSDs kernels for a loooong time before someone notice and fix it (e.g., https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20presentations/DEFCON-25-Ilja-van-Sprundel-BSD-Kern-Vulns.pdf https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20pre...)

bangonkeyboard 9y ago

macOS and iOS updates at Apple are now inextricably tied to new iPhone releases. There is a strict yearly deadline that the teams sprint toward, a timeline imposed by marketing rather than readiness. This affects prioritization of which features are pursued, where they lie in the stack, and how polished they get. Insufficient testing at today's Apple is not limited to software. They bragged about their extensive input testing lab [0] when the new line of Magic accessories was released, but the Magic Keyboard with Numeric Keypad launched last summer had all of its inventory pulled from the channel last month because users discovered that the model was so thin that its midsection bowed over time. [0]: https://medium.com/backchannel/what-i-saw-inside-apple-s-top-secret-input-lab-6637e2e5492e https://medium.com/backchannel/what-i-saw-inside-apple-s-top...

warent 9y ago
Everything ends in tears when managers mix up targets/estimates with commitments
nikofeyn 9y ago
it is also that they pursue features just for the sake of it. things get moved arund in the iPad from release to release for no good reason, often going backwards in usability. every release i have to relearn simple things like how to manage the screen brightness. i really wonder what they are thinking internally other than “we need to shake things up to make it appear we’re doing something with stale products”.
- maxxxxx 9y ago
  It seems phones and tablets have reached the stage where laptops were maybe 15 years ago. All the major features are done and innovation is pretty much over. So they have to make a lot of cosmetic changes that look like activity.
kifleswing 9y ago
Where did you hear that the keyboard was pulled from the channel? It just seems to be out of stock for me.
komali2 9y ago
Haven't deadlines at Apple always been driven by marketing? I'm looking for a source but I remember a story where the product director for iPod was told by steve jobs "make it simple, fast, beautiful, and have it done by Christmas."
- oneeyedpigeon 9y ago
  That's sure to send shivers down the spine of anyone reading it here but, to be fair to jobs, he managed to get exactly what he wanted on that occasion.
  - amildie 9y ago
    Asking for stuff requires no talent, vision, discipline or effort whatsoever. Pretty much anyone can do it. If you don't actually deliver, you don't actually matter.

wonderous 9y ago

Apple has always had QA issues, the difference now is that they’re increasingly tested by the users, hackers, etc.

ben_w 9y ago
Difference? MacOS userbase hasn’t changed much since 2011, I thought?
- eric_h 9y ago
  I've no information on how good this site's data is, but https://www.statista.com/statistics/218089/global-market-share-of-windows-7/ https://www.statista.com/statistics/218089/global-market-sha... seems to show that from 2013-2017 global macOS market share has increased from 7.95% to 11.3%.

cm2187 9y ago

iTunes had QA problems for more than 10 years, only the early versions were really solid. I am not sure that it is a recent problem.

tomc1985 9y ago
Subjectively it feels like Apple bugs have become larger and more prevalent, over the last few years. That and IMO clean OSX/iOS installs don't quite feel as polished as they used to. (I stopped using Apple products, except for a MBP, for a few years and recently started using them again, and the MBP still runs 10.10 for precisely this reason) The last solid OSX release was Snow Leopard
- Cthulhu_ 9y ago
  They've added features over the years without removing or polishing them; there's Launchpad which was added in a period where OSX seemed to lean towards becoming touch-friendly, but it didn't replace any existing feature (iirc) and just feels off. Might just be me though. Notification center? Don't use it.

mholt 9y ago

> Anyone got any inside scoop? I have a feeling that anyone who does would get fired for commenting here about it.

ianmcgowan 9y ago
You have to think that whoever was responsible for testing this is going to get fired. This is egregiously bad...
- fmihaila 9y ago
  This is a management issue. It shouldn't be possible for such a mistake to slip into production code. It has happened more than once in recent times.

princetontiger 9y ago

I'm going back to Windows after a 10 year hiatus. Mac has definitely declined.

rogaha 9y ago

I've experienced lots of issues wiht IOS and High Sierra as well. Apple definitely reduced their software quality since IOS 11.

2trill2spill 9y ago

It seems apple's software has been trending down in quality since Snow Leopard.

thanatropism 9y ago
I'll agree that Snow Leopard is the high water-mark.
- sanityUnbounded 9y ago
  I have been in the apple ecosystem for about 10 years. For a company that has been priding itself on end user security, the bugs that have been creeping their way into the OS are just... disappointing. What is the point of paying a premium for a well polished hardware/software bundle if the OS is malfunctioning in a non trivial manner. Design? Right now when I use my calculator app on my iPhone and do 2+2+2 I get 24. That's a pretty awful design. Actually, it's a lie.
- mikeokner 9y ago
  3rded. I wish time stopped at 10.6.8.
- e5an 9y ago
  It can't be a coincidence that this was the last major release before Steve Jobs died.
  - mercutio2 9y ago
    It can’t be a coincidence because it’s not true. Lion was released well before SJ’s death. Unless you’re arguing Lion wasn’t major because you didn’t like it, but that’s an argument that proves to much, methinks.
- IBM 9y ago
  It's common among a small group of Mac users to hold Snow Leopard up as the peak of software quality, but only because of rose-colored glasses [1]. [1] https://www.computerworld.com/article/2528936/mac-os-x/snow-leopard-bug-deletes-all-user-data.html https://www.computerworld.com/article/2528936/mac-os-x/snow-...

umanwizard 9y ago

Anecdotal but I started noticing a decline in quality after Steve Jobs died.

bsaul 9y ago
i'm still wondering if that impression ( which i share) is real or not. At the minimum, i'd say i feel apple release less innovating os versions while producing at least the same number of bugs.

mikestew 9y ago

Take this for the anecdata that it is. I interviewed at Apple, referred by old Microsoft friends that worked there. As I was trying to get a feel for things before the interview, I asked about the software testing. I was told, "don't expect what you're used to at Microsoft". The reference there is from when Microsoft often had more testers on a team than devs (ah, the good ol' days). The summary of what I was told by friends, and the questions I asked during the interview, is that testers at Apple aren't the testers that Microsoft used to have. Microsoft had testers working in MS Research, researching ways to better test software. Apple, from the impressions I got, is doing good to have testers than can write "hello, world". This was from the app side of things, not OS; I don't know if it's any different on the OS side. But since I don't work there, I have no good inside info. But just from gut feel, I don't think my anecdata is too far off the mark. Based just on the bugs made public, I just don't get the impression that there are testers at Apple whose sole reason for being there is to tear into a piece of software and break it. There was a bug a few weeks ago posted to HN that I commented on. I don't have a link without digging through my comments, but it was something along the lines of "how could a tester not find this in five minutes of exploratory testing?" This bug is similar. It would take more than five minutes, but were this my area to test I'd pick at it once in a while when I had a few minutes. As I pick at it, I wouldn't expect to find anything, but I've got a minute between builds, so instead of randomly clicking Facebook I'll randomly click this dialog. What did the dev forget? What weird state was not accounted for? Some kind of state overflow if I click the button enough times? Shove some Unicode in there, that didn't find anything; meh, maybe I ought to move o...hey, wait a minute. Did that thing just log me in as root? But my gut says that Apple doesn't employ a lot of testers like that.

yodsanklai 9y ago
> But since I don't work there, I have no good inside info Actually, I've been wondering why I hear less about people working at Apple than at other big tech companies. It seems everyone and their mother work at Google or Facebook, but no so much at Apple. Do they have less software engineers, or their employees are required to be more discrete?
- OJFord 9y ago

draw_down 9y ago

No clues from me, but the problem is undeniable. Their stuff just does not work well anymore, and it has been so for a while now.

spamizbad 9y ago

The loss of people like Avie Tevanian and Bertrand Serlet took its toll. Are there any "(tech) household name" engineers doing system-level work on iOS/macOS these days? It seems like Google and Facebook have a slew of them.

LeoNatan25 9y ago
In the case of Facebook, all the “household named” developers do nothing to improve the quality of their output, be it their user-facing software or developer-facing open source.
danieldk 9y ago
Dominic Giampaolo of BeOS/BeFS fame. He now works on APFS at Apple. Their work is really - impressive APFS was announced in June 2016 and rolled out on iOS devices in March 2017. Given that the APFS roll-out was relatively uneventful and how they tested it [1], it seems that they can still do low-level engineering and proper testing. Of course, until recently they had Chris Lattner as well. [1] For some iOS releases, they converted HFS to APFS in-place, report the results back to Apple, but did not write the APFS 'superblock' to keep the filesystem HFS+. It's quite a smart idea, because they got reports from millions of devices without actually switching them to APFS.
- 72deluxe 9y ago
  Whilst APFS is an improvement and I fondly remember using BeOS back in the day, I am not sure APFS is that impressive feature-wise compared to NTFS. It's still miles behind NTFS, which is now ancient.

adamio 9y ago

Repressive collusion to fix salaries and restrict industry movement, doesn't really inspire your employees to try their best

deleted 9y ago
[deleted]

zaidf 9y ago

Unrelated to Mac OS but I used to wonder all the time why iTunes connect was so shoddy. I got my answer when I learned Apple had outsourced a ton of backend work including iTunes Connect, App Store backend to Infosys in India. They’re now retreating from that strategy: https://factordaily.com/apple-to-pull-back-development-work-in-house-from-infosys-cognizant/ https://factordaily.com/apple-to-pull-back-development-work-...

wingworks 9y ago
well that explains allot, iTunes store/ App Store have always been painfully slow, compared to doing anything in Safari.
api 9y ago
Everyone retreats from that strategy.

socalnate1 9y ago

A [?] don't know what you are talking about.

k3a 9y ago

And seeing this I am wondering why people still trust closed-source software. My long term dream is using 100% free software on a HW with minimum binary blobs.

pasta 9y ago
This also can happen in open software. So I don't think your comment is valid. Open software enables people to take a look inside to what is going on. It isn't a cure for bug free development.
- fstephany 9y ago
  It reminds me the KMail bug: https://www.ctrl.blog/entry/kmail-cve-2017-9604-openpgp https://www.ctrl.blog/entry/kmail-cve-2017-9604-openpgp Some security bugs exist in the Linux/BSDs kernels for a loooong time before someone notice and fix it (e.g., https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20presentations/DEFCON-25-Ilja-van-Sprundel-BSD-Kern-Vulns.pdf https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20pre...)