6 ms·

Based on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? As an outsider it looks to me like

Based on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? As an outsider it looks to me like they're just pushing too hard to keep up with a yearly release schedule but that's based on no inside knowledge at all.

coolwhhip 9y ago
Shipping gets you power and influence. Outside of personal conviction and judgement, there’s not incentive to do much else.
- fastball 9y ago
  Right, but engineers probably are going to have some level of conviction and judgement when it comes to something they helped build. They're personally invested in a feature they've worked on. If I was an engineer at Apple who worked on iMessage, and I heard acquaintances talk about the bugginess about iMessage, I would probably start out being defensive but then I would regret that we didn't ship a better product. I think an EPM is less likely to feel like that, as they are not so closely tied to the actual product, which is what the Reddit poster was saying.
  - vlovich123 9y ago
    It's easy to blame "the other". Having worked at Apple I never felt like it was the EPM causing quality issues for us. It was us setting an aggressive schedule ourselves and, more importantly, having poor testing. The EPM was mainly just making sure tasks didn't get missed, reprioritizing them as the schedule moved based on discussions with managers & other engineers etc. Sure at some point in the schedule the central EPM committee for a release would start locking things down & punting less critical issues, but that's something that's done for every single release even for engineer-driven companies because you have to ship at some point. Keep in mind also that these EPMs themselves tend to be fairly competent engineers in their own right so these aren't MBAs making decisions, they've just moved on as their career evolved. They care about quality just as much as anyone. The difficult part is making the call of cutting a feature especially if it's a keynote feature because there's just not enough time to actually ship it because that's predicting the future. Another difficult call is what bug constitutes a delay of an OS release, especially when the bug report may not have the necessary details to indicate it might be more critical than it seems. TLDR: It's not as easy as saying it's all the EPMs fault. There's plenty of blame to go around because nobody is perfect & predicting the future (which is what scheduling is) is very hard.
masklinn 9y ago
> Based on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? It's not clear because there have been absolute shit-show releases going back to the early days of OSX. Even the fondest-remembered releases were only so after a ton of polish, and that was with releases slipping significantly (you might see 3 years pass between major updates, and the new version would still be completely unstable). I'm sure yearly releases don't help, especially if there isn't a postgres-type culture of "we ship what's ready", but software engineering issues at Apple go back a long time.
- gurkendoktor 9y ago
  What has definitely changed is the pace with which people are expected to upgrade. When OS X Snow Leopard came out, you had to visit a store to buy the DVD. Nobody would look down on you for running a "legacy OS" if you only upgraded when .3 was out. The ways in which this has changed affect both tech-savvy and casual users: 1. Older versions of iOS (which didn't exist back then) don't receive security fixes, so the only way to stay safe is to upgrade within days of the initial release (ideally before Apple publishes its security KB article). 2. Older versions of macOS often lag behind when it comes to security fixes. There is no clear policy on whether it is safe to stay on release n-1. Apple needs to be nudged to even document macOS security issues[1]. If you are moderately paranoid about security, you have to install all updates on day one (as on iOS). 3. iOS and macOS automatically download OS upgrades and nag you to install them. 4. There is a narrative on community sites that Apple's products are only good because Apple relentlessly kills legacy code and features (I don't even agree with the latter part). Dropping support for anything older than a year is considered a badge of honor because it means there will be "progress". The more paying customers you inconvenience, the edgier you are. 5. Apple feeds into this "old software is bad" meme by publicly LOLing at Android's adoption curve every year. 6. Emoji. I wish I was kidding, but only seeing squares when other people go out of their way to express their emotions is actually annoying. 7. If you are already on an OS that is buggy, you are more likely take the plunge and update to a brand-new OS in the hope that this time, things will be better. [1] https://twitter.com/patrickwardle/status/953416156802703360 https://twitter.com/patrickwardle/status/953416156802703360
vlovich123 9y ago
It's really hard to put any single root cause on it. Firstly, mobile OSes are massive beasts with lots of different teams working on it. Co-ordinating this becomes very difficult. Additionally, you typically get about 1 build a day which means there's a long lead time to find a bug that's managed to propagate out and then a long lead time for it to get fixed (at least 1 day). I think this decrease in quality has been seen across all OS vendors, it's just that Apple has become one of the dominant OSes so there's more press about it than there was about previous Apple releases. So now you have a larger volume of bugs since bug count is based on the amount of code written, assuming best case that developer quality is roughly consistent across all teams. Since you have a larger volume and only so many hours in a day, trying to ship all software at once at an annual cadence means your bug count will go up regardless of how much QA you throw at the problem (since in addition to finding issues you have to fix them). Automated testing might help because you could catch bugs before they're even submitted. Well, automated testing is still very hard on these devices when you're building the OS. For example, how do you do something as simple as build & run unit tests? If the answer involves loading code onto a mobile device that means (at Apple scale) you need a farm of devices. Moreover, your test isn't valuable if it's running against yesterday's OS. Might not even build/run because you've now acquired a dependency on an API made available in the new OS. So now you have to co-ordinate building the software using an SDK that may be unstable (build breakages happen), run it on an OS you somehow have to pre-validate works otherwise & then have some confidence that the bug found was introduced by your commit & not some combination of build issue, compiler bug (oh yeah - the compiler changes drastically too while your building your next OS), or downstream dependency bug. Finally, on top of all this you need to make sure you can recover bricked devices in an automated fashion so that the maintenance overhead of your automated testing cluster is manageable. All of these aren't impossible problems to solve. The problem is that it requires a lot of effort & physical infrastructure. Additionally, Apple has had a very ad-hoc development environment where no 2 teams are the same (it was slowly changing to unify large pieces of infrastructure when I left) which complicates the ability to roll out central infrastructure. Additionally, there are different needs - the kernel has different testing requirements/strategies from system daemons, from drivers from 1p apps from backward compatibility testing. When I left a couple of years ago Craig had actually directed his teams to improve the automated testing infrastructure but at the end of the day the software is growing more quickly than the investment that was made in QA. There's also only so quickly you can roll out central automation infrastructure to an organization as large as Apple so it's not as simple as "do more QA" or "add more QA resources" not to mention that "QA resources" in automated testing typically requires very skilled software engineers to design & build the systems. Allowing a longer bake time is exactly the right move here as it recognizes the need to decrease the volume of bugs while infrastructure catches up.

coolwhhip 9y ago

Shipping gets you power and influence. Outside of personal conviction and judgement, there’s not incentive to do much else.

fastball 9y ago
Right, but engineers probably are going to have some level of conviction and judgement when it comes to something they helped build. They're personally invested in a feature they've worked on. If I was an engineer at Apple who worked on iMessage, and I heard acquaintances talk about the bugginess about iMessage, I would probably start out being defensive but then I would regret that we didn't ship a better product. I think an EPM is less likely to feel like that, as they are not so closely tied to the actual product, which is what the Reddit poster was saying.
- vlovich123 9y ago
  It's easy to blame "the other". Having worked at Apple I never felt like it was the EPM causing quality issues for us. It was us setting an aggressive schedule ourselves and, more importantly, having poor testing. The EPM was mainly just making sure tasks didn't get missed, reprioritizing them as the schedule moved based on discussions with managers & other engineers etc. Sure at some point in the schedule the central EPM committee for a release would start locking things down & punting less critical issues, but that's something that's done for every single release even for engineer-driven companies because you have to ship at some point. Keep in mind also that these EPMs themselves tend to be fairly competent engineers in their own right so these aren't MBAs making decisions, they've just moved on as their career evolved. They care about quality just as much as anyone. The difficult part is making the call of cutting a feature especially if it's a keynote feature because there's just not enough time to actually ship it because that's predicting the future. Another difficult call is what bug constitutes a delay of an OS release, especially when the bug report may not have the necessary details to indicate it might be more critical than it seems. TLDR: It's not as easy as saying it's all the EPMs fault. There's plenty of blame to go around because nobody is perfect & predicting the future (which is what scheduling is) is very hard.

masklinn 9y ago

> Based on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? It's not clear because there have been absolute shit-show releases going back to the early days of OSX. Even the fondest-remembered releases were only so after a ton of polish, and that was with releases slipping significantly (you might see 3 years pass between major updates, and the new version would still be completely unstable). I'm sure yearly releases don't help, especially if there isn't a postgres-type culture of "we ship what's ready", but software engineering issues at Apple go back a long time.

gurkendoktor 9y ago
What has definitely changed is the pace with which people are expected to upgrade. When OS X Snow Leopard came out, you had to visit a store to buy the DVD. Nobody would look down on you for running a "legacy OS" if you only upgraded when .3 was out. The ways in which this has changed affect both tech-savvy and casual users: 1. Older versions of iOS (which didn't exist back then) don't receive security fixes, so the only way to stay safe is to upgrade within days of the initial release (ideally before Apple publishes its security KB article). 2. Older versions of macOS often lag behind when it comes to security fixes. There is no clear policy on whether it is safe to stay on release n-1. Apple needs to be nudged to even document macOS security issues[1]. If you are moderately paranoid about security, you have to install all updates on day one (as on iOS). 3. iOS and macOS automatically download OS upgrades and nag you to install them. 4. There is a narrative on community sites that Apple's products are only good because Apple relentlessly kills legacy code and features (I don't even agree with the latter part). Dropping support for anything older than a year is considered a badge of honor because it means there will be "progress". The more paying customers you inconvenience, the edgier you are. 5. Apple feeds into this "old software is bad" meme by publicly LOLing at Android's adoption curve every year. 6. Emoji. I wish I was kidding, but only seeing squares when other people go out of their way to express their emotions is actually annoying. 7. If you are already on an OS that is buggy, you are more likely take the plunge and update to a brand-new OS in the hope that this time, things will be better. [1] https://twitter.com/patrickwardle/status/953416156802703360 https://twitter.com/patrickwardle/status/953416156802703360

vlovich123 9y ago

It's really hard to put any single root cause on it. Firstly, mobile OSes are massive beasts with lots of different teams working on it. Co-ordinating this becomes very difficult. Additionally, you typically get about 1 build a day which means there's a long lead time to find a bug that's managed to propagate out and then a long lead time for it to get fixed (at least 1 day). I think this decrease in quality has been seen across all OS vendors, it's just that Apple has become one of the dominant OSes so there's more press about it than there was about previous Apple releases. So now you have a larger volume of bugs since bug count is based on the amount of code written, assuming best case that developer quality is roughly consistent across all teams. Since you have a larger volume and only so many hours in a day, trying to ship all software at once at an annual cadence means your bug count will go up regardless of how much QA you throw at the problem (since in addition to finding issues you have to fix them). Automated testing might help because you could catch bugs before they're even submitted. Well, automated testing is still very hard on these devices when you're building the OS. For example, how do you do something as simple as build & run unit tests? If the answer involves loading code onto a mobile device that means (at Apple scale) you need a farm of devices. Moreover, your test isn't valuable if it's running against yesterday's OS. Might not even build/run because you've now acquired a dependency on an API made available in the new OS. So now you have to co-ordinate building the software using an SDK that may be unstable (build breakages happen), run it on an OS you somehow have to pre-validate works otherwise & then have some confidence that the bug found was introduced by your commit & not some combination of build issue, compiler bug (oh yeah - the compiler changes drastically too while your building your next OS), or downstream dependency bug. Finally, on top of all this you need to make sure you can recover bricked devices in an automated fashion so that the maintenance overhead of your automated testing cluster is manageable. All of these aren't impossible problems to solve. The problem is that it requires a lot of effort & physical infrastructure. Additionally, Apple has had a very ad-hoc development environment where no 2 teams are the same (it was slowly changing to unify large pieces of infrastructure when I left) which complicates the ability to roll out central infrastructure. Additionally, there are different needs - the kernel has different testing requirements/strategies from system daemons, from drivers from 1p apps from backward compatibility testing. When I left a couple of years ago Craig had actually directed his teams to improve the automated testing infrastructure but at the end of the day the software is growing more quickly than the investment that was made in QA. There's also only so quickly you can roll out central automation infrastructure to an organization as large as Apple so it's not as simple as "do more QA" or "add more QA resources" not to mention that "QA resources" in automated testing typically requires very skilled software engineers to design & build the systems. Allowing a longer bake time is exactly the right move here as it recognizes the need to decrease the volume of bugs while infrastructure catches up.