6 ms·

I have a slightly different perspective (I worked on iOS 7 - 11). One of the things about Apple that's different to many of the other tech conglomerates is that

I have a slightly different perspective (I worked on iOS 7 - 11). One of the things about Apple that's different to many of the other tech conglomerates is that engineering teams are given huge freedom to work the way they want (for better and for worse). Because engineering is so fragmented you should always take what one person says about the process with a pinch of salt. So that means there are hundreds of source repos, patch review systems, and the like. When I was there I saw teams using Phabricator, GitLab, GitHub, plain e-mailed diffs, and more. In the same way EPMs and project process varies massively across teams, so what that employee experienced is by no means the norm. Certainly I didn't find this to be true at all: > Nothing could be worked on if it wasn’t in Radar with a priority number attached and signed off by the teams’ EPM. No room for a side project or time away from your daily duties because there were always P1s to fix. In fact, I worked on a major feature that was keynoted one year that came out purely because of my side project work. If anything it was a frustration to enter the "real world" where PMs had so much more power and influence than they did when I was at Apple (where PMs didn't really exist, and EPMs existed only to manage project timelines and drive features that HI / ID came up with).

chuckdries 9y ago
Do you think some level of standardization of process would help apple as a whole? Would it help eliminate the relatively poor experience of reddit-OP?
- objclxt 9y ago
  I think it's both a blessing and a curse. On the one hand it makes it very easy for teams to find an approach that works for them, and does away with a lot of bureaucracy. On the other, it promotes a lot of inefficiencies (I knew of at least 8 separate instances of Phabricator being used by different teams), and makes it very difficult to centralize core tooling and operations. It also makes it very hard for engineers to have internal mobility (you typically would have to do a full interview loop as an internal transfer, which is in stark contrast Google / FB's approach). When I left there was talk of centralizing more tooling to attempt to alleviate some of these issues, but I don't know if any progress has been made since then.
  - deleted 9y ago
    [deleted]
majormajor 9y ago
When I've seen "always follow priority order," it's always been somewhere between "[wink wink] that's a good idea, figure out how to link it to the top-priority stuff" and "all priorities are negotiable" anyway. Was your experience similar, or just more of a team truly without external priority ordering?
cageface 9y ago
Based on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? As an outsider it looks to me like they're just pushing too hard to keep up with a yearly release schedule but that's based on no inside knowledge at all.
- coolwhhip 9y ago
  Shipping gets you power and influence. Outside of personal conviction and judgement, there’s not incentive to do much else.
  - fastball 9y ago
    Right, but engineers probably are going to have some level of conviction and judgement when it comes to something they helped build. They're personally invested in a feature they've worked on. If I was an engineer at Apple who worked on iMessage, and I heard acquaintances talk about the bugginess about iMessage, I would probably start out being defensive but then I would regret that we didn't ship a better product. I think an EPM is less likely to feel like that, as they are not so closely tied to the actual product, which is what the Reddit poster was saying.
    - vlovich123 9y ago
      It's easy to blame "the other". Having worked at Apple I never felt like it was the EPM causing quality issues for us. It was us setting an aggressive schedule ourselves and, more importantly, having poor testing. The EPM was mainly just making sure tasks didn't get missed, reprioritizing them as the schedule moved based on discussions with managers & other engineers etc. Sure at some point in the schedule the central EPM committee for a release would start locking things down & punting less critical issues, but that's something that's done for every single release even for engineer-driven companies because you have to ship at some point. Keep in mind also that these EPMs themselves tend to be fairly competent engineers in their own right so these aren't MBAs making decisions, they've just moved on as their career evolved. They care about quality just as much as anyone. The difficult part is making the call of cutting a feature especially if it's a keynote feature because there's just not enough time to actually ship it because that's predicting the future. Another difficult call is what bug constitutes a delay of an OS release, especially when the bug report may not have the necessary details to indicate it might be more critical than it seems. TLDR: It's not as easy as saying it's all the EPMs fault. There's plenty of blame to go around because nobody is perfect & predicting the future (which is what scheduling is) is very hard.
ndesaulniers 9y ago
plain emailed diffs? this isn't LKML
- exBarrelSpoiler 9y ago
  The primitive state of tooling in some corners of a sprawling corporate empire would make your head spin.
- bri3d 9y ago
  There have existed teams at every company I have ever worked for who argue that this is the Very Best Way (tm) to develop code. It allows for the True Gatekeepers to keep their hold over the perceived software quality and quickly routs any sort of outside interest from modifying the project in any way the True Gatekeepers don't desire. In entirely broken organizations this is an effective way to allow 2-3 good developers to produce software in spite of the culture. In any organization beyond entirely broken, this prohibits a team from scaling beyond 2-3 key core members and tends to allow that small group to become the BOFH, maintaining a reign of terror over all non-blessed developers, regardless of seniority or title, who dare contribute to the product. In some senses this is the Linus "benevolent dictatorship," but in most, it's a despotic system that collapses as soon as one of the core contributors moves on or retires. In my experience most attempts to move these sorts of teams towards real revision control and development practice lead to an extinction event in which they practice every form of hazing possible to maintain control, requiring careful management. Think extreme code-format nitpicking (combined with a lack of automated tooling or a styleguide that would let "others" get it right), followed by appeals to whichever authority maintained their team, outright hostility, and eventually ultimatum or holding the code hostage.
  - chopin 9y ago
    But how is emailing different from something like eg. Gerrit or pull requests. Ultimately, it's always the maintainers who get to decide what gets into the repo. What you describe is a problem occurring independent of the way changes are merged. I agree that it can be a real problem.
gigatexal 9y ago
Wow. That sounds way too chaotic.

chuckdries 9y ago

Do you think some level of standardization of process would help apple as a whole? Would it help eliminate the relatively poor experience of reddit-OP?

objclxt 9y ago
I think it's both a blessing and a curse. On the one hand it makes it very easy for teams to find an approach that works for them, and does away with a lot of bureaucracy. On the other, it promotes a lot of inefficiencies (I knew of at least 8 separate instances of Phabricator being used by different teams), and makes it very difficult to centralize core tooling and operations. It also makes it very hard for engineers to have internal mobility (you typically would have to do a full interview loop as an internal transfer, which is in stark contrast Google / FB's approach). When I left there was talk of centralizing more tooling to attempt to alleviate some of these issues, but I don't know if any progress has been made since then.
- deleted 9y ago
  [deleted]

majormajor 9y ago

When I've seen "always follow priority order," it's always been somewhere between "[wink wink] that's a good idea, figure out how to link it to the top-priority stuff" and "all priorities are negotiable" anyway. Was your experience similar, or just more of a team truly without external priority ordering?

cageface 9y ago

Based on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? As an outsider it looks to me like they're just pushing too hard to keep up with a yearly release schedule but that's based on no inside knowledge at all.

coolwhhip 9y ago
Shipping gets you power and influence. Outside of personal conviction and judgement, there’s not incentive to do much else.
- fastball 9y ago
  Right, but engineers probably are going to have some level of conviction and judgement when it comes to something they helped build. They're personally invested in a feature they've worked on. If I was an engineer at Apple who worked on iMessage, and I heard acquaintances talk about the bugginess about iMessage, I would probably start out being defensive but then I would regret that we didn't ship a better product. I think an EPM is less likely to feel like that, as they are not so closely tied to the actual product, which is what the Reddit poster was saying.
  - vlovich123 9y ago
    It's easy to blame "the other". Having worked at Apple I never felt like it was the EPM causing quality issues for us. It was us setting an aggressive schedule ourselves and, more importantly, having poor testing. The EPM was mainly just making sure tasks didn't get missed, reprioritizing them as the schedule moved based on discussions with managers & other engineers etc. Sure at some point in the schedule the central EPM committee for a release would start locking things down & punting less critical issues, but that's something that's done for every single release even for engineer-driven companies because you have to ship at some point. Keep in mind also that these EPMs themselves tend to be fairly competent engineers in their own right so these aren't MBAs making decisions, they've just moved on as their career evolved. They care about quality just as much as anyone. The difficult part is making the call of cutting a feature especially if it's a keynote feature because there's just not enough time to actually ship it because that's predicting the future. Another difficult call is what bug constitutes a delay of an OS release, especially when the bug report may not have the necessary details to indicate it might be more critical than it seems. TLDR: It's not as easy as saying it's all the EPMs fault. There's plenty of blame to go around because nobody is perfect & predicting the future (which is what scheduling is) is very hard.

ndesaulniers 9y ago

plain emailed diffs? this isn't LKML

exBarrelSpoiler 9y ago
The primitive state of tooling in some corners of a sprawling corporate empire would make your head spin.
bri3d 9y ago
There have existed teams at every company I have ever worked for who argue that this is the Very Best Way (tm) to develop code. It allows for the True Gatekeepers to keep their hold over the perceived software quality and quickly routs any sort of outside interest from modifying the project in any way the True Gatekeepers don't desire. In entirely broken organizations this is an effective way to allow 2-3 good developers to produce software in spite of the culture. In any organization beyond entirely broken, this prohibits a team from scaling beyond 2-3 key core members and tends to allow that small group to become the BOFH, maintaining a reign of terror over all non-blessed developers, regardless of seniority or title, who dare contribute to the product. In some senses this is the Linus "benevolent dictatorship," but in most, it's a despotic system that collapses as soon as one of the core contributors moves on or retires. In my experience most attempts to move these sorts of teams towards real revision control and development practice lead to an extinction event in which they practice every form of hazing possible to maintain control, requiring careful management. Think extreme code-format nitpicking (combined with a lack of automated tooling or a styleguide that would let "others" get it right), followed by appeals to whichever authority maintained their team, outright hostility, and eventually ultimatum or holding the code hostage.
- chopin 9y ago
  But how is emailing different from something like eg. Gerrit or pull requests. Ultimately, it's always the maintainers who get to decide what gets into the repo. What you describe is a problem occurring independent of the way changes are merged. I agree that it can be a real problem.

gigatexal 9y ago

Wow. That sounds way too chaotic.