10 ms·
This comment from a former employee on the iOS team provides some more context around the software quality issues. I think this is the more interesting point a
by askafriend 9y ago
This comment from a former employee on the iOS team provides some more context around the software quality issues.
I think this is the more interesting point around which we should have this discussion.
https://www.reddit.com/r/apple/comments/7x0eif/how_apple_plans_to_root_out_bugs_and_revamp/du4z6bx/ https://www.reddit.com/r/apple/comments/7x0eif/how_apple_pla...
- deleted 9y ago[deleted]
- objclxt 9y agoI have a slightly different perspective (I worked on iOS 7 - 11). One of the things about Apple that's different to many of the other tech conglomerates is that engineering teams are given huge freedom to work the way they want (for better and for worse). Because engineering is so fragmented you should always take what one person says about the process with a pinch of salt. So that means there are hundreds of source repos, patch review systems, and the like. When I was there I saw teams using Phabricator, GitLab, GitHub, plain e-mailed diffs, and more. In the same way EPMs and project process varies massively across teams, so what that employee experienced is by no means the norm. Certainly I didn't find this to be true at all: > Nothing could be worked on if it wasn’t in Radar with a priority number attached and signed off by the teams’ EPM. No room for a side project or time away from your daily duties because there were always P1s to fix. In fact, I worked on a major feature that was keynoted one year that came out purely because of my side project work. If anything it was a frustration to enter the "real world" where PMs had so much more power and influence than they did when I was at Apple (where PMs didn't really exist, and EPMs existed only to manage project timelines and drive features that HI / ID came up with).
- chuckdries 9y agoDo you think some level of standardization of process would help apple as a whole? Would it help eliminate the relatively poor experience of reddit-OP?
- objclxt 9y agoI think it's both a blessing and a curse. On the one hand it makes it very easy for teams to find an approach that works for them, and does away with a lot of bureaucracy. On the other, it promotes a lot of inefficiencies (I knew of at least 8 separate instances of Phabricator being used by different teams), and makes it very difficult to centralize core tooling and operations. It also makes it very hard for engineers to have internal mobility (you typically would have to do a full interview loop as an internal transfer, which is in stark contrast Google / FB's approach). When I left there was talk of centralizing more tooling to attempt to alleviate some of these issues, but I don't know if any progress has been made since then.
- deleted 9y ago[deleted]
- majormajor 9y agoWhen I've seen "always follow priority order," it's always been somewhere between "[wink wink] that's a good idea, figure out how to link it to the top-priority stuff" and "all priorities are negotiable" anyway. Was your experience similar, or just more of a team truly without external priority ordering?
- cageface 9y agoBased on your experience there what would you say is the cause of the recent slip in software quality in both iOS and macOS? As an outsider it looks to me like they're just pushing too hard to keep up with a yearly release schedule but that's based on no inside knowledge at all.
- coolwhhip 9y agoShipping gets you power and influence. Outside of personal conviction and judgement, there’s not incentive to do much else.
- fastball 9y agoRight, but engineers probably are going to have some level of conviction and judgement when it comes to something they helped build. They're personally invested in a feature they've worked on. If I was an engineer at Apple who worked on iMessage, and I heard acquaintances talk about the bugginess about iMessage, I would probably start out being defensive but then I would regret that we didn't ship a better product. I think an EPM is less likely to feel like that, as they are not so closely tied to the actual product, which is what the Reddit poster was saying.
- vlovich123 9y agoIt's easy to blame "the other". Having worked at Apple I never felt like it was the EPM causing quality issues for us. It was us setting an aggressive schedule ourselves and, more importantly, having poor testing. The EPM was mainly just making sure tasks didn't get missed, reprioritizing them as the schedule moved based on discussions with managers & other engineers etc. Sure at some point in the schedule the central EPM committee for a release would start locking things down & punting less critical issues, but that's something that's done for every single release even for engineer-driven companies because you have to ship at some point. Keep in mind also that these EPMs themselves tend to be fairly competent engineers in their own right so these aren't MBAs making decisions, they've just moved on as their career evolved. They care about quality just as much as anyone. The difficult part is making the call of cutting a feature especially if it's a keynote feature because there's just not enough time to actually ship it because that's predicting the future. Another difficult call is what bug constitutes a delay of an OS release, especially when the bug report may not have the necessary details to indicate it might be more critical than it seems. TLDR: It's not as easy as saying it's all the EPMs fault. There's plenty of blame to go around because nobody is perfect & predicting the future (which is what scheduling is) is very hard.
- ndesaulniers 9y agoplain emailed diffs? this isn't LKML
- exBarrelSpoiler 9y agoThe primitive state of tooling in some corners of a sprawling corporate empire would make your head spin.
- bri3d 9y agoThere have existed teams at every company I have ever worked for who argue that this is the Very Best Way (tm) to develop code. It allows for the True Gatekeepers to keep their hold over the perceived software quality and quickly routs any sort of outside interest from modifying the project in any way the True Gatekeepers don't desire. In entirely broken organizations this is an effective way to allow 2-3 good developers to produce software in spite of the culture. In any organization beyond entirely broken, this prohibits a team from scaling beyond 2-3 key core members and tends to allow that small group to become the BOFH, maintaining a reign of terror over all non-blessed developers, regardless of seniority or title, who dare contribute to the product. In some senses this is the Linus "benevolent dictatorship," but in most, it's a despotic system that collapses as soon as one of the core contributors moves on or retires. In my experience most attempts to move these sorts of teams towards real revision control and development practice lead to an extinction event in which they practice every form of hazing possible to maintain control, requiring careful management. Think extreme code-format nitpicking (combined with a lack of automated tooling or a styleguide that would let "others" get it right), followed by appeals to whichever authority maintained their team, outright hostility, and eventually ultimatum or holding the code hostage.
- chopin 9y agoBut how is emailing different from something like eg. Gerrit or pull requests. Ultimately, it's always the maintainers who get to decide what gets into the repo. What you describe is a problem occurring independent of the way changes are merged. I agree that it can be a real problem.
- gigatexal 9y agoWow. That sounds way too chaotic.
- bpyne 9y agoBased on this comment, it sounds like engineers are more like line workers. "EMs at Apple are powerless to push back. Every engineer's performance is tied to the number of Radars fixed and closed. Every EM's performance is tied to their team's total Radars fixed and closed, so they have an incentive to keep everyone focused on the prize."
- Klathmon 9y agoAlways take comments like that with a grain of salt. The people who feel they are being ignored or feel they aren't being used to their full potential are going to look for someone or something to blame. I'm sure there is some amount of "higher number of fixes is better" going on (any larger company has it), but i've seen poor team members read into things like this way too much in order to justify why they aren't doing well in comparison to others. It may be true that apple weighs stats like that too heavily, but we aren't going to get a good picture of how heavily they weigh them from a few anecdotes.
- objclxt 9y agoNever saw or heard anything about my performance being tied to the number of radars I fixed or closed, and I never heard anyone else I worked with say that either. Doesn't mean it didn't happen, like every big company there are managers who are good, and managers who don't really get it.
- bpyne 9y agoThanks for the perspective Klathmon and objclxt. The person's comment hits a fear of mine that people still draw the false analogy of manufacturing for software development.
- throwaway7187 9y agoPerfect time for the ol' throwaway... I'm a former iOS EPM (not speaking for Apple, obviously, since I don't work there anymore), and although the Reddit commenter got the atmosphere of constant crisis right, he/she is misplacing the blame and misunderstanding the power dynamic. EPMs at Apple essentially have zero power over engineers' workload. They take the list of stuff the engineering managers said they want to get done this year and say "You guys are crazy, you'll never be able to do this without 3x the hours/manpower." Then they proceed to drive the team as hard as necessary to make sure that they actually deliver what they said they were going to deliver. That's it. The idea that there is this cabal of mighty EPMs twirling their mustaches and loading developers down with work is pretty far from reality. It's true that you shouldn't be working on anything not in Radar (the bug tracker) but this is true anywhere you'll work. Project managers however do not sign developers up for all those radars--on the contrary--we're usually trying desperately to help you get rid of scope and get the task list down to what's actually do-able! One of the great things that IMHO sets Apple apart is how engineering-driven they are. I've never worked anywhere else where engineers had so much freedom to decide what they're working on. The fact that they always decide to work on 3x what they can actually achieve is kind of on them. But that drive to try to do so much is part of what keeps innovation strong at Apple.
- exBarrelSpoiler 9y agoThe comment was half-right in describing the situation radar-obsessed culture (Radar-Driven Development?) It places the blame on EPMs because EPMs are just the bearer of bad news, middlemen who carry out the whims of the UI Gods and VPs. The original commenter couldn't see those above the EPMs, the ones overseeing the BRBs and other inquisitions.
- closeparen 9y agoWe use the bug tracker for coordination and collaboration. If you have a customer-affecting bug or a stakeholder-facing feature you’ll use the relevant task to discuss it, but we certainly don’t let the bug tracker dictate how engineers spend their time. That sounds like hell.
- 9y ago
- abalone 9y agoBreaking: Engineer Thinks PM is a Bozo. Film at 11. It's not really context. The real problem here is the premise that "Apple has software quality issues" is taken as a given, without support. Sinofsky makes the point that Apple is operating a such a huge, unfathomable scale and depth -- and this is coming from one of Microsoft's top guys in its heyday -- that even rare bugs that affect 0.01% of users translate into a "stadium full" of people, affecting their lives deeply. So, combine that with a sensationalist, anecdote-driven news cycle and you get an optics problem. He observes that "in any absolute sense," Apple's SW+HW quality has "exceeded everyone else."[1] [1] https://twitter.com/stevesi/status/963142502604779520 https://twitter.com/stevesi/status/963142502604779520 (via Gruber)
- PakG1 9y agoIf you're not receiving criticism, you don't matter.
- Someone1234 9y ago"There are only two kinds of languages: the ones people complain about and the ones nobody uses" - Bjarne Stroustrup I've found that quote to apply to everything. Familiarity breads contempt, even just from boredom, or wanting to shift blame for the monotony of the task you're using the tool for. For example I think Excel is a great piece of software, but I know someone who spends all day every day in it doing dull but important spreadsheets. They're what I consider an Excel expert. But they hate Excel because every little minor issue is magnified for them because they spend all day every day in it.
- gurkendoktor 9y agoI agree that this kind of criticism exists. However, that doesn't mean that all criticism is invalid, or that a large amount of criticism is necessarily a marker of success. Some software examples: Internet Explorer has not been winning the browser wars for a decade. I've deleted Skype after its interface had been sufficiently ruined, and I don't know anyone who still uses it for business. When iTunes got too annoying, I left its ecosystem for Spotify. It's not looking good for Siri in the battle for the smart home. Speaking of programming languages, I complain a lot about Swift, don't use it, and am moving in the general direction of IntelliJ IDEs. Excel and C++ got lucky, they will be hard to replace.
- Bahamut 9y agoThat post is not particularly useful - each team/org works differently. “Working on iOS” is such a vague introduction, it doesn’t really give any indication of his/her role or a feel for how every team/org operates. I can claim the same, and my own experience has been far from that poster’s experience. Any time I have volunteered to spend extra time to accomplish a goal, I have been told by my manager that it isn’t necessary for me to do so. We also do our own planning on our team and the management chain is there to help us be successful.
- draw_down 9y agoNo idea if that’s Apple’s issue, but I find it easy to believe. Many PMs have made life shitty in the course of my career.
- cwyers 9y ago> Nothing could be worked on if it wasn’t in Radar with a priority number attached and signed off by the teams’ EPM. Is this supposed to sound bad? Because it just sounds... professional.
- closeparen 9y agoIt sounds professional to need permission to spend time on something? I typically associate professionalism with the model where you’re accountable to the functionality, impact, quality, and health of your components, with an EM supporting your negotiations around scope and timelines, and letting you budget your own time.
- cwyers 9y agoI mean, nothing in the quote suggests that programmers can't submit items to Radar, that they can't lobby for them to be assigned a high priority, etc. Pretty much every software project beyond a certain scale uses an issue tracker of some sort.
- closeparen 9y agoIf you need to lobby for permission to work on something broadly within the scope of what you're paid to work on, you aren't being treated like a professional. An issue tracker is a communication tool for use in the context of a largely self-directed activity. The only role at my company where your day-to-day activity is dictated to you by the tickets on your screen is customer support. On my team, if you're only working the asks handed down from above and not proactively taking initiative to make things better, you're barely pulling your weight.
- zimpenfish 9y ago> On my team, if you're only working the asks handed down from above and not proactively taking initiative to make things better, you're barely pulling your weight. The problem with a lot of engineers is that "make things better" often doesn't translate into "keeps customers happy and money coming in". I could work on making our internal software "better" for the next two years but it wouldn't bring in enough money from customers to justify paying me to do it. And I'm smart enough to know that I have zero idea about what will make things "better" for the actual customers.
- raspasov 9y agoThis sounds to me so close to this: https://www.youtube.com/watch?v=l4dCJJFuMsE https://www.youtube.com/watch?v=l4dCJJFuMsE (Steve Jobs on process vs content)
- deleted 9y ago[deleted]
- perseusprime11 9y agoInteresting comment. I thought Project Managers are a thing in the past. Looks like it is alive and kicking and taken on a new life at Apple.
- exBarrelSpoiler 9y agoThe role of EPMs has been known to the public for some time: https://www.quora.com/What-is-the-role-of-an-engineering-project-manager-at-Apple https://www.quora.com/What-is-the-role-of-an-engineering-pro...