17 ms·
“A Cold War Every Day” inside Apple's internal tools group
- rkho 7y agoA while ago I interviewed for a senior fullstack role with a very homogenous team at Apple -- every single person on the team had been converted from the same consulting company (I believe it was Wipro). It was a bizarre interview with three of the engineers on the team on the same call asking me very specific programming language trivia and nothing about my experience nor ability to navigate complex political structures -- you know, the things that you would expect a senior fullstack engineer to have before advancing them to the next round. Unsurprisingly, I wasn't advanced to the next round. In light of this article, I can't help but wonder if it was all a ruse to simply demand another nepotistic FTE conversion from their same consultancy.
- timwaagh 7y agoPerhaps, but perhaps the role just wasn't fit for you. A lot of 'senior dev' roles are much more about programming language trivia and much less about leadership.
- rkho 7y agoThat could be the case, I just find it difficult to believe that an ICT3 role at Apple of all companies would be about programming language trivia, given that the specific team I spoke with mentioned how they were very cross-collaborative with other LOBs.
- filoleg 7y ago>I just find it difficult to believe that an ICT3 role at Apple of all companies would be about programming language trivia Fully agreed, but I don't think it is like that across Apple at all. It was like that just for that specific team OP was interviewing with, and it makes sense in the context of that whole team being Wipro converts. This is definitely unusual to have a team like that in the first place, so OP just simply got unlucky. Given a company of Apple size, no matter how pleasant and competent an average team is, there will always be a few outlier teams that are just whack and awful to work with. I can confidently say the same about my workplace. I like my team and org a ton (in terms of competency, how the responsibilities are assigned, etc.), and I believe that an average team in the company is very competent and high quality. However, I would be lying if I said that I haven't encountered at least a couple of teams in other orgs at the company that I would never want to work on due to similar issues.
- timwaagh 7y agoYou know something about this specific case. I do not. I hope you land the kind of role you want, if you dont already have it yet.
- deleted 7y ago[deleted]
- creddit 7y agoICT3 is just one step above entry at Apple. It's not a senior role.
- rkho 7y agoSorry, I misinterpreted leveling. It was an ICT4 role. Can't update my comment, so I will just leave it here as-is.
- astrange 7y agoNote that senior isn't actually rare in a large tech company. It goes something like intern, junior, senior, staff, principal, distinguished, fellow. It's hard to get above "staff".
- voz_ 7y ago> A lot of 'senior dev' roles are much more about programming language trivia and much less about leadership. I am curious - where do you work? Asking so that I can potentially avoid that company, as this is the most inane and borderline backwards thing I've ever seen written about senior engineering.
- uwuhn 7y agoGoing to guess defense contractor.
- timwaagh 7y agoClose enough. I work for a 'body shop' and I'm currently working for a european government client.
- voz_ 7y agoAhh europe - where, anecdotally, all startup and eng is done backwards.
- mkohlmyr 7y agoPoor take. There are plenty of good engineering organisations in Europe, just as there are more than enough terrible ones in the US. That being said I agree I wouldn't want to work somewhere like what was described above. It's certainly not a style of interview I would approve of or conduct myself.
- piva00 7y agoNot sure what the jab is about. OP works for a body shop in a governmental org, where does that let you imply that startup and eng are backwards as neither are OP's experiences? If the anecdote is related to your experiences working on FB then, please, specify it, blanket statements aren't what I expect from HN comments.
- watwut 7y agoEveryone above 30 is senior developer. When they all consider themselves leaders and try to lead simultaneously, it is pretty much political hell. In some startups, everyone above 25 is senior developers. Everyone else is junior, there is no middle role. So it is not like companies had much choice how to label people. Imo, most of what people write about "senior engineering" is what they personally admire in this or that senior, what they aspire to be or want others to be. It is not what actually role labeled "senior developer" is.
- mabbo 7y agoI can't tell if you're being sarcastic, or if you live in some kind of Kafkaesque dystopia I never want to be part of. At my employer, the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it.
- hermitdev 7y ago> the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it. I've +20 years developing professionally. I remember the stuff I use day to day, and know how to navigate the docs of the technologies I use frequently for the less used tidbits. Being a good Senior isnt about knowing the answers to everything, but knowing how to find the answers quickly. Also, how to ask the right questions. I don't know is always an acceptable answer, but should be followed up with "I'll get back to you in an hour or two.", or something to that effect (obviously for some problems, may need to experiment/prototype).
- WWLink 7y agoWhat, you don't remember the argument order to every common file/memory operation like the back of your hand?! /s
- fulafel 7y agoPossibly the GP meant there is not much leadership skill required in their senior dev roles and the amount of programming language trivia is same as everywhere.
- RugnirViking 7y agoWhile I hate to assume people's position, I would venture a guess that the parent is a junior developer who has had one or two experiences that lead them to believe they are better at programming than some senior devs in their company, and has yet to realise how much of a lead they have in soft skills.
- gwbas1c 7y agoSome people just don't know how to run an interview
- seemslegit 7y agoIs there any big consumer product company out there that doesn't deprioritize management and engineering talent for internal tools and systems ?
- dcolkitt 7y agoIt's an interesting question. And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies. That doesn't seem like an intuitive conclusion, but it's a hard circle to square. As someone who mostly builds internal tools (though in a very different context), it certainly seems like good systems are an Archimedean lever that acts as a force multiplier across the entire org. Yet, if good internal tooling was important, you think at least one company would have been able to achieve success partially on this basis. The consumer product industry is extremely competitive, so the counter-hypothesis of entrenched sclerotic incumbents refusing to pick up $20 bills off the sidewalk doesn't really seem likely.
- seemslegit 7y agoInternal tools are used by employees who unlike consumers will tolerate more usability friction, non-critical bugs and feature delays than consumers and will compensate for them with their own effort so the tools can still be critical in their core functionality but not polished or well-maintained. Ironically and anecdotally, WeWork tried to build a strong engineering org for their internal systems before it blew up.
- MereInterest 7y ago> And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies. That is on the implicit assumption that businesses are reasonable, and accurately know how to manage tradeoffs. Your counter-hypothesis seems rather likely to me, if we look at the internal behavior of a company. When negotiating, the more dots that you need to connect before reaching "and then we get more money", the weaker of a negotiating position you have. It's the same reason why sales teams get bonuses for making sales, but developers don't get bonuses for enabling sales.
- danzig13 7y agoSounds exactly like government contracting.
- throwaway321213 7y agoWhy are you afraid to mention the word "Indian"?
- larrik 7y agoIs this the team handling iTunesConnect and other such "services"? I'd definitely believe it.
- danpalmer 7y agoFrom what I understand, probably not. iTunesConnect has had an overhaul in recent years to give it a much shinier and more reliable frontend. I believe the original was closer to the original WebObjects code, and now they've extracted out the user-facing bit into something a bit more modern. There's a lot of legacy WebObjects stuff around the iTunes backend, and my guess is that it's good engineers held back by old tech, rather than just bad engineers.
- saagarjha 7y agoIt's App Store Connect now!
- captainredbeard 7y agoIt’s their sister org for lack of a better term.
- rllearneratwork 7y agono
- madeofpalk 7y agoIS&T is purely internal. Basically no one outside of Apple uses anything IS&T makes.
- nemo 7y agoIS&T is largely internal, but there are a number of IS&T owned external web apps for registration, surveys, and support related things. I worked in IS&T on teams developing them years ago.
- saagarjha 7y agoNo, services are run by Apple Media Products under Eddy Cue. They're not great, either, but I think they have more full-time engineers than IS&T does.
- Waterluvian 7y agoI could use some blog posts about quite functional teams. Like someone on a team for a decade talking about why it works so well and discusses cultural and political issues and how the team overcame them.
- leoc 7y agoSee my old comment https://news.ycombinator.com/item?id=19468090#19468699 https://news.ycombinator.com/item?id=19468090#19468699 . That kind of thing used to have a much higher public profile about 30 years ago.
- ChrisMarshallNY 7y agoI managed a high-functioning C++ image processing group for 25 years. I kept senior-level engineers on staff for decades. We tended to write "engine code." Like pipelines and whatnot. The company had a very (very) long tradition of engineering (Like, 100 years). There were many downsides to the way they worked (over-structured, mostly), but they always gave my team and me a great deal of respect. They didn't use contractors very often. To this day, design quality, code quality, product quality, documentation, and focus on deliverable are the major cornerstones of my software work, but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience. I guess I was in a silo of quality-focus, all those years. They would treat very minor bugs like extinction-level emergencies (not fun).
- person_of_color 7y agoIs this company still in NYC?
- ChrisMarshallNY 7y agoNot NYC. Long Island, but headquartered in Tokyo. I’m avoiding directly naming them, so I don’t have to deal with any agita, but they make photographic equipment, and are a household name.
- pm90 7y ago
- captainredbeard 7y agoIS&T shouldn’t be part of Apple. It’s culture and engineering talent aren’t representative of core engineering.
- chillacy 7y agoIn the old adage that "you get what you pay for", the internal tools at Apple aren't quite the best as a result. The Radar tool I recall was pretty horrible in particular.
- Austin_Conlon 7y agoDoes IS&T also build the developer-facing Feedback Assistant?
- saagarjha 7y agoNo, there's another team for that.
- astrange 7y agoRadar is the world's best bug tracker. The sole reason for this is that the originator gets to close bugs, not the fixer.
- taormina 7y ago
- 5cott0 7y agoBody shop wars. What gets really interesting is working for a body shop with consultants from 2 other shops on the same team.
- coleca 7y agoI had a similar experience w/Apple. A startup I worked for was brought into Cupertino for a meeting w/their internal business teams. They wanted us to build them an app which seemed relatively simple involving their internal Cafe Mac cafeterias, data centers, and possibly retail locations. It was something that a company like Apple could build in their sleep. But the business folks we met with said that's how it is at Apple, all the engineering talent goes towards the product side and almost nothing is left for internal IT. They told us how they struggled to get anything done and there were almost no resources available, so the business teams had to go and hire their own IT if they needed things done.
- deleted 7y ago[deleted]
- ethbro 7y ago> But the business folks we met with said that's how it is at Apple That's how it is at most companies. Split orgs (IT & business) result in only work of sufficient size, scope, and impact being able to cross the barrier. Multi-year logistics management rewrite? You'll get two IT teams. Frank in accounting needs to schedule a daily job to copy from one datastore to another? He doesn't have permissions to, and has been doing it manually for the last 5 years. IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them.
- jldugger 7y agoIS&T is run by the CFO and it shows.
- riskneutral 7y agoSounds like the politics of every large corporate IT department, but with some twists given that it's Apple. Maybe this helps explain why Apple is not Amazon.
- dehrmann 7y agoIf there's a company CEOs should be afraid of, it's Amazon. They killed bookstores, mall stores, how many other retailers? They were the first major cloud provider. They tried their hand at phones, streaming video, and streaming music, and the bought a grocery store. There's nothing they won't try, and they have the war chest to do it.
- ping_pong 7y agoOne of the world's most successful and wealthy companies in history shouldn't be bottom fishing for the best "deal" on employees. I wonder how much better they could be if they decided to hire at the top of the market like Netflix instead of the bottom. That is probably why Apple software is among the worst in the industry, and it's a shame. I know IS&T isn't strictly software like iTunes but it fits the pattern I've seen at Apple. But I guess if it doesn't affect their sales maybe I'm wrong and it doesn't matter and they are right.
- saagarjha 7y agoThe quality of Apple's software engineering organization is head and shoulders above IS&T's; iTunes (…and Apple Music, News, etc.) is run under a different (services) group.
- IST-Throwaway 7y agoFascinating to see an article about IS&T, definitely not something I ever expected to hear about on HN! I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed. Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users. As far as we could tell, IS&T was run as a unique company inside Apple, which did it's fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org. Although my experience is several years old, everything in this article rings true. The contracting companies they had us working for were taking a huge cut, the quality of the code they produced was dismal, (as soon as we were no longer allowed to re-write their code major things began breaking almost immediately) and people getting transferred around constantly and having no time to understand any one project was common. (rkho's comment about their hiring process seeming like it was simply a beard for a nepotistic contractor conversion was something we definitely saw a number of times.) All in all it was an extremely eye-opening experience. Considering how "do it the Apple way" every other department we interacted with was, being in the IS&T buildings was like landing on an alien planet.
- _bxg1 7y agoWhen will companies learn that cheap development labor always ends up being more expensive than starting with quality? This same story has been playing out over and over for decades.
- 7y ago
- motohagiography 7y agoSounds like a bank, a government agency, or any business over a certain size. I tell people you only call it politics when you are losing. More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked. Instead of a hierarchy, rethink a company as a hub and spoke model with concentric rings. The main differences are the implication in a hierarchy that there is "gravity," keeping people down and that they need energy and leverage to climb "up," which further implies there is a place to "fall," and that there is only one way "up," instead of many possible paths to the centre from all directions. There is no gravity, only gates and barriers, and even these are just information. Politics is how a middle manager runs interference and creates distractions to make sure you can't see over, around, or through them, and that the people behind them closer to the money can't see you. Tech is usually outside the main perimeter, mediated by contracting companies or middle managers whose job is to compartmentalize the value people create, and be sure it is replaceable. Viewed this way, of course this demented political farce is how Apple works, because it's how everything seems to work when you have internalized the precise and specific mental model someone uses to take advantage of you. Sorry if you can't unsee it now, but hopefully it will be funny and we can get good, competent people who value tangible skills into positions of power.
- taneq 7y ago> More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked. This is a brilliant description of it. The money people aren't stupid, despite what the workers think. They just have more information and different motivations which, in most larger organisations, they actively avoid explaining to the operations people. Edit: Changed 'an incredible description' to 'a brilliant description' because it is entirely credible.
- coffeemug 7y agoCould you explain this more? Who are the operations people? (finance? HR?) Who is the money side? (product? sales?) How exactly would "operations people" have leverage if they knew how the "money side" worked? Like, what specifically would they do? Also leverage over whom? And ultimately, couldn't whoever all these groups eventually report to (the CEO but possibly lower) resolve all this by firing some people? I don't have a good model of how these things work. I encountered it first hand, but couldn't develop a good model after a long time of thinking about it. Tons of things happened that I didn't understand at all, and I couldn't understand why higher-level people wouldn't just resolve it. Help understand this better?
- ummonk 7y ago"Don't work in IS&T" has been the common wisdom in Blind for years now. Interesting how long it takes for the media to pick up stories like this.
- ksec 7y agoDoes Radar belongs to IS&T? And I wonder if this IS&T power grabbing and politics happen and grow within the past 10 years. i.e Happen under Tim Cook's watch. It surely doesn't seems very Steve Jobs approach.
- apple2019 7y agoI would be grateful if any Apple insiders here can discuss the risk of layoffs if one were to join the company now. I have an offer to join an engineering team at Apple, but the chances of getting hit by layoffs is giving me pause. More details on my situation are here: https://news.ycombinator.com/item?id=22809769 https://news.ycombinator.com/item?id=22809769
- doingmyting 7y agoI don't work for Apple but they have 200 billion in cash on hand. I doubt you'd be at any lower risk of being laid off at any other company all else being equal
- sjg007 7y agoApple has a ton of cash. I think layoffs are unlikely. Many times new hires are protected as well for a bit.
- apple2019 7y agoDo you work for Apple? Might you have any insight into how the company decides how / who to layoff, when the time comes? I would have thought that the newest would be the most vulnerable, since they haven't had the time to absorb institutional knowledge and make internal connections.
- stormbrew 7y agoIS&S is not the same as IS&T. Totally different orgs.
- samdung 7y agoThe contract companies (Wipro, Infosys) hire graduates by the 100's from mediocre engineering colleges, slap on a little bit of training and advertise them as world class. The starting salaries are something like INR 25,000 (USD 350) per month. An onshore posting for these low skilled graduates is a quick way to make money. So they suck up to their managers who suck up to bigger managers. Their main criteria is to get hold of contract work by undercutting, knowing very well there is vendor lock in.
- rclayton 7y agoLol. Just like Govt contracting.
- PeterStuer 7y ago"They’re just fighting for the roles,” Sabapathy told me. “That’s all they care about, not the work, not the deliverables, the effort they put in, or even talent. They’re not looking for any of those aspects.” This is spot on for every large organization I ever consulted for. Moreover. Even in much smaller businesses, typically starting at around 70-100 people (sooner if management has prior large business experience) you can already see this pattern starting to seep in. This is also why the whole private vs public sector efficiency debate is farcical. They are both identical in this aspect.
- epberry 7y ago> For Apple, fixing its broken IS&T division would not only be the right thing to do from a moral standpoint — it would help the company’s business as well. If Apple is going to become inventive again, it will need to give its employees more time to develop new ideas. I actually laughed. What kinds of applications do people think IS&T is building? Supply chain management, HR, support software (Apple forums anyone?)... I don't see how Apple sticking its core engineering resources or at all innovating within IS&T helps their business. Steve was the one that coined the 10X rule, or popularized it. That idea must still hold within Apple's executive ranks. I guarantee Apple's top engineers never interface with IS&T and their time won't be saved by improving it. In fact I would bet that managing the deliverable of one of IS&T's projects is a sign that you're not one of the golden ones. That said, I obviously think these contractors should be treated much better and overworking people for low pay is never okay. Maybe Apple could use some smaller dev shops that treat their people better.
- xp84 7y agoThe part I don't get is, why does a company, especially one like Apple who would count software quality as a core competency, choose to pay $120-150 an hour to get contractors who aren't even good enough to command a salary (or hourly rate) of half that. I get why like, United Airlines does that. Or the US Government. They probably rightly assume that they can't properly manage a huge software development or IT org. But Apple? Really? What the hell are they gaining? If they paid a developer full time $120k a year plus benefits that still beats $120+ an hour. WTF.
- chmaynard 7y agoThe days when the primary mission of IS&T was to develop custom software to run Apple are probably long past. The big exception is Radar, an ideosyncratic issue-tracking system which was developed in-house by IS&T around 1990. As far as I know, Radar is still used in R&D and elsewhere at Apple. Tim Cook probably couldn't care less about the internal affairs of IS&T. As long as they do their job, play by the rules, and don't impose too much overhead on the company, it's all good.