6 ms·
Apple’s fix for corrupt binaries
- crazygringo 14y agoDoes anyone know what the actual problem ever was?
- dljsjr 14y agoThere was an error with the FairPlay DRM signing process that lead to a lot of app binaries becoming corrupt; people would go to download updates from the app store, and after doing so would be completely unable to launch the app. Not even a splash screen or a display of the freeze-dried screenshot from multitasking. Half a second of black, then kicked back out to Springboard. Marco did a really good job of cataloging it because Instapaper was one of the affected applications: http://www.marco.org/2012/07/04/app-store-corrupt-binaries http://www.marco.org/2012/07/04/app-store-corrupt-binaries
- jonhendry 14y agoSupposedly one server involved had somehow gotten munged and was producing garbage.
- deleted 14y ago[deleted]
- phuff 14y agoDoes anybody have any insight on how a process like this is debugged at Apple? I've heard, for example, that at Amazon there is a process of blame-finding after a major issue/outage like this, whereas at Google I hear things are more post-mortem let's fix the process that led the human error involved in the outage, rather than blame the person who made the faulty commit. Anybody know how things work inside Apple's culture?
- CervezaPorFavor 14y agoNobody ever lives to tell the story...
- SpikeGronim 14y ago"that at Amazon there is a process of blame-finding" Ex-Amazonian here. It's important to note that Amazon's Cause of Error (COE) process is not about blame. It is about determining what happened, why it happened, and what concrete steps are being taken so that it does not happen again. Individuals are not blamed as part of this process and that's in the official rules. The goal is to iterate and avoid making the same mistakes again.
- phuff 14y agoI've heard from a lot of ex-amazonian's that in practice there's a lot of blaming as part of the process at least in part because of the compensation/promotional processes. But maybe that's changed recently? Of course, I've also heard that there is a wide diversity of culture between teams, so maybe that plays in to it, too.
- RandallBrown 14y agoI'm sure it depends a lot on your team and your manager. Also, some people see a process like that and automatically assume blame is being assigned even if it isn't.
- SpikeGronim 14y agoTotally happens, even though it is not supposed to. If a dev didn't outright break the rules then they shouldn't be blamed. The rules in this case would be something like "a peer must review before deploy" - if you bring down the site after breaking that rule, you're going to get fired most likely. I've never had COEs come up in my or others' performance reviews. If the boss who owns the COE "gets it" and has internalized the old-school Amazon culture then there won't be blame. The bosses really take the hits here, they do get personally blamed for these things. If they can't stand between the team and the more senior management then they are not doing it right. If you and your boss mutually hate each other (which unfortunately I have seen) then it won't go well.
- oemera 14y agoThis is Apple culture at its best: No one ever talks about it. All employees are so loyal to this company that you can't even imagine one speaking about such insights.
- recoiledsnake 14y ago>All employees are so loyal to this company that you can't even imagine one speaking about such insights. I think it is more of fear about losing their job rather than loyalty. It is not uncommon to find that any leak will be tracked down and the leaker summarily fired.
- Flow 14y agoAlso, if you like to speak and write about how things work, I'm sure you have loads of opportunities to do that inside Apple. It's just unprofessional to babble around about new tech like it's a cookie recipe.
- snowwrestler 14y agoFear of what, getting fired? Is it hard for Internet-scale application engineers to get a job in the Valley these days? Is it really so hard to imagine that employees at Apple feel loyalty to the company? Employee loyalty is not an uncommon thing.
- usea 14y agoLoyalty? More like fear. I'm not saying that we should be privy to the inner-workings of every company, but a lack of transparency is hardly a culture worthy of praise.
- oemera 14y agoWhy shouldn't it be? Only cause people don't like it doesn't mean it is a bad thing... You don't have to put all your efforts and insights in blog posts!
- vacri 14y agoNot sharing ways to do things is good? Another example of Apple being the antithesis of open source philosophy.
- thechut 14y agoCan somebody please provide some context for this?
- btucker 14y agohttp://www.marco.org/2012/07/04/app-store-corrupt-binaries http://www.marco.org/2012/07/04/app-store-corrupt-binaries
- thechut 14y agoThanks! That sounds bad, glad I'm not an iOS developer
- pooriaazimi 14y agoCome on... Shit happens. And it was the first glitch in the App Store after 5 years of operations! They've sold 30 billion apps (and updated probably well over 200 billion). :D
- mikeash 14y agoThe first glitch? You're kidding, right?
- pooriaazimi 14y agoWell, many, many things suck about the App Store and the submitting process, but they are not that important. This one really ruined a lot of people's holiday, tons and tons of angry and 1-star reviews and if I'm not mistaken, people lost local data. If it was a little more widespread, it could've been a real fiasco.
- nirvdrum 14y agoMinor issue, but the App Store did not launch when the iPhone did. According to someone on wikipedia, the App Store isn't quite 4 years old yet. Still an impressive record if it truly is the first issue they've had.
- justinschuh 14y agoGood on Apple for fixing this quickly. Even the best team can let a bug like this slip through, and the best solution is a fast response. However, I'm confused by the blog post implying that the solution to unbricking the apps was somehow novel or praiseworthy. This is pretty much textbook for an auto-update system. You just bump the revision to force an application update; there shouldn't be any need to reinstall or muck with the data files. I'd honestly have been shocked if they hadn't been able to handle it just the way they did (which would have been noteworthy, but not in a good way).
- falling 14y agoHe's praising it just because he didn't think of it, being too busy enjoying every news site parroting his whining yesterday.
- mikeash 14y agoWhat good would it have done for him to think of it? He couldn't fix the problem; only Apple could.
- deleted 14y ago[deleted]
- thought_alarm 14y agoHowever, I'm confused by the blog post implying that the solution to unbricking the apps was somehow novel or praiseworthy. In the 4 years of the App Store we've never seen a distribution problem of this magnitude (it would be an absolute nightmare for any devs affected) and we really have no idea how Apple would respond to such a problem. We've also never seen Apple unilaterally update a specific group of apps like this. Many App Store devs likely didn't think Apple would or could take such action, and those of us familiar with what it takes for a device to accept and install new bits are now wondering just how this is being done. Are they manually bumping each affected app's version string? Is there there some hidden field that forces a device to reinstall the same version of an app? It's curious.
- psychotik 14y agoHow did they fix/change the version string, or did they? I don't see how simply re-released a previously released version would cause affected users to update their binaries. What am I missing?
- spaghetti 14y agoThis snafu, the Galaxy Nexus thing and the fact that the simple update to my highly rated and heavily used iOS app has been "waiting for review" for ten days made me start Android Dev. Sorry Apple. You took too long. Hire more reviewers already.