7 ms·

My money is personally on massive, unrecoverable data loss. If you've ever poked around with the way that Apple's website works, you can see that the entire pl

by nwh 13y ago

My money is personally on massive, unrecoverable data loss. If you've ever poked around with the way that Apple's website works, you can see that the entire place is a huge mess. There's old servers running ancient (pre-2004) perl scripts alongside the brand new iCloud gear. I can't imagine how the authentication for AppleID is working as login details still work on the ancient pages (think pinstripes and glassy buttons). Depending what URL you hit, the webserver is using php3, php4, perl, python or maybe WebObjects (java). At one point I wrote a scraper that was targeting one of their product pages, and kept getting random, unexplainable results. It turned out that one of their product areas was behind a round-robin load balancer, with three completely different apache versions on each server. The page was dying on one but not the other two. In the end I just had to repetitively scrape until I hit a good response. Even the domain for the "maintenance" page for the developer section is telling. It's just a broken template system regurgitating a bit of the homepage. http://devimages.apple.com/maintenance/ http://devimages.apple.com/maintenance/ http://devimages.apple.com/ http://devimages.apple.com/ Truly a hacked together system. Some engineers at Apple must be having a truly awful weekend, no matter the cause and solution.

nicholassmith 13y ago
I've spent more time using the developer resources recently, and I completely agree. Clicking one link will take you an entirely different feeling architecture, like parts have been built on top without thinking about what the impacts could be. It's probably been a house of cards for a long and it's now fallen down.
saurik 13y ago
Interesting; the http://devimages.apple.com/ http://devimages.apple.com/ URL seems to return the underlying contents of the file, bypassing the SSI (I'm so happy they use SSI... seriously: I love SSI ;P). You can then see the raw  directives, and pull the individual parts. (It isn't quite then fair to say that it is a "broken bit of their template system"; it is more that it is a poor way to setup a static large-file caching endpoint, and may itself lead to a security vulnerability. To be clear: that's probably worse ;P.)
- nwh 13y ago
  Ah, I didn't recognise Server Side Includes, I've never used them before. There's a lot more weird around apple.com if you go looking. I'd love to hear the story behind their MX/SMTP hostnames (thousands of them) getting called "badger" and "msbadger".
  - deleted 13y ago
    [deleted]
- DoubleMalt 13y ago
  Careful. You might have committed a felony by accessing a page with SSIs. It was definitely not intended to be accessed under that URL by you ;)
  - marshray 13y ago
    That's no winking matter.
- legutierr 13y ago
  Might I ask what vulnerabilities exist for SSI, and why you describe SSI as a "poor way to setup a static large-file caching endpoint"? I haven't used SSI before, but after researching various technologies, it looks like a solution for my own caching use-case, and I am planning on implementing it. I realize that it's a rather old technology, but as long as there's nothing wrong with it, I think it could work for me. Is there something wrong with it? The only thing that I have seen is that the "exec" command is dangerous, but NGINX doesn't seem to implement it.
  - sergiosgc 13y ago
    If you are now sampling this kind of technology, have a look at ESI (edge side includes), and have a look at varnish cache. You'll be pleasantly surprised.
  - saurik 13y ago
    I did not describe "SSI" as a "poor way to setup a static large-file caching endpoint"; that description was for the way they setup "devimages." to overlap with the content for "www.". Let's imagine that the code in question was not SSI, but instead PHP. You might have this running behind a copy of Apache and mod_php. Your website includes both code and images. This website runs like this, but maybe the way you configured Apache is too slow to handle downloading all the images: these large-file static assets need to be handled by a different system. So, you install a copy of nginx and point it at the same folder. You give this copy of nginx a new hostname, such as "devimages.example.com" as opposed to the Apache "www.example.com". Problem solved, right? No. Now if you go to http://devimages.example.com/index.php http://devimages.example.com/index.php you get access to the source code to the website (which is generally considered a big security failure). That seems to be what's happening here: the "devimages." hostname is configured to bypass the SSI parser, and you are getting to see how the page is composited. This could be a problem. However, it also might not be a problem: it largely depends, which is why I was just idly speculating; it certainly feels wrong, though. As for SSI? I was serious: I love SSI, all my stuff uses it.
filleokus 13y ago
I work at an Apple Premium Reseller, and when we order stuff from Apple we use http://reseller.apple.com http://reseller.apple.com (which now seems to redirect to https://ecommerce.apple.com/ https://ecommerce.apple.com/) and all those systems feels hacked together just as you're saying. The Reseller Store has been down the whole time the Developer Center has been down. Even more interesting perhaps, our company's "master apple id", which is used to place multi 100K USD orders from Apple, got some strange seemingly manual password reset from Apple about the same time as the Developer Portal went down. Apple's hardware service portal (http://gsx.apple.com http://gsx.apple.com) has also been even more flakey this week than normal. Something is broken over at Apple.
- sudont 13y ago
  "Won't fix, as designed." B2B web apps have been hacky since time immemorial. Where I work, a different engineering team wrote a CMS in a bunch of Java with somehow Perl CGI to handle all of the little hacks that couldn't be put in to the main code base. I found this out by fixing a hack named emailservice_bkp_new2.cgi which was a hard coded rss parser. Our other client relations applications are even worse, often in IIS 6.0, and dating from 2003. Simply put, there isn't much "business reason" behind fixing something that isn't broken, but very shoddy. The difference now is that "prosumer" application developers have enough economic weight to register as a force, but not enough to cause Apple to reconsider their entrenched developer relations.
- tazjin 13y ago
  Do you have access to the CRT? If yes, does that still work? It hinges on the same login infrastructure as reseller.apple.com AFAIK. I have nothing to do with Apple anymore so I can't check it myself :)
e3pi 13y ago
> My money is personally on massive, unrecoverable data loss. Do we read this, you have an AAPL short order in tomorrow's market open? I find no financial news on this.
- nwh 13y ago
  I own no stock in any company. I have a reasonably pessimistic view based solely on the way this is being handled, and the way in which their web applications behave.
- nknighthb 13y ago
  Through a combination of direct ownership and mutual funds, I own thousands of dollars of Apple stock, and would be materially harmed by a major drop in its price. I also own numerous Macs and iOS devices, and am indirectly harmed by Mac and iOS developers being unable to access code and information important to their work. The company I work for may also be harmed by this, as we engage in some iOS development (though I'm not directly involved in that these days). I find the explanation of massive, unrecoverable data loss plausible, and your insinuation inane.
straight_talk_2 13y ago
Unrecoverable? I'd bet NSA has a complete backup ;-)
yapcguy 13y ago
Maybe it's worse? Maybe it's related to an earlier incident? From Feb this year: "Earlier today it was reported that Apple’s computers had been compromised by a zero-day exploit in Java. Apple quickly released an update to patch the flaw for all Macs, but not before some of its own employees had been hacked.... A site called iPhoneDevSDK has been revealed as the means by which a dangerous exploit was injected via a Java plugin." http://www.cultofmac.com/216618/this-iphone-developer-forum-is-responsible-for-hacking-apple-employees-macs/ http://www.cultofmac.com/216618/this-iphone-developer-forum-...
ChuckMcM 13y ago
Well if this is the case then there is some money to be made shorting the stock on Monday :-) I really like some of Apple's products (my Macbook pro is my favorite laptop of all time so far) but sometimes, like dealing with them as a guy who buys Macs for everyone who works for me, I think "Man they guys have a ways to go when it comes to this enterprise service stuff." I hope that what ever it is, it doesn't put too big a crimp in them.
ychw 13y ago
I think so too. If you look back a couple of years, this kind of things (even if not as severe) happened several times. I actually blogged about it (when their DNS was messed up): http://www.clingmarks.com/apple-please-be-serious-about-itc/851 http://www.clingmarks.com/apple-please-be-serious-about-itc/... http://www.clingmarks.com/ios-vs-android-mobile-dev-platform-heads-up/601 http://www.clingmarks.com/ios-vs-android-mobile-dev-platform...
tlogan 13y ago
I don't believe they have massive, unrecoverable data loss. Think about that: If they are so retarded that they dont have backup then Apple is really doomed. Really. But I think they have good IT... I think they detected serious security breach (which might be related to some incompetence but security is very very hard).

by nwh 13y ago

nicholassmith 13y ago

I've spent more time using the developer resources recently, and I completely agree. Clicking one link will take you an entirely different feeling architecture, like parts have been built on top without thinking about what the impacts could be. It's probably been a house of cards for a long and it's now fallen down.

saurik 13y ago

Interesting; the http://devimages.apple.com/ http://devimages.apple.com/ URL seems to return the underlying contents of the file, bypassing the SSI (I'm so happy they use SSI... seriously: I love SSI ;P). You can then see the raw  directives, and pull the individual parts. (It isn't quite then fair to say that it is a "broken bit of their template system"; it is more that it is a poor way to setup a static large-file caching endpoint, and may itself lead to a security vulnerability. To be clear: that's probably worse ;P.)

nwh 13y ago
Ah, I didn't recognise Server Side Includes, I've never used them before. There's a lot more weird around apple.com if you go looking. I'd love to hear the story behind their MX/SMTP hostnames (thousands of them) getting called "badger" and "msbadger".
- deleted 13y ago
  [deleted]
DoubleMalt 13y ago
Careful. You might have committed a felony by accessing a page with SSIs. It was definitely not intended to be accessed under that URL by you ;)
- marshray 13y ago
  That's no winking matter.
legutierr 13y ago
Might I ask what vulnerabilities exist for SSI, and why you describe SSI as a "poor way to setup a static large-file caching endpoint"? I haven't used SSI before, but after researching various technologies, it looks like a solution for my own caching use-case, and I am planning on implementing it. I realize that it's a rather old technology, but as long as there's nothing wrong with it, I think it could work for me. Is there something wrong with it? The only thing that I have seen is that the "exec" command is dangerous, but NGINX doesn't seem to implement it.
- sergiosgc 13y ago
  If you are now sampling this kind of technology, have a look at ESI (edge side includes), and have a look at varnish cache. You'll be pleasantly surprised.
- saurik 13y ago
  I did not describe "SSI" as a "poor way to setup a static large-file caching endpoint"; that description was for the way they setup "devimages." to overlap with the content for "www.". Let's imagine that the code in question was not SSI, but instead PHP. You might have this running behind a copy of Apache and mod_php. Your website includes both code and images. This website runs like this, but maybe the way you configured Apache is too slow to handle downloading all the images: these large-file static assets need to be handled by a different system. So, you install a copy of nginx and point it at the same folder. You give this copy of nginx a new hostname, such as "devimages.example.com" as opposed to the Apache "www.example.com". Problem solved, right? No. Now if you go to http://devimages.example.com/index.php http://devimages.example.com/index.php you get access to the source code to the website (which is generally considered a big security failure). That seems to be what's happening here: the "devimages." hostname is configured to bypass the SSI parser, and you are getting to see how the page is composited. This could be a problem. However, it also might not be a problem: it largely depends, which is why I was just idly speculating; it certainly feels wrong, though. As for SSI? I was serious: I love SSI, all my stuff uses it.

filleokus 13y ago

I work at an Apple Premium Reseller, and when we order stuff from Apple we use http://reseller.apple.com http://reseller.apple.com (which now seems to redirect to https://ecommerce.apple.com/ https://ecommerce.apple.com/) and all those systems feels hacked together just as you're saying. The Reseller Store has been down the whole time the Developer Center has been down. Even more interesting perhaps, our company's "master apple id", which is used to place multi 100K USD orders from Apple, got some strange seemingly manual password reset from Apple about the same time as the Developer Portal went down. Apple's hardware service portal (http://gsx.apple.com http://gsx.apple.com) has also been even more flakey this week than normal. Something is broken over at Apple.

sudont 13y ago
"Won't fix, as designed." B2B web apps have been hacky since time immemorial. Where I work, a different engineering team wrote a CMS in a bunch of Java with somehow Perl CGI to handle all of the little hacks that couldn't be put in to the main code base. I found this out by fixing a hack named emailservice_bkp_new2.cgi which was a hard coded rss parser. Our other client relations applications are even worse, often in IIS 6.0, and dating from 2003. Simply put, there isn't much "business reason" behind fixing something that isn't broken, but very shoddy. The difference now is that "prosumer" application developers have enough economic weight to register as a force, but not enough to cause Apple to reconsider their entrenched developer relations.
tazjin 13y ago
Do you have access to the CRT? If yes, does that still work? It hinges on the same login infrastructure as reseller.apple.com AFAIK. I have nothing to do with Apple anymore so I can't check it myself :)

e3pi 13y ago

> My money is personally on massive, unrecoverable data loss. Do we read this, you have an AAPL short order in tomorrow's market open? I find no financial news on this.

nwh 13y ago
I own no stock in any company. I have a reasonably pessimistic view based solely on the way this is being handled, and the way in which their web applications behave.
nknighthb 13y ago
Through a combination of direct ownership and mutual funds, I own thousands of dollars of Apple stock, and would be materially harmed by a major drop in its price. I also own numerous Macs and iOS devices, and am indirectly harmed by Mac and iOS developers being unable to access code and information important to their work. The company I work for may also be harmed by this, as we engage in some iOS development (though I'm not directly involved in that these days). I find the explanation of massive, unrecoverable data loss plausible, and your insinuation inane.

straight_talk_2 13y ago

Unrecoverable? I'd bet NSA has a complete backup ;-)

yapcguy 13y ago

Maybe it's worse? Maybe it's related to an earlier incident? From Feb this year: "Earlier today it was reported that Apple’s computers had been compromised by a zero-day exploit in Java. Apple quickly released an update to patch the flaw for all Macs, but not before some of its own employees had been hacked.... A site called iPhoneDevSDK has been revealed as the means by which a dangerous exploit was injected via a Java plugin." http://www.cultofmac.com/216618/this-iphone-developer-forum-is-responsible-for-hacking-apple-employees-macs/ http://www.cultofmac.com/216618/this-iphone-developer-forum-...

ChuckMcM 13y ago

Well if this is the case then there is some money to be made shorting the stock on Monday :-) I really like some of Apple's products (my Macbook pro is my favorite laptop of all time so far) but sometimes, like dealing with them as a guy who buys Macs for everyone who works for me, I think "Man they guys have a ways to go when it comes to this enterprise service stuff." I hope that what ever it is, it doesn't put too big a crimp in them.

ychw 13y ago

I think so too. If you look back a couple of years, this kind of things (even if not as severe) happened several times. I actually blogged about it (when their DNS was messed up): http://www.clingmarks.com/apple-please-be-serious-about-itc/851 http://www.clingmarks.com/apple-please-be-serious-about-itc/... http://www.clingmarks.com/ios-vs-android-mobile-dev-platform-heads-up/601 http://www.clingmarks.com/ios-vs-android-mobile-dev-platform...

tlogan 13y ago

I don't believe they have massive, unrecoverable data loss. Think about that: If they are so retarded that they dont have backup then Apple is really doomed. Really. But I think they have good IT... I think they detected serious security breach (which might be related to some incompetence but security is very very hard).