16 ms·
iOS 6.0 Causes CDN Overages
- deleted 14y ago[deleted]
- chrisrhoden 14y agoAuthor of the post here, happy to answer any questions. I'm pretty confident in our findings here, and some of the behavior is just bizarre.
- sigzero 14y agoDoes 6.0.1 fix it?
- jonny_eh 14y agoThe article says that the problem appears to be fixed in 6.0.1, at least for wifi.
- kookster 14y agoWe have some ideas on how to test 3G with 6.0.1, but haven't proven it one way or another yet.
- binarycrusader 14y agoFrom the fine article: "We have been unable to reproduce the issue using iOS 5 or using iOS 6.0.1". So the problem's already been fixed as far as they can tell.
- kookster 14y agoTrue. The problem is that we are seeing > 20% of users still on iOS 6.0.0, and still chewing up bandwidth. Upgrade, please upgrade.
- dsl 14y agoI saw similar behavior when I was working on the backend for an IOS app. We were able to track it down to a retry mechanism for streaming content when the server was responding too slowly. Maybe you are running into something similar.
- owenfi 14y agoI'm seeing this line in my log as well, this person downloaded about 750MB and they don't seem to be on iOS 6.0. Reqs User Agent 5,378 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14
- Mizza 14y agoOuch, elsewhere on the internet it seems like people are actually getting hit by this. Hopefully damage won't be too bad. Good research, PRX!
- kannan 14y agoNow that explains the sudden spurt in my 3G data usage in the last 3 weeks. This forced me to switch off my 3G :(. Hope Apple releases a fix soon
- farski 14y agoWe (PRX) haven't seen any evidence to show that it is still an issue in iOS 6.0.1, so you should be all set if you've upgraded, or an upgrade should fix it. If you are upgrade already and seeing overuse, please comment.
- kannan 14y agoI was (and still) using iOS 6.0.1 when I started seeing the surge in data usage
- dageshi 14y agoIs it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago. Perhaps I'm just having selective memory?
- hesdeadjim 14y agoI've had the exact same feeling. It seems like their quality control has gotten worse over the last year or so.
- publicfig 14y agoI think Apple has always had the tendency to make dumb mistakes, but I will say that I think reporting on it has changed. Remember all of the jokes about "It's not a bug, it's a feature" back when Jobs was there? Apple is by no means immune from mistakes and I don't think they ever were.
- nikatwork 14y agoExactly. The reality distortion field has collapsed and everyone is less willing to forgive mistakes. I mean come on, remember the MobileMe launch failure? That said, the famous Apple attention to detail does seem to be slipping, I sincerely hope this is just a transitional period and they tighten their game back up.
- publicfig 14y agoPeople also seem to forget how much of Apple's history was spent as a nearly faling tech company. Each little mistake is being reported to the point of saturation and while I think they're worth mentioning, every article seems to be a "beginning of the end" piece. As companies expand and begin to offer more and more products and services, it makes sense that their amount of bugs and mistakes would grow. When looking at gross amount of bugs and mistakes, you have to look at growth in offerings before major concern.
- danso 14y agoI dunno, how crummy the App Store has become is a perfect example of this. The App Store is one prominent way they make loads of money and build loyalty to the ecosystem, and yet it is a shitshow on IOS6...the responsiveness is as bad as I've seen on non-iOS software, but for me, the two really inexplicable failures are: 1. the use of modals to show individual apps on the iPad. So if you get to an app page and accidentally touch outside the modal, the app page disappears and you have to go back to Safari to find the link that brought you there. 2. On my brand new iPod, the Genius View has never NOt crashed. I know Genius (and most of Apple's search services) are poor to begin with, but it's a prominent button on the nav. How does it just not work on a brand new device? 3. I'm also guessing the lack of a light sensor on the new iPod was the result of a screwup in the dev cycle...I would've never guessed that its absence would be such a deficiency but when you can't see a single thing on the iPod merely because you stepped from indoors to outdoors...to the point where you have to go back inside to adjust the brightness...that's just infuriating. I can't remember the last time I've been infuriated by an Apple device...I feel infuriated all the time with my expensive Sony camera and how its video button is right where you can accidentally bump into it when shooting a pic...which prevents you from shooting a photo. The last few Apple devices I've had (two laptops, 2 phones, 2 iPads, countless iPods) have always been satisfactory in terms of polish...not iOS6 nor the new iPod touch
- kemiller 14y agoThis is exactly what I saw with the Podcast app last week. Used up 2GB in about 6 hours and put me into overage. I bet the class action lawyers are already circling.
- owenfi 14y agoOur very small outfit at verbmill.net is seeing similar signs. For instance, 700MB+ downloaded by a single client when an individual podcast is a maximum of ~50MB. Checking my logs now to see if I can narrow down device/version number. All of our files are hosted on S3.
- owenfi 14y agoOminous example. Reqs Total Avg B/s Client 666 2G 3M 127K mobile-198-xxx-xxx-088.mycingular.net. 666 AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us) I hope this user is on an unlimited plan :(
- fruchtose 14y agoWow, that is truly unfortunate. How long did it take for that 2GB to be chewed up?
- noinput 14y agoSame here, noticed a huge spike in bandwidth for my startup's shows hosted on CloudFront. 206 responses are flooding my logs and I moved everything off of CloudFront for my users to triage the cost. This breakdown explains it all: http://cl.ly/image/2H3u3u2O333g http://cl.ly/image/2H3u3u2O333g Talk about a pain, not to mention the 10x bill from Amazon that came as a surprise.
- kookster 14y agoExactly - we see this causing a crazy number of 206 calls.
- donavanm 14y ago"I moved everything off of CloudFront for my users to triage the cost." Can you explain more? Was there some cloudfront behavior exacerbating the issue? AFAIK this is just a client (mis)using range requests and CloudFront replying as requested.
- noinput 14y agoYou are correct and I should have wrote that note differently. I moved some high traffic files to some endpoints I have where bandwidth isn't a concern to change the formula a bit (primary) and the rest are back to using basic S3 for now. The bandwidth is still being consumed, however now it's easier on the wallet. After looking more closely at the differences for pricing between S3/CloudFront I realize it's appears cheaper to be on CloudFront. I assumed wrong that CloudFront data-out was billed in addition to S3 because of the way it's notated on the usage page ("AWS Data Transfer (excluding Amazon CloudFront)"). To prove it I ran two reports on CF (left) and S3 (right): http://cl.ly/image/0q0y3u2X0g1a http://cl.ly/image/0q0y3u2X0g1a and you can tell where I made the change, as well data is only counted on the service in question and not on both. Can anyone else confirm the above?
- donavanm 14y agoYeah, that's a bit confusing. Using Cloudfront with an S3 bucket is double billed, the first time. On a cache miss Cloudfront pulls the object from S3 and serves it to the client. S3 bills the regular data transfer out. Cloudfront then bills the regular data out to client rate. On subsequent requests, cloudfront cache hits, you're only billed by Cloudfront. Cloudfront request + byte rates are cheaper than S3 in Us-east-1, IIRC. So on popular or high ttl objects it's cheaper to serve through Cloudfront. On low ttl or low rps, like a few requests per day, it's cheaper to use standalone s3. The same origin + CDN vs CDN Hit math applies to EC2 as well. I do wish the billing was clearer in these scenarios.
- blinkingled 14y agoWonder if the CDNs don't have any safeguards in place to avoid single client downloading same resource multiple times?
- chc 14y agoThat doesn't seem practical. How would they know if it's the same client? Just watching for multiple requests from the same IP will result in a lot false positives (e.g. most college students will be SOL). And what would they do if they could detect it but the client didn't realize it already had the file? Just drop the request?
- blinkingled 14y agoLoad balancers can insert cookies to identify clients uniquely. (That solves the proxy/single-ip-multiple-clients problem - multiple clients from one proxy IP can still be uniquely tracked this way.) The newer load balancers (F5/iRules for e.g.) are also capable of doing per resource, per client throttling using the inserted cookies or session table.
- donavanm 14y agoServer side state per client is insane. Pushing it client side would be what, a kb? Wih an average object size of 50kb that's 2%. I doubt CDN customers would be willing to take a 2% cost hit for this feature. Now, assume we solve all that. The state is pushed on the client. The performance hit is so small as to be free. And the CDN eats the dev and opex. So what does the CDN do when "bad" range requests are detected? Throw a 400? What customer is willing to break all IOS 6 clients? Also, iRules. Serious LOLs.
- blinkingled 14y agoFor starters, average podcast download is 40Mb or so. More than 50Kb in any case. Next maintaining state per client isn't anything new - it might not be ideal for CDNs but many LBs keep state for things such as maintaining session persistence etc. If there is a potential for clients intentionally or unintentionally jacking up your CDN bill - I am sure somebody has a solution to prevent it - especially since money is at stake. You laughed at iRules - I got that part, but what about iRules doing HTTP Request throttling? Far as I know it can do that based on source ip and port, uri etc. which should work for this kind of scenario with some modifications.
- meej 14y agoHmmm, I had to turn off the LTE on my iPad two days ago because I hit my limit 10 days prior to my monthly renewal. I upgraded to 6.0.1 last week, but I wonder if this bug explains how I managed to chew through my data plan faster than ever before.
- riobard 14y agoHmm, so this explained my 3G data overage last month. I have 6GB/month of data, and usually I'd be using around only 700MB. But last month my iPhone 4S (running iOS 6.0) consumed nearly 8GB in total, resulted in quite some overage fee. Should I ask Apple for a refund for this? :|
- pk2200 14y agoSame problem here. My monthly data usage increased by 10x after I upgraded to iOS 6.0. It's annoying that there's (apparently?) no mention of this issue in the 6.0.1 release notes.
- frogpelt 14y agoMaybe Apple was trying to stimulate the economy.
- mintplant 14y agoTry your data carrier, rather than Apple.
- gigamike 14y agoI had a similar experience. I normally consume 750-800 GB a month and on my first full month of iOS 6, Verizon said I consumed 8.89 GB. Luckily i'm grandfathered into an unlimited plan.
- malyk 14y agoLast month I moved and didn't have an ISP at home for the first 3 weeks or so. I set up tethering on my phone and a 5GB data plan. Yesterday I got a notice that I was over my data limit. I haven't done a whole lot of web surfing in the new house, but I have listened to a ton of podcasts using the Podcast app which I have never done before. So if I just blew through 5 GB of data in 2ish weeks and the only thing different was using the podcasts app... (I did watch 3 episodes of walking dead on Netflix which obviously contributed to the data use, but a 45 minute episode can't be that big)
- burriko 14y ago
- bquarant 14y agoNoAgendaShow.com has been discussing this since iOS 6 launched. Don't upgrade.
- untog 14y agoThis is where Apple's closed nature is very annoying. Assuming the article is correct, it was bug introduced in 6.0, and fixed in 6.0.1. So Apple knew about it, investigated it and published a fix. Yet they didn't tell anyone about it. Or the crippling effect this bug could have on their data allowances. Why not? Are they really that terrified of admitting to making a mistake?
- rachelbythebay 14y agoThis same behavior has been happening to me since before iOS 6 came out. I can show the same "206 storm" happening as far back as August of last year with iOS 4.3.5.
- randall 14y agoI used to work at http://castfire.com/ http://castfire.com/, and we've seen that similar issue from a LOT of iOS clients. Stats become increasingly difficult, because we saw one unique request bounce with upwards of 50+ 206s for similar / overlapping media... and then it bounced through a cell provider, to a WiFi network, and back, etc. (we identified by redirecting all requests through a complex process of assigning an ID to them) Hoping this is fixed. Podcast metrics are tricky, and this has been exacerbating the issue.
- chrisrhoden 14y agoThe playback stuff definitely makes use of many range requests, but what we found was that they were being used in a way that vaguely made sense in other versions (i.e. downloading a few hundred kb to continue buffering, pausing the download, and picking up again). With 6.0.0, we're seeing actual storms of these requests happening many times per second.
- michael_miller 14y agoTo play devil's advocate, why would it matter? Is there any fix app developers could implement, short of blocking requests from iOS 6 users?
- 14y ago
- fernandezpablo 14y agoMakes you think twice about those graphics where iOS devices take a big piece of the pie of the "internet usage"
- eridius 14y agoNo it doesn't. Those aren't tracking bytes streaming over the internet, but rather web browsers visiting pages.
- fernandezpablo 14y agoYes it does.
- smackfu 14y agoEven ignoring the bytes, it looks like one client may show up in the web logs as downloading the file a dozen times. If you aren't careful about cleaning up the stats, that may overrepresent iOS.
- robomartin 14y agoI have never considered Apple to be a solid software company. They are a marketing company in the tech sector. If they made washing machines they'd be sexy as hell. In my years of exposure to Apple tech I have always been in awe of their industrial design, mechanical design and manufacturing prowess. Their software, however, often leaves much to be desired. I say this both as a developer and a user. From OSX to iOS there are amazing patches of software manure in plain sight. And it isn't getting any better.
- addlepate 14y agoOK. I bet you use a WebKit browser.
- zizee 14y agoaddlepate wrote 37 minutes ago | link [dead] OK. I bet you use a WebKit browser. addlepate: you were hell-banned 100 days ago (unfairly in my mind).
- pretoriusB 14y agoHere's an easy clue: if your comments never get above or below 1 point for days, that means you are hell-banned.
- mkr-hn 14y agoThe only thing you could infer from that is that you're not a bad enough poster to warrant downvotes.
- pretoriusB 14y agoNot really. It's a sure sign that you are hell-banned. I mean, we can check statistically, and I bet we would find that nobody stays at 1 point forever in all his comments. (And it's also upvotes you won't be getting).
- 14y ago
- kevingadd 14y agoThe iOS 6.0 podcast bug apparently caused huge overages for GiantBomb as well. They're discussing it right now on one of their live streams; it apparently was so bad that their corporate/finance people are upset with them.
- Ensorceled 14y agoThis is the reason Forstall has left the building, iOS 6 is a disaster: * Maps was not ready for prime time (so bad it required a public apology) * Podcast was not ready for prime time. * Passbook was not ready for prime time. * App store is a step backwards. * Battery and WIFI issues. * Soooo many bugs. There are usually problems but this is the first time I've seen so much utter crap shipped including tons of minor bugs and issues (well Mobile Me was pretty bad :-) I'm using Audible, Downcast and the built in Music app where I used to just use the Music app. iOS 6 was a huge step backwards for me. Rather than waiting until it's ready (think copy/paste on iOS) Forstall was responsible for shipping a mess. Hopefully this will fix itself with new leadership.
- gurkendoktor 14y agoHow do you know that Forstall was the person advocating a yearly OS upgrade cycle? It seems just as likely to me that this rigid schedule was forced onto him and he couldn't deliver. Things like OS X Lion (a huge change under the hood) and the Maps places database just aren't things that can be built to a deadline.
- Ensorceled 14y agoWell, a mess was shipped and he was the guy in charge. So I'm assuming he was the guy responsible. If a mess really was forced upon him he should have resigned instead of sticking around to ship the mess. I have in that exact same situation, "This will not be associated with me, I'm out"
- gurkendoktor 14y agoNow you are talking about saving face, not about saving iOS. Maybe he was the best guy for the job and he still couldn't do it.
- rachelbythebay 14y agoThis just seems to be how they handle audio media. It's not even anything specific to podcasts. My own scanner stuff is just a web page, and it gets hit by this from time to time. Each call is a single file, it's static, and there's nothing funny going on with it. It looks like the playback stuff is jumping around in some attempt to figure out exactly what it is first. It seems like it would be far better if they always downloaded at least some reasonable quantity and then decided to bail out later if it seemed overly huge, but this is where we are. To me, it feels like someone who's trying to be fancy and do things like seek() in a stream. Just because HTTP has progressed to the point where you can effectively do that with Range: requests doesn't mean it's a good idea. Offering seek() type behavior for a HTTP resource seems like a great way to get people to use it and not even realize what sort of messes they may be creating. Leaky abstractions, eh? Here's just one example from last week. Note: the file is 7001 bytes. There's no reason for this insanity, and yet here it is. x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-" x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
- erickhill 14y agoI believe this is directly related to phone data usage overages as well. Having used an iPhone 4 on ATT for 2 years, I could almost guess my data usage perfectly each month. By the 2nd day of using my new Verizon iP5, I was receiving warning texts from Verizon that I was about to exceed my data limits. Thinking it was me, I increased my plan. That only stopped the bleeding for another few days. Oh, and I installed the "fix" that was sent out to supposedly repair the known issue. (I was always on Wi-Fi networks but still being charged for data.) I recently called Verizon to dispute the data. They quickly erased all of my data overages. Snap, like that. The man I spoke with called a technician on my behalf. The people who can erase overages do not officially have this issue in their database as something to be aware of. But the technician he spoke with said there was absolutely a problem, and both Verizon and Apple knew about it and were working on it. Then, I was given a "diamond ticket" to absolve me of any future overages until a fix is found.
- andrewcastmate 14y agoI don't have anything wildly new to add, but I'm part of a podcast hosting site (http://Castmate.fm http://Castmate.fm) and we've seen this bigtime, it's crazy. We host some high-traffic podcasts with a huge range of listeners, and I've gone through the logs and regularily see stuff like NUMEROUS listeners making 4000+ http requests in 30 minutes, all the 206 calls everyone here is reporting, it's just nuts, so bizarre. I feel terrible for all the people getting hit with overages, it's so gross. I did some gorilla math the other day and feel pretty confident that this has boosted our network usage by ~25%, I should have narrowed it down to just the affected iOS 6.0 devices, must be astronomical.
- kokey 14y agoI am anxiously awaiting an update to the Podcast app which fixes the issues that I am having, but perhaps I'm too optimistic. I listen to some podcasts from the BBC and the app doesn't recognise that a new daily show with the same name is new, the only way around it I have found is to delete my subscription and manually find the podcast every morning. Apart from that the thing crashes and runs very slowly at times.