32 ms·
I don't think anyone on hackernews would be surprised by this. The scale they have to operate at is absolutely insane. To scale flawlessly on day 1 is a big ask
by MentallyRetired 7y ago
I don't think anyone on hackernews would be surprised by this. The scale they have to operate at is absolutely insane. To scale flawlessly on day 1 is a big ask of any development team.
- deleted 7y ago[deleted]
- cookiecaper 7y agoIndeed. Last night I was telling my wife that it'll be real interesting to see how well the launch goes and that I'd expect them to have continuous issues for the first 3-6 months (which may be Netflix's only prayer). Video is hard to get right, with quirks across many combinations of platform/device/OS/browser version/video card/playback resolution/format features, and it consumes around 90% of total internet traffic. Netflix has spent the last decade building and tuning a truly world-class content delivery system (big props to all the Netflix HN readers). There's no way Disney would've been able to match that prior to go-live. The logical conclusion, of course, is that Disney should've just bought Netflix, and it wouldn't be too surprising to see that happen now that their internal attempt has proven that pixie dust doesn't go quite far enough to magically remediate the limitations of physics. Gotta get Netflix-esque content delivery appliances colocated alongside ISP hardware for that type of thing.
- EpicEng 7y ago>I'd expect them to have continuous issues for the first 3-6 months I don't think it will be anywhere close to that long. Disney already has plenty of experience in streaming video (BAMtech / Hulu).
- geerlingguy 7y agoThe sad thing is they're having issues with loading login pages, authenticating via API, app API backend routes... pretty much every layer of their app stack and that's _before_ any video can be streamed to end users. I hoped they would've had the first few layers pretty scalable and load tested, but either they had orders of magnitude more load than they ever planned for, or they didn't actually load test to millions of users (more likely IMO).
- buckminster 7y agoIt makes some sense if it's the user onboarding that's failing. That's the one part that should see much less traffic in normal operation.
- awill 7y agoThat's actually the hard part. Video streaming is easy, and done with CDNs.
- Slartie 7y agoSure, handing over giant checks to Akamai is easy, but it becomes a lot harder if you do the CDN part yourself. Like all the really big streaming video providers do, because otherwise they wouldn't be competitive at scale.
- Mindwipe 7y agoWell, it's not "easy", at Netflix scale. Lots of Netflix traffic is done via local caching servers at ISPs as CDN delivery isn't reliable enough. Minimising playback latency is really, really hard and NF do very well here. Robust geofiltering in those scenarios is hard if you're trying to check the video essence and not just the license. Having said that, there are a lot of reasonably mature solutions in the market to address many of these points, and it doesn't appear that was the issue yesterday.
- durden0 7y agoThis makes more sense to me then all the talk about CDN's. CDN's don't solve all problems.
- sassifrass 7y agoIt’s true that getting video “right” can be extremely challenging, but it’s worth noting that Disney has quite significant previous experience here, including at massive scale. Disney+ operates on top of an expanded MLBAM (BAMTech Media [1]) platform, which has been powering all of ESPNs OTT, Playstatiom Vue, and ran HBO’s OTT offering for a while [2] (Game of Thrones). Disney also opted only to launch in the US, Canada and Netherlands today. So overall, this is still a bit of a botched launch, and the expectation absolutely should have been that they would handle the volume. [1] https://en.wikipedia.org/wiki/BAMTech https://en.wikipedia.org/wiki/BAMTech [2] https://www.fastcompany.com/90373706/hbos-tech-is-prepping-att-for-the-streaming-wars https://www.fastcompany.com/90373706/hbos-tech-is-prepping-a...
- selebrazin 7y agoA small correction: Disney+ launched in the Netherlands in September 2019 [1] which was treated as a public test. [1] https://www.theverge.com/2019/9/12/20862167/free-disney-plus-hands-on-pilot-marvel https://www.theverge.com/2019/9/12/20862167/free-disney-plus...
- JaRail 7y agoIt's not the worst idea. If they had performance problems in multiple regions, their engineers would be chasing peak hours around the clock. If they focus on NA first, they can at least get some sleep at the end of the day.
- bluedevil2k 7y agoBuying Netflix? I think Netflix is going to start a steady decline as the content owners continually pull their stuff from Netflix for their own streaming services. The Netflix original stuff, let's be honest, is hit or miss, with most of the movies being a hard miss.
- baddox 7y agoI don't have much of a problem with Netflix content, but it's the price that just doesn't make sense. They charge extra for UHD, and that premium plan costs more than HBO (which is probably still the king of content quality, though with a crappy 1080p stream) and over twice as much as Disney+ and Apple TV+ (which don't charge extra for UHD).
- nickthegreek 7y agoThe plan that offers UHD also offer multiple screens. It it is a legitimate way for customers to pay to stream multiple things at the same time. So while it costs around the same as the HBO plan, it offers something that some other services do no. Apple TV+ allows family sharing, but I dont know about Disney+.
- baddox 7y agoFYI, the "Standard" Netflix plan offers 2 simultaneous streams in HD for $12.99 per month in the US. The "Premium" plan offers 4 simultaneous streams in UHD for $15.99 per month. But anyway, I only have one screen, so it's still the case that Netflix charges considerably more just to receive UHD content. As price discrimination goes, that's a pretty scummy way to do it. They've shot and produced this content in UHD, yet they give a much lower quality version to their base customers. I know they have every right to do that, and it might make business sense for them (since people with UHD TVs probably have more disposable income), but it has always felt scummy to me.
- bryanlarsen 7y agoNetflix's costs are higher for UHD streaming: storage, bandwidth, production, et cetera. What's scummy about it?
- zzzcpan 7y ago> which may be Netflix's only prayer Isn't Disney projecting to get like 10-20% of subscribers compared to that of Netflix in five years? Why are people talking like Disney streaming is such a big thing?
- dewey 7y agoBecause Disney has a huge back catalog that they can use in the coming years (https://en.wikipedia.org/wiki/Disney_Vault https://en.wikipedia.org/wiki/Disney_Vault) would be one reason for that (and a lot of nostalgia).
- endorphone 7y agoA bunch of old movies that everyone has seen a hundred times, probably have on DVD, Blu-ray, and in all likelyhood mp4 form. Eh. The Disney "Vault" thing seems like it's a gimmick that is pretty threadbare now.
- dewey 7y agoIt’s not just about the existing movies. It’s about owning the franchises and licenses to them.
- wolco 7y agoWith 10% of netflix customers volume but how many will drop netflix of that number for disney? Disney will do fine so will netflix. Plenty of room for both.
- cookiecaper 7y agoDisney has [more] skin in the game now and will always give their own service highly preferential treatment. This isn't just about Mickey Mouse -- Disney owns many other major content producers. Personally, I've been hearing people complain about Disney-owned content disappearing from Netflix for more than a year already. I'm sure they'll continue to license something to third-party streaming platforms, but they'll do their darnedest to make sure that if consumers are only going to choose one streaming service, it'll be one they control and operate. It's nothing against Netflix itself -- they're one of the tech companies I admire the most. But they're stuck depending on relatively-short-term license agreements to offer most of the content that people are willing to pay for. Whether or not the consumer is willing to pay for multiple streaming services, the media companies are going to be much less willing to make the content available outside of their own little streamer. This means rough seas ahead for Netflix.
- mantoto 7y agoWhy would that be hard? Distributing a limited amount of static content around the world sounds reasonable easy to me.
- jonwachob91 7y ago>>> The logical conclusion, of course, is that Disney should've just bought Netflix Why buy Netflix when you could get a similarly qualified team and infrastructure by buying Hulu? Oh wait... Disney already owns Hulu... Whoops. What about ESPN. ESPN has a really good streaming product. Oh wait... Disney already owns that one too... Disney didn't need to buy Netflix, they just needed to use the resources they already owned!
- a-wu 7y agoESPN's streaming is one of the worst ones I've ever used. It is constantly buffering on my 275 Mbps downlink and every other time it has issues with authenticating to my provider.
- jonwachob91 7y agoWow. That is interesting. I haven't used ESPN's streaming platform in about a year (since they did that switch and stopped including streaming in cable packages). But I used to stream European Soccer games on my second monitor at work virtually every day. My work internet was maybe 75 mbps and it worked fine.
- casefields 7y agoIt’s gotta be location dependent. I’m in Los Angeles and have never had a single issue with ESPN, even going back to the ESPN3/WatchESPN days.
- a-wu 7y agoI actually used to have better service when WatchESPN was standalone. When they started integrating into the main ESPN site (and new player) it started declining for me.
- mikelbring 7y agoI have noticed ESPN works way better on my Roku than my Xbox. A lot of the apps do. I use to think it was just ESPN but started to notice this.
- btables 7y agoDisney bought one of the most successful video streaming companies in the world to drive Disney+ called Bamtech, they didn't start from scratch at all - https://www.forbes.com/sites/maurybrown/2017/08/08/disney-accelerates-purchase-of-bamtech-espn-disney-will-see-new-digital-streaming-media-apps/#7d347ddc4a5d https://www.forbes.com/sites/maurybrown/2017/08/08/disney-ac...
- scarface74 7y agoThe logical conclusion, of course, is that Disney should've just bought Netflix They bought BAMTech years ago. The company that built MLB@Bat and saved HBO Go/Now when their in house tech couldn’t cut it.
- dylanjha 7y ago> Video is hard to get right, with quirks across many combinations of platform/device/OS/browser version/video card/playback resolution/format features Video is really hard to get right, a Hollywood-style big bang launch on day 1 is difficult to pull off for any software, let alone video. I feel for the engineers working there, I wonder what the meetings were like last week, did the engineers know they were not prepared for this?
- JPKab 7y agoCame home today, and my wife had Disney + set up and running. Son was watching the awful Phantom Menace movie, and unfortunately it was working like a charm. Anecdotal, but let's be honest:no matter the problems, Disney can hire talent it needs. Question is whether the corporate culture there can not end up being shitty environment for engineers.
- swasheck 7y agoBroad device support is also a large factor that favors Netflix. We have children who range from 3-14 and Disney has something for all of them - except support for the devices we have. Our TV is an older (NetCast) LG, we “only” have a Wii-U, and we don’t have old computer hardware just laying around. Netflix, Prime, Hulu, Google Play, YouTube support our TV. Not Disney+. Looks like we’re gonna do our one week and then wait for a while.
- dehrmann 7y ago> The logical conclusion, of course, is that Disney should've just bought Netflix If we're playing this game, Apple should buy Disney. YTD, Apple's market cap has grown by more than the entire market cap of Disney.
- Mindwipe 7y agoThe actual streaming delivery seemed to go okay yesterday, they have enough CDN peering. It was the authentication services that had issues. Those are fairly common day one problems for lots of online services that are fixable. I'm not sure some day one capacity issues would justify the enormous expensive of buying NF tbh.
- riazrizvi 7y agoIt breaks when you hit the main CTA button to start a free trial now. I don’t understand how a company with billions of dollars in resources can fail to solve such a well trodden problem. Even if there were a billion users on day 1. I wonder if the problem is due to an abdication of parts of the engineering problem, an over reliance on 3rd party services, that meant that no-one in the engineering team was able to study the whole engineering problem from first principles, which would result in inadequate fail overs.
- berdon 7y ago"well trodden problem" does not make it any easier. Scaling is hard. For everyone. Testing that scaling works is harder.
- sterileopinions 7y ago>"well trodden problem" does not make it any easier. Yes it does.
- OnlineGladiator 7y agoI have to agree with sterileopinions here. The fact that it's been done multiple times before means you know a lot about how hard it will be, the necessary resources, and can learn from past mistakes. That doesn't make it easy. But it definitely makes it easier. To say otherwise is to argue that experience isn't useful.
- sterileopinions 7y agoThank you.
- pilsetnieks 7y ago> The fact that it's been done multiple times before means you know a lot about how hard it will be, the necessary resources, and can learn from past mistakes. "All happy families are alike; each unhappy family is unhappy in its own way." -- Leo Tolstoy, Anna Karenina
- dwoozle 7y agoBut these guys knew the kind of volume they would have on Day 1, and have the resources to get it right, and the brand value to protect. It’s not like a startup that grows like crazy.
- deleted 7y ago[deleted]
- testplzignore 7y agoDisney+ is arguably the largest software release of all time in terms of scale. My estimate is that they will be handling 1-2% of all internet traffic by the end of the week. I can't think of a launch of this scale that has gone smoothly. Modern console releases, AAA games with online multiplayer, Pokemon Go, healthcare.gov... all cases where organizations literally have billions in cash. It does not matter how much money you have or how great your engineering team is - you can't go scale 0 to 100 on launch day.
- ThePadawan 7y agoOK - so if it is so obvious that you can't scale from 0 to 100 on launch day, why not anticipate that? Ticket sales platforms for major sports events and concerts don't try to do it - they plan for it by having customers queue. Did Disney not attempt this?
- ssully 7y agoI've never handled a project like this, but I feel like the goal should be to meet what average demand will be once the high demand of the first few days goes down. No one is going to remember that they couldn't login to Disney+ on the first day as long as the experience on the second and going forward is strong.
- gamblor956 7y agoOK - so if it is so obvious that you can't scale from 0 to 100 on launch day, why not anticipate that? I imagine the gigantic Arctic cold front passing over the East Coast right now created a much larger demand than they were expecting on a non-holiday Tuesday morning while school is (supposed to be) in session.
- tootie 7y agoI logged in early this morning and nothing was working but it was up an running within maybe an hour. It's almost more impressive that they had an unintended service interruption so quickly and recovered right away.
- dangerboysteve 7y agoI signed up an hour ago for the trial. There were some minor hiccups but I was able to download some stuff.
- jonplackett 7y agoThis would seem a fair assessment if it were not for my experience using Disney Life, their old streaming service. It has been out for years and is generally terrible. - App is incredibly slow and takes ages to load. Worst on Kindle Fire and older devices (Netflix/Prime video work fine) - sometimes a download film will just stop working and crash the device when you try to open it. Only way to fix is to delete the whole app and reinstall. Deleting only the download does not work. - downloads are huge regardless of device/connection, appears to be no intelligent bandwidth usage. - logs you out on every single update of the app or operating system (pain on Apple TV where it auto updates every minor version) - logging back in requires you to go to their website rather than be able to do it on the device. - never remembers where you are in a film properly - interface on Apple TV is impossible to use (partly Apple TV controllers fault though) Yet I still subscribe, because the content itself is great.
- draw_down 7y agoRight, which is why I was surprised they went with a big-bang launch, as opposed to bringing smaller cohorts of users online. However, from what I read, they tested it in smaller countries first, so maybe that wouldn't have helped much.
- lmilcin 7y agoThis could have been avoided if the service was introduced slowly. Big launch days planned months in advance don't really play well with software development reality. It is difficult to built software at scale without having all those users to provide for the load. They could start with small number of people and get more by invitation. This could have been done before all scaling effort.
- ragebol 7y agoThey launched in the Netherlands some 2 month ago, I've had no issues.
- markdown 7y agoMeh, this was easily prevented simply by not promoting it so much. They should have done a staggered rollout country by country starting with the smaller countries and adding on a country or two every day.
- darkerside 7y agoThis is probably one of the big reasons for rolling out with a beta or invite only soft open. The ability to control growth while also building hype.
- ragebol 7y agoI've been using Disney+ since they pre-launced in the Netherlands for the last ca. 2 months IIRC, without any issue what so ever. But then again: NL is a small country.
- mcv 7y agoI've had issues getting it to Chromecast. Fortunately they released the app for Samsung TVs last week. That helps a lot. Now there's only the issue that that app was suddenly logged out for no clear reason. Let's hope that doesn't happen again.
- jamil7 7y agoNot surprising at all, what is surprising to me is that companies still insist on doing the "Big Reveal" style launches. Why not open it up gradually in phases via a beta program or something?
- Mindwipe 7y agoIt's hard in this case when the release schedule is so heavily based around expiring licensing agreements for content elsewhere tbf.
- mcv 7y agoThey did that to some extent. I've had Disney+ for about a month now. But going from a few people in a small country to a billion people all at once is still a massive change.