11 ms·
I love technical content like this. Not only is it incredibly interesting and informative, it also serves as a perfect counterpoint to the popular "why does Net
by paxys 4y ago
I love technical content like this. Not only is it incredibly interesting and informative, it also serves as a perfect counterpoint to the popular "why does Netflix need X thousand engineers, I could build it in a weekend" sentiment that is frequently brought up on forums like this one.
Building software and productionalizing/scaling it are two very different problems, and the latter is far more difficult. Running a successful company always requires an unlimited number of very smart people who are willing to get their hands dirty optimizing every aspect of the product and business. Too many people today think that programming starts and ends at pulling a dozen popular libraries and making some API calls.
- fennecfoxy 4y agoLmao. _But it does_. At least for the most part. It's such a previous generation thing to be angry at the way that modern development is done. Of _course_ dev is now web heavy and people are pulling in all sorts of libraries to make their lives easier. But pretending that those same devs wouldn't also be capable of development on lower/less abstracted levels (well sure, maybe not _all_ of them) is insulting.
- le-mark 4y ago> Too many people today think that programming starts and ends at pulling a dozen popular libraries and making some API calls. The needle keeps moving doesn’t it? A tremendous breadth of difficult problems can be effectively addressed by pulling together libraries and calling APIs today that weren’t possible before. Today’s hard problems are yesterday impossibilities. The challenge for those seeking to make an impact is to dream big enough.
- ezconnect 4y agoThe basic problem is the same, pushing the hardware to its limits.
- whatshisface 4y agoThe basic problem is delivering value to someone.
- ezconnect 4y agoProgrammers are not passionate to deliver value to someone, that's the businessman problem.
- bcrosby95 4y agoNot every programmer is passionate about the same thing. I got into this field because I love building things that make people's lives easier.
- echelon 4y agoSure. Anecdotal, but most of the people I've worked with as ICs couldn't give a damn about that. They want dollarydoos. One of the 10X-ers I know (they exist and are real), told me repeatedly how he'd much rather be doing his own thing. He hates the business needs. But income is important and that's why he's dedicated to doing it. I'm surprised at how focused and good he is given his disposition, and I want to hire him when I scale my business more. Drive and passion are sometimes just spontaneous. An old CEO of mine even quipped that we were not family and that we were there to do a job. All true. Most of the people doing that job were only there for the money. Most jobs that drive sales and revenue simply aren't fun or rewarding. There's lots of infrastructural glue and scaling. Tiring, boring, monotonous work. 24/7 oncall work. The money is good, though.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- mihaic 4y agoI don't think this actually answers why Netflix needs to many engineers. This seems like the sort of thing that one or two experienced engineers would spend a year refining, and it would turn out like this. This is the sort of impressive work that I've never seen scale.
- drewg123 4y agoAuthor here... Yes, most of this work was done by me, with help from a handful of us on the OCA kernel team at Netflix (and external FreeBSD developers), and our vendor partners (Mellanox/NVIDIA). With that said, we are standing on the shoulders of giants. There are tons of other optimizations not mentioned in this talk where removing any one of them could tank performance. I'm giving a talk about that at EuroBSDCon next month.
- LtdJorge 4y agoThank you for sharing!
- Sytten 4y agoI think a fair criticism would be how many engineers they have compared to their competitors. Disney+ is on a similar scale, can they do the same/similar job with less people? And considering netflix pays top of market, how much does Disney spends for their engineering effort to get their result. Would netflix benefit from just throwing more hardware at the problem vs paying more engineers 400-500k/y to optimize?
- entropie 4y agoI wasn't able to watch disney+ via chromecast for like a year in 4k. Stuttering every 10 seconds or so. I never had problems like this with netflix.
- criddell 4y agoI guess you weren't a Comcast customer in 2014 trying to watch Netflix and getting low quality, stuttering video. At the time lots of people tried to frame it as a net neutrality issue but in the end I think it was a peering dispute that involved a third party. https://www.wsj.com/articles/SB10001424052702304834704579401071892041790 https://www.wsj.com/articles/SB10001424052702304834704579401...
- loopercal 4y agoI think this just validates their points. Netflix has more engineers and 8 years of them building and fixing things, so they have fewer issues.
- paxys 4y agoDisney (the company) has 20x the number of employees as Netflix, and just 2x the market cap (in fact they were briefly worth the same last year), ~2x the revenue and 2/5 the net income. So Netflix is clearly doing something right.
- ziddoap 4y ago>Disney (the company) has 20x the number of employees Is that all of Disney or just Disney+? It doesn't seem like that would be a useful statistic if that includes completely unrelated positions (e.g. does that 20x statistic include Disney employees working at Disney Land/World serving up hotdogs? Because they probably don't contribute much to the streaming service)
- rvnx 4y agoI think nobody said Netflix' infrastructure can be built in a weekend. However, the scale doesn't matter that much after a certain point once the scaling "wall" has been pierced. If you are a biscuit factory that produces 100'000'000 biscuits per year or 500'000'000 biscuit per year then the gap between 100M and 500M isn't that impressive so much anymore as it's mostly about scaling existing processes. However, if you turn a 1'000 biscuit shop into a 1'000'000 biscuits company then it's very impressive.
- bmurphy1976 4y agoNonsense. It's still impressive. A 5x increase at that scale can be a phenomenal challenge. Where do you source the ingredients? Where do you build the factories (plural because at that scale you almost certainly have multiple locations in different geographic locales subject to different regulatory structures). Where do you hire the people? How do you manage it? What about the storage and shipping and maintenance of all the equipment and on and on? How much do you do in house how much do you outsource to partners? What happens when a partner goes belly up or can't meet your ever increasing needs? Your comment is a great example of what the OP pointed out.
- jon-wood 4y agoMy favourite example of these sort of extreme scaling issues are the fact that McDonald's apparently declined to sell products with blueberries in them because modelling showed they'd have to buy the world's entire supply of blueberries in order to do so.
- notamy 4y agoI thought this was hyperbolic, so I looked into it: > The menu team comes up with interesting ideas like including kale in salads. The procurement team and suppliers then try to get the menu team to understand the challenges. How do you bring kale to 14,000 restaurants? As one example, when they introduced Blueberry Smoothies in the U.S., McDonald’s ended up consuming one third of the blueberry market overnight. https://www.forbes.com/sites/stevebanker/2015/10/14/mcdonalds-prioritizes-sustainable-beef/?sh=18078bf827b3 https://www.forbes.com/sites/stevebanker/2015/10/14/mcdonald... I couldn't find any other source to back it up, but still wow! That's an absurd number.
- Quarrelsome 4y ago> why does Netflix need X thousand engineers, I could build it in a weekend I would like to hope nobody asks that. Video is the one of the, if the not the hardest data plumbing use-case on the internet.
- dragontamer 4y agoI'd say realtime communications is harder. A lot of these tricks being discussed here cannot be applied to Skype calls.
- OJFord 4y agoSurely GP would agree, unless you mean even audio-only calls? Otherwise it's just an extra requirement(s) on top of 'video'.
- jon-wood 4y agoStreaming pre-recorded video and streaming realtime video are almost entirely different use cases. Pre-recorded video streaming is, under the hood, really just a high-volume variant of serving up static web pages. You have a few gigabytes of file to send from the server its stored on to the device that wants to play back the video. As this presentation demonstrates that isn't trivial at scale, but the core functionality of sending files over the internet is what it was designed to do from day one. Because you can generally download video across the internet faster than it can be played back its possible to build up a decent sized buffer which allows you to paper over temporary variance in network performance without the customer noticing. Realtime video streaming has two variants. One to many Twitch style video streaming is relatively simple, since you can encode video into files and upload them to a server for people watching to download those files. This is how HLS streaming works, and most of the techniques Netflix use to optimise video delivery can also be applied here at the cost of adding latency between the event being streamed and people consuming it. That latency will often sit at about 30 seconds, and people generally find that acceptable. Skype style realtime video streaming is much harder. You're taking video from one person's camera, and then sending it over the internet to one or more people's device. You can't do any sort of pre-processing on that, or stage the video on servers closer to the consuming users, because you have no way of generating that video until the point your users decide to start talking to each other. Because you can't pre-stage that video you need to be able to establish a network route between the people on a call, potentially in an environment where none of the participants have any open connection from the internet directly to the device they're streaming from. Slight fluctuations in network performance can potentially degrade video delivery to the point of it being unusable. The most common route to deal with that is systems that attempt to establish a direct connection (ideally over a local network) between participants, and if that doesn't work going via relay servers operated by the software provider. These servers provide a single point on the internet all parties can connect to, and then allow passing packets as if they were all on the same network.
- onlyrealcuzzo 4y agoTo be pedantic, scaling by itself isn't that difficult. Scaling cost-effectively is.
- eru 4y agoYes, and No. At some point, even scaling at all would be hard. (Just like sending a human to Alpha Centauri is hard, even if you had unlimited funds.)
- Dylan16807 4y agoLike it how? Accomplishing a grand feat is nearly the opposite of scaling. If Netflix built out more slower servers, that would be acceptable scaling. I don't see any plausible scenario where that becomes too difficult. Even if they had billions of subscribers.
- eru 4y agoThere's lots of scaling involved in grand feats. Eg enriching a tiny bit of uranium is 'relatively' easy. Enriching enough of the stuff to build an atomic bomb is a question of scaling. And, yes, in principle you can just use more and more centrifuges etc. Humanity has already built some small infrastructure off-earth, like the ISS or satellite networks. Also various unmanned probes that toiled away for years in space and on Mars. Building a habitat big enough to plausibly keep humans alive for the decades or more it would take to get to the Alpha Centauri would involve enormous scaling efforts. And that's not even touching on propulsion, yet. However, yes, grand feats require more than just scaling.
- Dylan16807 4y ago> Eg enriching a tiny bit of uranium is 'relatively' easy. Enriching enough of the stuff to build an atomic bomb is a question of scaling. And, yes, in principle you can just use more and more centrifuges etc. But we're not scaling for scaling's sake. We're scaling with the number of customers. If it was profitable to enrich some milligrams by hand, then you'd be able to get more scientists and engineers and better equipment and automate some parts and make even more profit. If you go the centrifuge route, it's not because you had to do the harder methods, it's because it's massively cheaper per gram of product. Such a scenario is very different from "we can't do it at all, and we need to build complex infrastructure just to make it possible".
- Aaronmacaron 4y agoI think the problem is that the "easy" parts of netflix such as the UI or the recommendation engine seem like they were hacked together over the weekend. Of course deploying and maintaining something of the scale of netflix is incredibly hard. But if they can afford thousands of engineers who optimize the performance why can't they hire a few UI/UX engineers to fix the godawful interface which is slightly different on every device? I think this is where this sentiment stems from.
- Shaanie 4y agoI'm surprised you think Netflix's UI and UX is that poor. Which streaming service do you think does a better job?
- hugey010 4y agoNone of them, since they basically all copied Netflix! The grid view limits users to slowly looking over limited categories of content. Any list based tree structure would be better in my opinion.
- Gigachad 4y agoI think you are overestimating your knowledge of design and UI. Mainstream software has been rigorously researched, tested, and proven to work. If we left UI design up to HN users we would end up with some plain text directory listing with vim keybindings.
- ckw 4y agoAt some point around 2006 I could sort all Netflix titles by user rating and even better, what Netflix expected I would rate a given title. This expected rating became incredibly accurate after I’d rated 50-100 titles according to the extant five star system. It was so good that I found myself watching many titles that I otherwise wouldn’t have considered, because the system was invariably right. I could also safely avoid titles I was very interested in when the system surmised I would be disappointed. And I could very easily inspect, sort, and edit my watch list. Today I rarely use Netflix, and I wouldn’t pay for it. Periodically I open the app and add to my list those titles that are immediately visible which I know I want to watch, like comedy specials of comedians I’m familiar with, but I don’t inspect further, because if I haven’t already heard about a title from some external sources I trust, it’s not worth my time to check. That list just grows and grows, though titles are often removed as they become unavailable, but I never prune the list, because when I remove one title, I’m taken all the way back to the beginning of the list. Trying is just a wast of time. It is baffling to me that anyone could interact with Netflix’s current UI and conclude that it was anything but a raging dumpster fire.
- geodel 4y agoSeems to be mixing too many things here. Many scaling/ hardware challenges need a lot of people but it can still be true that Netflix has choke full of engineers making half-assed turd Java frameworks day in and day out. I know this because we are forced to use these crappy tools as they are made by Netflix so supposed to be best. It's just that they succeeded in streaming market with low competition and great success bring in lot of post facto justifications on how outrageously great Netflix tech infra is. I mean it may be excellent for their purpose but to think their solution can be industry wide replicated seems not true to me.
- tankenmate 4y agoQue? You don't seem to have much justification for your points; it seems more like a rant as you have had a bad experience using software provided by Netflix. It would be great if you could provide more details about what was wrong with it rather than just "we are forced to use these crappy tools". I'm genuinely interested. In my personal experience lots of companies (admittedly all large companies, but many of which sell their services / software / hardware to smaller companies) have a use for serving hundreds of Gbps of static file traffic as cheaply as possible. And the slides for this talk seem exactly on the money (again from my experience slinging lots of static data to lots of users).
- paxys 4y agoSo Netflix published a framework which seemingly isn't suitable for your use case, your managers forced you to use it, and your response is to blame...Netflix?
- seydor 4y agoI dont see the point. A centralized data hose that is replacing what internet was designed to be : a decentralized, multi routed network. The problem may be useful to them, but unlikely to be useful to anyone who doesn't already work there. I dunno, if it was possible to monetize decentralized or bittorrent video hosting, i think it would solve the problem in a more interesting and resilient way. With fewer engineers. But it's like, every discussion today must end with something about the pay and head count of engineers.
- oleganza 4y agoI understand and even share a little bit of your sentiment, but I'm tired of stretched "X is now not what X was supposed to be". Strictly speaking, the Internet was supposed to help some servers survive and continue working together despite some others being destroyed by a nuke. That is more-or-less the case today: we see how people use VPNs to route around censorship. Whether you were supposed to stream TikTok videos directly from the phones of their authors or through a centralized data hose - i'm not sure that was ever the grand idea. Also "decentralized" and "monetize" don't go well together because innovation is stimulated by profit margins and rent-free decentralized solutions by definition have those margins equal to zero (otherwise the solution is not decentralized enough).
- jedberg 4y agoIt's funny you mention this. When I worked at Netflix, we looked at making streaming peer to peer. There were a lot of problems with it though. Privacy issues, most people have terrible upload bandwidth from home, people didn't like the idea of their hardware serving other customers, home hardware is flakey so you'd constantly be doing client selection, and other problems. So it turns out decentralized multi routed is not a good solution for video streaming.
- seydor 4y agorecently i see a lot of people with very high upload speeds. Nobody is using them though, but nominally they are there.
- AtNightWeCode 4y agoScaling streaming for a company at the size of Netflix is very easy. You can use any edge cache solution, even homemade. The complexity at N seems to stem from other things.
- n0tth3dro1ds 4y ago>You can use any edge cache solution Umm, those solutions exist (from places like AWS and Azure) because Netflix was able to do it without them. The cloud platforms recognized that others would want to build their own streaming services, so they built video streaming offerings. You have the cart in front of the horse. The out-of-the-box solutions of today don’t exist without Netflix (and YouTube) building a planet scale video solution first.
- AtNightWeCode 4y agoN had problems in US because they served data from CA. Today, N uses edge caching and the data for me in Europe is sent less than 10km to my home. And it should be cheap. We are talking about serving static content here. It is not very difficult.
- jedberg 4y agoWhy do you think Netflix served out of California? They only did that for the first few months, until they adopted Akamai, Limelight, and L3 CDNs. That was long before Netflix launched in Europe.
- AtNightWeCode 4y agoWell they use to, they tried to bully various ISPs into increasing their throughput before they jumped the edge cache wagon, long time after competitors. Akamai is a stellar company. Don’t think N uses A services today. At the end of the day. N mostly serves static content to users and I highly doubt that hardware costs is a very relevant parameter.
- tetha 4y agoThe way I've been putting it to people lately is: Never underestimate how hard a problem can grow by making it big. And also, at times, it is hard to appreciate how difficult something becomes if you haven't walked the path at least partially. Like, from work, hosting postgres. At this point, I very much understand why a consultant once said - "You cannot make mistakes in a postgres 10GB or 100GB and a dozen transactions per second in size". And he's right, give it some hardware, don't touch knobs except for 1 or 2 and that's it. The average application accessing our postgres clusters is just too small to cause problems. And then we have 2 postgres clusters with a dataset size of 1TB or 2TB peaking at like 300 - 400 transactions per second. That's not necessarily big or busy for what postgres can do, but it becomes noticeable that you have to do some things right at this point and some patterns just stop working hard. And then there are people dealing with postgres instances 100 - 1000x bigger than this. And that's becoming tangibly awesome and frightening by now, using awesome in a more oldschool way there.
- mlrtime 4y agoNot only make it big, engineer it in a way that makes it profitable for the business. I'm sure there are many teams that could design such a network with nearly unlimited resources, but it is entirely different when you have profit margins.
- victor106 4y agoAs someone once said “Big is different”
- jiggawatts 4y agoI know how to operate a hose! How hard can it be to manage a large stream? It's "just" more water: https://www.youtube.com/watch?v=jxNM4DGBRMU https://www.youtube.com/watch?v=jxNM4DGBRMU
- rakoo 4y agoSure, if you place yourself in an arbitrarily hard problem, it takes a lot to solve it. "How we dug a 100m pit without using machines in 2 days" is an incredible feat, but the constraints only serve those who put them. Serving large content has been solved for decades already. It's much easier and reliable to serve from multiple sources, each at their maximum speed. Want more speed ? Add another source. Any client can be a source. Netflix artificially restrains itself by only serving from their machines. It is a very nice engineering feat, but is completely artificial. As a user it feels weird to think of them highly when they could just have gone the easier road.
- zinclozenge 4y agoHow would you do it if you had much more modest scale requirements? Say a few thousand simultaneous viewers. I'm kicking around an idea for a niche content video streaming service, but I don't know much about the tech stacks for it.
- vagrantJin 4y agoA few thousand? Just use Nginx and a backend lang of your choosing.
- zinclozenge 4y agoNot even bother using a cdn?
- ev1 4y agoFor low-traffic niche content that might not be a cache hit in the first place in every region? I wouldn't bother. Unless you use storage at the CDN - which is probably very not cost effective for you.
- rakoo 4y agoUse bittorrent. Every viewer is also a source. The more people watch, the less your servers are loaded. Bittorrent is built towards "offline" viewing. Try Peertube for a stack that is more built for streaming and has bittorrent sharing built-in (actually webtorrent, because the browser doesn't speak raw TCP or UDP, but the idea is the same)
- nwallin 4y ago> the popular "why does Netflix need X thousand engineers, I could build it in a weekend" sentiment that is frequently brought up on forums like this one. I don't think that's a popular sentiment about Netflix. Twitter, Reddit, Facebook, yes, but Netflix, YouTube, Zoom, not so much.
- xnx 4y ago> Building software and productionalizing/scaling it are two very different problems, and the latter is far more difficult. Is this claim based on some example I should know? Countless companies never achieve product/market fit, but very few I can think of fail because they weren't able to handle all their customers.
- ternaryoperator 4y agoThis! I am frustrated at how often devs will not accept that simple things become incredibly complicated at scale. That favorite coding technique? That container you wrote? Those tests you added? All good, but until you've tested them at scale, don't assert that everyone should use them. This dynamic is true in the other direction too: that techniques often taken for granted simply are not feasible in highly resource-constrained environments. With rare exception, the best we can say with accuracy is that "I find X works well enough in the typical situations I code for."
- trasz 4y agoThis goes both ways - things written “to scale” often, or usually, do worse in more typical, smaller scenarios, because of all the bloat that’s irrelevant to 98% of use cases. That’s one of the fundamental problems in todays Open Source world: everything’s being optimized for a very rare case.
- throwaway012377 4y ago> simple things become incredibly complicated at scale In a way it's the opposite. Things scale up by removing complex interactions. Software gets faster by solving the same problem with less steps. Any beginner can write to much code and pile up too many software and hardware components. It takes knowledge to simplify things.
- ProZsolt 4y agoIt reminds me of a saying: "Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."
- maerF0x0 4y ago> Any beginner can write to much code and pile up too many software and hardware components. I'd take it a step further, the beginner has no choice but to write too much code/cruft. By definition they lack the experience required to know the options before them, many of which are simpler.
- jdyyc 4y agoI work on a very technically trivial service at a large company. It's the kind of thing that people run at home on a raspberry pi, docker container or linux server and it consumes almost no resources. But at our organization this needs to scale up to millions of users in an extremely reliable way. It turns out this is incredibly hard and expensive and takes a team of people and a bucket of money to pull it off correctly. When I tell people what I work on they only think about their tiny implementation of it, not the difficulty of doing it at an extreme scale.
- throwaway012377 4y ago> "why does Netflix need X thousand engineers, I could build it in a weekend" Believe me or not, I was in a company doing web file streaming in 2009 using Nginx, sendfile and SSL offloading on the NIC. It was installed by one dude. A standard Linux distro, standard kernel and no custom software. Just compile the SSL offloading kernel module once.
- yibg 4y agoAnd how many concurrent users did you have?
- throwaway012377 4y agoIt was about large file streaming so bandwidth is the bottleneck, not number of users. What matters is that the configuration was good enough to use the NICs at 100%
- summerlight 4y agoYeah, that's perhaps nice (and hopefully moderately interesting) enough for a hobbyist work. Good luck for multiplying the scale by 1,000,000 for many dimensions.
- throwaway012377 4y agoCompletely wrong. The configuration I described was replicated on different servers and they scale linearly, obviously. It was saturating the NICs and therefore would have been good enough for Netflix at the time. It's strange that you assumed otherwise.
- zppln 4y agoI'm not sure this particular presentation helps your point though? I sifted through it and if anything I was struck with how simple it seemed? I'm sure there's more to running Netflix though and in my mind they're allowed to have as many engineers as they see fit.
- maerF0x0 4y agoand maybe it can help stem asinine "System Design" interviews like "Design netflix" I call them asinine because these kinds of architectures aren't written out in one day (let alone 60 minutes) and it's silly to not build an evolving setup (unless you are replacing a legacy system in a company with scale, but then again you have more than 60 minutes...). I wish system design interviews would not just think about if you know how to write high scale designs, but if you know how to build tiny designs that minimize footprint before scale and give maximum flexibility for scaling when its indicated ...
- Yottr 4y agoThat's hilarious. To me it reinforces how few resources you need to run an operation like Netflix. The hardware is brilliantly fast. The more engineering you do, the slower it gets. Engineering is an aristocratic tradition. What you see today is an imitation of that, done badly, which Netflix tries to circumvent.