7 ms·
Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this: 1) t
by shermantanktop 2y ago
Every time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this:
1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine.
2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like.
So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”
- grogenaut 2y agoyou are so right about that. tho I'm sure that many of the netflix folks are still doing their after action analysis in prep for Dec 25 NFL. now take this realization and apply it to any news article or forum post you read and think about how uninformed they actually are.
- fragmede 2y agoIf NFL decides to keep Netflix for that, that is. The bandwidth for that fight was rookie numbers, and after that fiasco, why would the NFL not break their contract and choose someone with a proven track record doing bigger live events, like the World Cup?
- phil21 2y agoBecause Netflix pays them either way, I would imagine. Breaking a contract on a sure thing to the tune of tens (hundreds?) of millions of dollars for a maybe is a large business risk. Reputational damage is going to be far more Netflix than the NFL if they totally club it. That and this fight is going to likely be an order of magnitude more viewers than the Christmas NFL games if the media estimates on viewership were remotely accurate. You’re talking Super Bowl type numbers vs a regular season NFL game. The problems start happening at the margin of capacity most of the time.
- listenallyall 2y agoBut "reputational damage" doesn't affect profits. Nobody is canceling Netflix because they had issues watching the fight, just like nobody will cancel if the NFL experience sucks on Netflix. They will bitch and moan on Twitter, but it's essentially just talk.
- dpkirchner 2y agoRight? A common complaint by outsiders is that Netflix uses microservices. I'd love to hear exactly how a monolith application is guaranteed to perform better, with details. What is the magic difference that would have ensured the live stream would have been successful?
- miked85 2y agoIt's not guaranteed, but much fewer points of failure.
- chipdart 2y ago> It's not guaranteed, but much fewer points of failure. Can you explain where this is relevant to buffering issues? Also, you are very wrong regarding failure modes. The larger the service, the more failure modes it has. Moreover, in monoliths if a failure mode can take down/degrade the whole service, all other features are taken down/degraded. Is having a single failure mode that brings down the whole service what you call fewer points of failure?
- miked85 2y agoI can't, since I don't know Netflix's architecture - I was responding to "I'd love to hear exactly how a monolith application is guaranteed to perform better, with details."
- chipdart 2y ago> I was responding to "I'd love to hear exactly how a monolith application is guaranteed to perform better, with details." I asked nothing about Netflix. My question was directed at your remark regarding monoliths vs microservices. Now, can you answer the question?
- miked85 2y agoMonoliths might be better in some cases, microservices in others. Performance can obviously differ based on implementation.
- karaterobot 2y agoThe only time I worked on a project that had a live television launch, it absolutely tipped over within like 2 minutes, and people on HN and Reddit were making fun of it. And I know how hard everyone worked, and how competent they were, so I sympathize with the people in these cases. While the internet was teeing off with easy jokes, engineers were swarming on a problem that was just not resolving, PMs were pacing up and down the hallway, people were getting yelled at by leadership, etc. It's like taking all the stress and complexity of a product launch and multiplying it by 100. And the thing I'm talking about was just a website, not even a live video stream.
- ryoshu 2y agoThose are the times when you identify who is there to help and who is there to be performative.
- shermantanktop 2y agoThose performative people are worse than useless. They take up critical bandwidth and add no real value. An effective operational culture has methods for removing those people from the conversations that matter. Unfortunately that earns you a reputation for being “cutthroat” or “lacking empathy.” Both of those are real things, but it’s the C players who claim they are being unfairly treated, when in fact their limelight-seeking behavior is the problem. If all that sounds harsh, like the kitchen on The Bear, well…that’s kinda how it is sometimes. Not everyone thrives in that environment, and arguably the ones who do are a little “off.”
- swyx 2y agowhat was the ultimate cause/fix of issues in your case? a database thing?
- nikau 2y agoInsufficient testing
- 2y ago
- deleted 2y ago[deleted]
- survirtual 2y agoFor an event like this, there already exists an architecture that can handle boundless scale: torrents. If you code it to utilize high-bandwidth users upload, the service becomes more available as more users are watching -- not less available. It becomes less expensive with scale, more available, more stable. The be more specific, if you encode the video in blocks with each new block hash being broadcast across the network, just managing the overhead of the block order, it should be pretty easy to stream video with boundless scale using a DHT. Could even give high-bandwidth users a credit based upon how much bandwidth they share. With a network like what Netflix already has, the seed-boxes would guarantee stability. There would be very little delay for realtime streams, I'd imagine 5 seconds top. This sort of architecture would handle planet-scale streams for breakfast on top of the already existing mechanism. But then again, I don't get paid $500k+ at a large corp to serve planet scale content, so what do I know.
- nemothekid 2y agoTorrents are awful for live events. 1. Everyone only cares about the most recent "block". By the time a "user" has fully downloaded a block from Netflix's seedbox, the block is stale, so why would any other user choose to download from a peer rather from netflix directly? 2. If all the users would prefer to download from netflix directly rather than a p2p user, then you already have a somewhat centralized solution, and you gain nothing from torrents.
- alex-mohr 2y agoIf Netflix were working correctly and could handle the load, you'd absolutely be correct. But it does seem the capacity of a hybrid system of Netflix servers plus P2P would be strictly greater than either alone? It's not an XOR. And note that in this case of "live" streaming, it still has a few seconds of buffer, which gives a bandwidth-delay product of a few MB. That's plenty to have non-stale blocks and do torrent-style sharing.
- nemothekid 2y agoIf switching to a peer causes increased buffering (which it will, because you still have to wait for the peer to download from Netflix) then you will still have the original problem Netflix is suffering from. If the solution to users complaining about buffering is to build a system with more inherent buffering then you are back at square one. I think it’s might be helpful to look at netlfix’s current system as already a distributed video delivery system in which they control the best seeds. Adding more seeds may help, but if Netflix is underprovisioned from the start you will have users who cannot access the streams
- deleted 2y ago[deleted]
- ryandv 2y agoI've noticed this amongst the newer "careerist" sort of software developer who is stumbling into the field for money, as opposed to the obsessive computer geek of yesteryear, who practiced it as a hobby. This character archetype is a transplant, say, less than five years ago from another, often non-technical discipline, and was taught or learned from overly simplistic materials that decry systems programming, or networking, or computer science concepts as unnecessary, impractical skills, reducing everything to writing JavaScript glue code between random NPM packages found on google. Especially in a time where the gates have come crashing down to pronouncements of, "now anybody can learn to code by just using LLMs," there is a shocking tendency to overly simplify and then pontificate upon what are actually bewilderingly complicated systems wrapped up in interfaces, packages, and layers of abstraction that hide away that underlying complexity. It reminds me of those quantum woo people, or movies like What the Bleep Do We Know!? where a bunch of quacks with no actual background in quantum physics or science reason forth from drastically oversimplified, mathematics-free models of those theories and into utterly absurd conclusions.
- lclarkmichalek 2y agoWhat does this have to do with the topic being discussed?
- svnt 2y agoBecause it is about people speculating on events that seem connected to their own experience, but in actuality aren’t, because they don’t understand the breadth of the distribution of the abstraction they are discussing. This happens when your terms are underspecified: someone says “Netflix’s servers are struggling under load” and while people in similar efforts know that basically just equivalent to “something is wrong” and the whole conversation is basically esoteric to most people outside a few specialized teams, these other people jump to conclusions and start having conversations based on their own experience having to do with what is (to them) related (and usually fashionable, because that is how most smaller players figure out how to do things).
- bumby 2y agoIn short, people with glib answers tend to rely on over simplified models that don’t reflect reality.
- doctorpangloss 2y agoYou don’t belong to either group. What does this make you?
- gscott 2y ago[dead]
- shermantanktop 2y agoYou may have belonged to one of those groups in the past, or maybe you will someday. I certainly have. Many of the more seasoned folks on HN have. Stuff goes wrong, random internet people jump on the opportunity to speculate and say wildly off-the-mark comments, and the engineers trying to keep the ship from sinking have to sit quietly for fear of making the PR backlash worse.
- tomcam 2y agoMost or all of your replies are to people who hallucinate things you didn’t say. Your patience is inspiring.
- shermantanktop 2y agoI was interviewing a dev candidate some years ago and they were totally lost trying to traverse a tree on the whiteboard. I kept helping them get unblocked, because my philosophy is that anyone can get stuck once, but if I’m supposed decide whether to hire you, I should get the most/best data I can. Another person was observing the interview, for training purposes, and afterwards said to me: “Do you have kids? You have so much patience!”
- tomcam 2y agoIf I weren’t retired, I would totally apply to work for you
- david-gpu 2y agoCompletely agreed. There are also former employees who have very educated opinions about what is likely going on, but between NDAs and whatnot there is only so much they are willing to say. It is frustrating for those in the know, but there are lines they can't or won't cross. Whenever an HN thread covers subjects where I have direct professional experience I have to bite my tongue while people who have no clue can be as assertive and confidently incorrect as their ego allows them to be.
- shermantanktop 2y agonods knowingly
- fragmede 2y agoSome people can just let others be wrong and just stay silent, but some people can't help themselves. So if you say something really wrong, like this was caused by Netflix moving to Azure, they should have stayed on AWS! someone will come along to correct you. If you're looking for the right answer, post the wrong one, alongside some provoking statement (Windows is better than Linux because this works there), and you'll get the right answer faster than if you'd asked your question directly. https://xkcd.com/386/ https://xkcd.com/386/
- grayhatter 2y ago> Some people can just let others be wrong and just stay silent, but some people can't help themselves. As one of thoes who cant help themselves; the way you phrase it feels a bit too cynical, I've always interpreted it as people want to help, but don't want to offer something that's wrong. Which is basically how falsifiable science works. It's so much easier to refute the assertion that birds generate lift with tiny backpacks with turboprops attached. Than it is to explain the finer details of avian flight mechanics. I couldn't describe above a superficial level how flapping works, but I can confidently refute the idea of a turboprop backpack. (Everyone knows birds gave up the turboprop design during the great kerosene shortage of 1128)
- 2y ago
- croes 2y agoYou are basically saying, everybody who criticizes Netflix now has no clue. That’s a bold claim given that people with inside knowledge could post here without disclosing they are insiders. Is that some kind of No True Scotsman?
- shermantanktop 2y agoI’m just pointing out that there are Netflix engineers reading all these words. For every thread like this, there are likely people who are readers but cannot be writers, even though they know a lot. That means the active posters exclude that group, by definition. These threads often have interesting and insightful comments, so that’s cool.
- pfraze 2y agoAt the scale that Netflix just dealt with? Yeah I honestly think this is a case where less than 5000 people in the world are really qualified to comment.
- chgs 2y agoNot clear what scale they were attempting, but yes delivering a live stream to 10m+ users on the public internet with a reasonable end to end latency (under 30 seconds glass to viewer) is not a trivial problem, and it’s not something Netflix do a lot. It’s a very different problem to distributing video on demand which is Netflix’s core business.
- tomcam 2y ago> You are basically saying, everybody who criticizes Netflix now has no clue. GP clearly meant some people not everybody. You are the one making bold claims.
- _giorgio_ 2y ago[flagged]
- anothernewdude 2y ago> who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.” And looking through the comments, this is just wrong.
- eqvinox 2y ago3) the people supplying 1) and 2) with tools (hard- or software) We (yep) don't know the exact details, but we do get sent snapshots of full configs and deployments to debug things... we might not see exact load patterns, but it's enough to know. And if course we can't tell due to NDAs.
- johnnyanmac 2y agoI'm sure 2) can post. But it won't be popular, so you'll need to dig to find it. Most people are consumers and at the end of the day, their ability to consume a (boring) match was disrupted. If this was PPV (I don't think it is) the paid extra to not get the quality of product they expected. I'm not surprised they dominate the conversation.
- bobdvb 2y agoI am 2, I absolutely will get argued with by people who think they know better. I'm also not going to criticise my peers because they could recognise me and I might want to work with them one day.
- arduanika 2y agoAnd nonetheless, it freezes up.