4 ms·

Exactly what is Netflix doing that requires such skilled programmers? As a consumer of its product I haven't noticed any significant groundbreaking changes to t

Exactly what is Netflix doing that requires such skilled programmers? As a consumer of its product I haven't noticed any significant groundbreaking changes to the platform in years. I find more people complaining about the software experience as years go by. What have these rockstar programmers done exactly? I can tell you it's for sure not the UX, or its recommendation engine, which are frankly both abysmal. I'm guessing it's less the quality of the code they produce and more the quantity. Their rockstar programmers can work 10x more efficiently than other programmers, thus the CEO can hire 10x fewer programmers.

hayksaakian 6y ago
Right that's the theory and you can read about it in his various books. So instead of paying $260k per year for 10 engineers (2.6M), they might pay 451k for 3 "10x" engineers (1.3M). Costs are 50% lower overall, and if the engineers are even 2x more productive than the average you "break even". Smaller staff means you need fewer managers. I'm not saying this is actually how it works in real life (can a developer really be 10x more productive? who know). This is just their operating theory. Source: https://www.levels.fyi/?compare=Netflix,Google,Facebook,Microsoft&track=Software%20Engineer https://www.levels.fyi/?compare=Netflix,Google,Facebook,Micr...
- rstupek 6y ago
  Smaller teams mean less time spent communicating as well so higher productivity?
nixgeek 6y ago
They write about some of the interesting engineering work on various blogs. https://netflixtechblog.com/ https://netflixtechblog.com/
claudiulodro 6y ago
I haven't worked at Netflix, but I believe there are a lot of problems around video encoding and infrastructure which are quite tricky to solve. Netflix uses something like 15% of the world's internet bandwith, and the infrastructure/delivery challenges are probably enormous.
drdebug 6y ago
They lead in several areas such as video quality measurement (VMAF https://github.com/Netflix/vmaf https://github.com/Netflix/vmaf), robustness (Chaos Monkey https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/), video compression optimization (https://netflixtechblog.com/optimized-shot-based-encodes-for-4k-now-streaming-47b516b10bbb?gi=5eca1df094af https://netflixtechblog.com/optimized-shot-based-encodes-for...) and probably more.
- omgwtfbyobbq 6y ago
  I feel like they might be going a bit overboard with video compression optimization. As an example, Star Trek Voyager in HD looks significantly worse than OTA 480i. On the plus side, it looks good in 4k.
  - drdebug 6y ago
    I agree, their quality is sometimes questionable but to be fair OTA has preallocated/fixed bandwidth, while netflix on the other hand likely gains from optimizing for lower bitrates.
    - omgwtfbyobbq 6y ago
      They can optimize for lower bitrates, which they've been doing for a while, but the recent drop in bitrate is really awful. https://variety.com/2020/digital/news/netflix-degrading-hd-video-quality-europe-bit-rates-1234605316/ https://variety.com/2020/digital/news/netflix-degrading-hd-v... Even if they are bandwidth constrained, I wouldn't be surprised if some shows looked better at 480i/.5Mbps than 1080p/.5Mbps.
      - drdebug 6y ago
        Yes, and it's very frustrating too: they lowered the bitrate when Thierry Breton, the European Commissioner for Internal Market https://en.wikipedia.org/wiki/Thierry_Breton https://en.wikipedia.org/wiki/Thierry_Breton said the internet would not support the extra load due to work-from-home/COVID. Breton was CEO of Orange, internet & 4G provider. This is all sounds very suspicious, and it was clearly wrong. Some of us pay for UHD quality and we don't get what we initially paid for.
drewg123 6y ago
I work for Netflix Open Connect (CDN). We've been leaders on CPU and power efficiency with things like async sendfile, kernel TLS, and more. We've had servers doing 100Gb/s in production for 4 years, while competing CDNs are just now rolling out 100Gb/s servers. I have a prototype machine right now serving 375Gb/s of TLS encrypted customer video traffic with just 32 cores.
- Sohcahtoa82 6y ago
  > serving 375Gb/s of TLS encrypted customer video traffic with just 32 cores. How does that even work? The fastest system RAM you can get that I know of is DDR4-5000, which transfers at 40 GB/s (320 Gb/s). How can you encrypt and send data faster than your memory can even transfer it to the CPU? Or is there something I'm misunderstanding?
  - drewg123 6y ago
    Hardware TLS done in the NIC :)
runevault 6y ago
Anything at a large enough scale becomes a hard problem. And they are one of the largest scales there is considering how many people use their service daily.

hayksaakian 6y ago

Right that's the theory and you can read about it in his various books. So instead of paying $260k per year for 10 engineers (2.6M), they might pay 451k for 3 "10x" engineers (1.3M). Costs are 50% lower overall, and if the engineers are even 2x more productive than the average you "break even". Smaller staff means you need fewer managers. I'm not saying this is actually how it works in real life (can a developer really be 10x more productive? who know). This is just their operating theory. Source: https://www.levels.fyi/?compare=Netflix,Google,Facebook,Microsoft&track=Software%20Engineer https://www.levels.fyi/?compare=Netflix,Google,Facebook,Micr...

rstupek 6y ago
Smaller teams mean less time spent communicating as well so higher productivity?

nixgeek 6y ago

They write about some of the interesting engineering work on various blogs. https://netflixtechblog.com/ https://netflixtechblog.com/

claudiulodro 6y ago

I haven't worked at Netflix, but I believe there are a lot of problems around video encoding and infrastructure which are quite tricky to solve. Netflix uses something like 15% of the world's internet bandwith, and the infrastructure/delivery challenges are probably enormous.

drdebug 6y ago

They lead in several areas such as video quality measurement (VMAF https://github.com/Netflix/vmaf https://github.com/Netflix/vmaf), robustness (Chaos Monkey https://netflix.github.io/chaosmonkey/ https://netflix.github.io/chaosmonkey/), video compression optimization (https://netflixtechblog.com/optimized-shot-based-encodes-for-4k-now-streaming-47b516b10bbb?gi=5eca1df094af https://netflixtechblog.com/optimized-shot-based-encodes-for...) and probably more.

omgwtfbyobbq 6y ago
I feel like they might be going a bit overboard with video compression optimization. As an example, Star Trek Voyager in HD looks significantly worse than OTA 480i. On the plus side, it looks good in 4k.
- drdebug 6y ago
  I agree, their quality is sometimes questionable but to be fair OTA has preallocated/fixed bandwidth, while netflix on the other hand likely gains from optimizing for lower bitrates.
  - omgwtfbyobbq 6y ago
    They can optimize for lower bitrates, which they've been doing for a while, but the recent drop in bitrate is really awful. https://variety.com/2020/digital/news/netflix-degrading-hd-video-quality-europe-bit-rates-1234605316/ https://variety.com/2020/digital/news/netflix-degrading-hd-v... Even if they are bandwidth constrained, I wouldn't be surprised if some shows looked better at 480i/.5Mbps than 1080p/.5Mbps.
    - drdebug 6y ago
      Yes, and it's very frustrating too: they lowered the bitrate when Thierry Breton, the European Commissioner for Internal Market https://en.wikipedia.org/wiki/Thierry_Breton https://en.wikipedia.org/wiki/Thierry_Breton said the internet would not support the extra load due to work-from-home/COVID. Breton was CEO of Orange, internet & 4G provider. This is all sounds very suspicious, and it was clearly wrong. Some of us pay for UHD quality and we don't get what we initially paid for.

drewg123 6y ago

I work for Netflix Open Connect (CDN). We've been leaders on CPU and power efficiency with things like async sendfile, kernel TLS, and more. We've had servers doing 100Gb/s in production for 4 years, while competing CDNs are just now rolling out 100Gb/s servers. I have a prototype machine right now serving 375Gb/s of TLS encrypted customer video traffic with just 32 cores.

Sohcahtoa82 6y ago
> serving 375Gb/s of TLS encrypted customer video traffic with just 32 cores. How does that even work? The fastest system RAM you can get that I know of is DDR4-5000, which transfers at 40 GB/s (320 Gb/s). How can you encrypt and send data faster than your memory can even transfer it to the CPU? Or is there something I'm misunderstanding?
- drewg123 6y ago
  Hardware TLS done in the NIC :)

runevault 6y ago

Anything at a large enough scale becomes a hard problem. And they are one of the largest scales there is considering how many people use their service daily.