15 ms·

Netflix now supports TLS 1.3

308 points by caution 6y ago

MaxBarraclough 6y ago
I noticed they didn't explicitly mention why they feel the need to ensure authentication+confidentiality+integrity for their streams, given that the data they're dealing with is films and TV shows, rather than, say, payment details. As I understand it, they use HTTPS to prevent spying and data-mining by unscrupulous ISPs. It doesn't affect their DRM at all, which would work just as well over plain HTTP.
- daurnimator 6y ago
  Also to prevent traffic shaping: ISPs throttle traffic that contains netflix.com in the SNI. ESNI (encrypted SNI) comes with TLS 1.3.
  - cheschire 6y ago
    Ah, the good ol' radar detector reflector. There was a skit I saw a couple decades ago where a person was showing off his radar detector, then in order to combat that, the police had developed a radar detector reflector, so then he had made a radar detector reflector protector or something like that, then they made a protector detector, and so on. It was a couple minutes of explaining his best efforts to counter the police counters to his counters and on and on. But, you know, funny.
    - tbrock 6y ago
      You’re probably thinking of the movie “the big hit”. It’s trace busters: https://m.youtube.com/watch?v=Iw3G80bplTg https://m.youtube.com/watch?v=Iw3G80bplTg The trace busta busta busta.
  - v7p1Qbt1im 6y ago
    Networking noob here. Can‘t the ISP see Netflix‘s ASN/commonly used IP‘s and shape that way?
    - gsnedders 6y ago
      Yes. The case where it's much more useful is traffic going in/out of major CDNs to shared IP addresses. For Netflix or someone else running their own CDN the gains are much smaller.
      - MaxBarraclough 6y ago
        So ISPs could still impose traffic shaping on Netflix if they wanted to, I take it? I can't see how Netflix could prevent this. Ultimately, their servers ('OCAs') have easily detectable IP addresses, right?
        throwaway287391 6y ago
        Fast.com probably helps quite a bit. If ISPs throttle Netflix' traffic their fast.com measurements will look bad and customers will complain/sue. (AFAIK there's no way to distinguish between fast.com tests and actual Netflix video consumption since the former's traffic patterns are identical(?) to a Netflix video streaming client's.) Creating fast.com always seemed like a pretty brilliant move by Netflix to me.
        MaxBarraclough 6y ago
        Good point. Smaller players can't make the same move though.
        Tijdreiziger 6y ago
        I think that was the whole point of fast.com, right? Speeds to known 'speedtest' servers were manipulated by ISPs, and this was bad for Netflix's business because customers would ostensibly have good speeds yet have slow Netflix, so fast.com was created as a way to measure the 'real' speed to Netflix's servers.
  - georgyo 6y ago
    ESNI was dropped from the TLS1.3 spec. It is currently a draft protocol. Almost nothing supports ESNI yet. Chrome does not have it yet. Firefox does but it very difficult to enable, there is a config flag but it does nothing on its own unless you also enable DNS over HTTP in Firefox. OpenSSL has no support for ESNI yet either. ESNI also never tells the user if it is working or not yet, making downgrades fairly easy. ESNI is a long way from being deployed, let alone useful.
    - moduspol 6y ago
      They're Netflix, though, so they've got a lot of use cases where they control both ends of the connection (e.g. they've got a native app for the end user). If it's important to them to have ESNI support, they presumably could do it for those use cases.
    - middleclick 6y ago
      CloudFlare has ESNI enabled (and so do websites hosted by it, like medium.com) so it's not as if no one is doing it.
      - georgyo 6y ago
        If no clients support it, and you can't tell as a user if it is working, then effectively no one is using it.
        middleclick 6y ago
        Firefox supports it and Chrome has plans to support it in the future. I agree that it's not much right now but it's a start.
        shawnz 6y ago
        Firefox supports it, although currently only as a configurable. Users can test their ESNI support online here: https://www.cloudflare.com/ssl/encrypted-sni/ https://www.cloudflare.com/ssl/encrypted-sni/
- pedrocr 6y ago
  > It doesn't affect their DRM at all, which would work just as well over plain HTTP. Does it work at all though? I does prevent paying customers from getting higher quality streams. But aren't all the shows still widely available in torrents at whatever quality you want?
- baq 6y ago
  from the summary: > From the field test, we are confident that TLS 1.3 provides us a better streaming experience. so they actually measured an improvement in performance metrics.
  - MaxBarraclough 6y ago
    That's a performance comparison against TLS 1.2. It doesn't speak to why they're concerned about authentication+confidentiality+integrity for audio-video streams in the first place.
    - CydeWeys 6y ago
      Are you hinting at something here, some threat maybe, that you think Netflix is responding to? Or do you just not think that more security is a good thing generally and should be done for its own sake?
      - MaxBarraclough 6y ago
        It's possible to distribute large binary blobs over unencrypted HTTP, and to provide security using hash checks. I believe apt no longer does this due to a flaw in their implementation that would not have been an issue if they'd used HTTPS. I'm unsure if Steam still uses insecure HTTP, but they certainly used to. [0] [1] I'm all for HTTPS everywhere, but for Netflix and YouTube, that means encrypting exabytes of data. That's going to cost them quite a bit. Seems reasonable to go for more detail than It's good practice in general. > Are you hinting at something here No, as I put in my first comment, I think we know the specific reasons (untrustworthy ISPs), it's just interesting they didn't mention them. [0] https://whydoesaptnotusehttps.com/ https://whydoesaptnotusehttps.com/ [1] https://developer.valvesoftware.com/wiki/SteamPipe#Server_administrators https://developer.valvesoftware.com/wiki/SteamPipe#Server_ad...
        CydeWeys 6y ago
        Given that hardware encrypting/decrypting is omnipresent, is it really that much more expensive to stream content securely? I'd like to see the numbers, but I just don't think it's actually a meaningful increase in cost. Network traffic itself tends to be the dominating factor in cost, with encryption just upping your electric bill slightly. It seems like the benefits of transmitting your streaming securely (e.g. defeating interloping ISPs) more than exceed the costs, hence why they're doing it.
- Klathmon 6y ago
  >I noticed they didn't explicitly mention why they feel the need to ensure authentication+confidentiality+integrity for their streams, given that the data they're dealing with is films and TV shows, rather than, say, payment details. Because secure should be the default where we can do it. If I'm watching a show, I don't necessarily want every single router along the way knowing what show i'm watching. Either because of advertising profiles or just because I don't want people knowing that I'm binge watching a specific show, or to go to an extreme, perhaps because what I'm watching isn't legal where I live. If I'm watching a documentary or news show, I don't want the possibility that someone could modify my stream and change parts of it. Misinformation is a big deal, and if we have a simple solution to get rid of whole attack vectors (especially before they become common), why not use it? And TLS gets rid of other attacks as well. token stealing is a lot harder if EVERYTHING is TLS encrypted vs if only some things are. It can also secure against vulnerabilities in the software that reads the streams. They no longer have to harden their systems as much against potentially hostile data, they can be behind well tested TLS termination that only allow verified data through to the "backend". Everything should be encrypted in transit, zero exceptions. Just because you don't think it's important to hide doesn't mean that others don't. Just because you think your parsing is secure doesn't mean that another layer isn't useful for defense in depth. We have the technology. It's free, cheap to run, and doesn't hurt the user experience in any significant way, so why not use it?
  - saurik 6y ago
    > If I'm watching a show, I don't necessarily want every single router along the way knowing what show i'm watching. Unless Netflix has also started padding their compressed video segments to a deterministic size, TLS doesn't really help with the "I don't want people to know what video I'm watching" problem, as the encrypted media segments are first compressed using a variable bitrate encoding that causes a unique pattern in their resulting chunk sizes; and so, with a little database of signatures--apparently even just at the TCP level, without caring at all about the HTTP and TLS layers... which surprised even me, and I'm quite cynical about this stuff--you can pretty rapidly fingerprint what movie someone is watching from a passive traffic dump. https://dl.acm.org/doi/10.1145/3029806.3029821 https://dl.acm.org/doi/10.1145/3029806.3029821 > Identifying HTTPS-Protected Netflix Videos in Real-Time (2017) > After more than a year of research and development, Netflix recently upgraded their infrastructure to provide HTTPS encryption of video streams in order to protect the privacy of their viewers. Despite this upgrade, we demonstrate that it is possible to accurately identify Netflix videos from passive traffic capture in real-time with very limited hardware requirements. Specifically, we developed a system that can report the Netflix video being delivered by a TCP connection using only the information provided by TCP/IP headers. > To support our analysis, we created a fingerprint database comprised of 42,027 Netflix videos. Given this collection of fingerprints, we show that our system can differentiate between videos with greater than 99.99% accuracy. Moreover, when tested against 200 random 20-minute video streams, our system identified 99.5% of the videos with the majority of the identifications occurring less than two and a half minutes into the video stream. Note that this exact same kind of attack has been applied successfully to other places where a pattern of sizes can differentiate usage, including: Google Maps tiles (which are either variably-compressed fixed-dimension images or variably-dense fixed-region vectors), web pages (which are different sizes already but also link to images and other resources that are random sizes, which helps create the fingerprint), and non-personalized type-head search queries (where the result for each letter brings up some unknown set of titles and URLs; as you type each character the sequence of sizes for your intermediate results can expose the query, potentially even without needing any model of the likely keywords). Maps: https://ioactive.com/ssl-traffic-analysis-on-google-maps/ https://ioactive.com/ssl-traffic-analysis-on-google-maps/ Search: https://eprint.iacr.org/2014/959.pdf https://eprint.iacr.org/2014/959.pdf (edit: I just realized it might be interesting for me to note that I am the lead developer of a new VPN protocol called Orchid and have "attempting to solve as many of these issues as generically and yet with as little required overhead as I can" on my todo list; to be very clear: I don't handle any of this stuff yet at all, but should get to it in the next few months... while the company behind Orchid that is paying me to work on the protocol talks a lot about itself already, I'm really intending to do my own "launch" of the product once I solve some of the stability issues and finish my traffic analysis mitigations.)
- tialaramex 6y ago
  https://tools.ietf.org/html/bcp188 https://tools.ietf.org/html/bcp188 Pervasive Monitoring Is an Attack
- londons_explore 6y ago
  ISPs can still datamine which shows each user is watching simply based on the bursts of data as shows have fast moving or slow moving scenes. There are only a few tens of thousands of shows on Netflix, so it ought to be easy to identify.
  - CydeWeys 6y ago
    Has anyone demonstrated that this attack is actually practical? It's super interesting. One way to defeat it, that is already happening to some extent (though not for this reason), is buffering. Maintain a large enough local buffer, and fill it at a constant bitrate, rather than runrate, and you lose this attack vector, no?
    - bostik 6y ago
      The attack is known from VoIP systems: https://www.researchgate.net/publication/321888273_Classification_of_VoIP_traffic_based_on_analysis_of_VBR_codecs https://www.researchgate.net/publication/321888273_Classific... Adapting the research to other non-CBR streams doesn't sound far-fetched.
      - CydeWeys 6y ago
        VoIP is a completely different use case though because it needs to be real-time. A latency of even 100ms is noticeable. By contrast, Netflix is streaming pre-recorded content, so any latency is not perceived at all so long as it is smaller than the buffer size. E.g. I could have a 5 minute buffer and suffer a 4 minute Internet outage and never so much as notice, whereas that would obliterate any VoIP use case.
        bostik 6y ago
        I agree it's not exactly the same. I would also love to inhabit the parallel universe where Netflix hands out a 4-minute buffer, because in this one 20 seconds seems to be on the high end. However, I strongly suspect that Netflix's video and audio are sent in different streams. Occasionally video for some titles is missing but audio gets through, confusing our daughter to no end. So while you can't infer individual syllables from the audio stream (as you would from VoIP), the audio streams should have varying enough size and chunking characteristics that allow to identify them.
- Spooky23 6y ago
  ISPs compete with them. They need to think of Comcast wanting to degrade the experience of watching a show that is also available on Peacock.
- derivativethrow 6y ago
  To give you an answer focused on the crypto itself, because the way they explained the TLS properties doesn't make this clear: 1. Integrity is redundant with authentication, so really you could say they're ensuring confidentiality + authentication. You can't authenticate a thing without implicitly obtaining assurance of integrity. It's a strictly stronger property. 2. Confidentiality is (usually) insecure and unreliable without authentication. Without authentication you have no PKI for a key exchange to symmetric encryption, so you can't even do TLS in the first place. And if you don't have a carefully applied MAC or a native AEAD mode, your symmetric mode isn't that secure either. So really what you're asking reduces to the question of why they need the most sophisticated TLS scheme for encrypting their streams. If they want the most secure TLS scheme for confidentiality, TLS 1.3 is the way to do it. They explained one particular facet of why this is the case, re: perfect forward secrecy.
- Ambroos 6y ago
  In browsers at least, DRM doesn't work if your full page doesn't use HTTPS. The encrypted media extensions (EME) are only available on secure contexts, which means the page needs to be HTTPS. Segments are typically fetched with standard XHR/fetch, which means they need to use HTTPS on secure origins as well. You really don't want to have to deal with mixed content. Unrelated to DRM, Apple's ATS on iOS kinda requires developers to only use HTTPS origins in their apps. At that point, you might as well just do HTTPS everywhere.
  - MaxBarraclough 6y ago
    > encrypted media extensions (EME) are only available on secure contexts, which means the page needs to be HTTPS I didn't know that, but you're right. [0] There was a time Netflix didn't use HTTPS. [1] I guess they must not have supported HTML5 at that time, and must have been using Silverlight for in-browser viewing. [0] https://www.w3.org/TR/encrypted-media/#privacy-secureorigin https://www.w3.org/TR/encrypted-media/#privacy-secureorigin [1] https://arstechnica.com/information-technology/2015/04/it-wasnt-easy-but-netflix-will-soon-use-https-to-secure-video-streams/ https://arstechnica.com/information-technology/2015/04/it-wa...
- Rebelgecko 6y ago
  >given that the data they're dealing with is films and TV shows, rather than, say, payment details. I definitely have had to enter payment details on Netflix.com before
  - MaxBarraclough 6y ago
    We're talking about why they encrypt their media streams.
    - Rebelgecko 6y ago
      I'm not super familiar with how the TLS handshake and sessions work, is it straightforward to mix and match like that? e.g. tell the client that this particular webpage (like with payment info) requires TLS 1.3, but less important pages (like a movie) can fall back to TLS 1.0 or SSL or whatever?
- elithrar 6y ago
  > I noticed they didn't explicitly mention why they feel the need to ensure authentication+confidentiality+integrity for their streams, given that the data they're dealing with is films and TV shows, rather than, say, payment details. I can't speak for all of Netflix's motivations, but one of the biggest for OTT platforms is preventing MitM of your streams by middle-proxies - including ISPs. That MitM prevention encompasses data mining/user privacy, ad injection/replacement, and maintaining control over stream quality (Quality of Experience). There used to be [and still is, in some places] a lot of transcode-to-lower-bitrate-and-cache-inside-our-network behaviour, especially amongst mobile carriers. Reduced stream quality would reflect on Netflix, not the ISP who might be doing this transparently. A user switching up to 1080p or expecting 4K HDR (as appropriate) wants to get that, and Netflix [or others] want to deliver that experience as intended. The reason we see a LOT less of this now is due to how easy (and thus, widespsread) TLS became.
- sedatk 6y ago
  To prevent another Max Headroom incident? Joking aside, Netflix's web page runs on a vast variety of devices from phones to smart TV's which don't have the same security profiles. You don't want someone to be able to inject a packet to your Netflix stream to pwn your TV.
- valleyjo 6y ago
  “Defense in depth”. If you can protect an attack vector you should. Makes everything more difficult for attackers.
- rmrfstar 6y ago
  I don't think this is a new feature? I run an out-of-band network tap and have seen only TLS 1.3 traffic for some time now.
  - deleted 6y ago
    [deleted]
saagarjha 6y ago
It’s interesting to see daily variation in play delay; is this just related to less network congestion during certain hours?
- jakub_g 6y ago
  You have different audience mix around the world (and even within each country) at different times of the day. At certain hours it's more mobile, other hours it's more desktop; more US or more Asia etc. (unclear what data is shown on the graph though)
- barbegal 6y ago
  Yes specifically queuing delay. It is unlikely that any significant proportion of packets are being retried but both servers and network infrastructure will have slightly longer queues.
erk__ 6y ago
I wonder if this is also done in-kernel on their edge machines. I know that they used to do it in kernel with ktls(?) on FreeBSD which they themselves are a developer of.
FpUser 6y ago
I am curious why would I care about what do they do with their streams as long as I get adequate playback quality.
jiggawatts 6y ago
Great. My cooking shows are encrypted to stop the communists learning my recipes. Now, can I watch the shows that I paid for in 4K on my PC? No? Come back when the advertised 4K is actual 4K.
- mimsee 6y ago
  How's it not 4K? You need to use Windows 10 Creator update or newer and the Edge browser (most likely the old one and not the newer Chromium based). You need an Intel 7th generation CPU (i3, i5, or i7) or an Nvidia GPU.[1][2] [1]: https://help.netflix.com/en/node/23931 https://help.netflix.com/en/node/23931 [2]: https://nvidia.custhelp.com/app/answers/detail/a_id/4583/~/4k-uhd-netflix-content-on-nvidia-gpus https://nvidia.custhelp.com/app/answers/detail/a_id/4583/~/4...
  - Tomdarkness 6y ago
    Despite having all of those requirements I still can't get 4k streams on my PC. All I've managed to find are rumours from unofficial sources that there's some issue with HDCP 2.2 when using displayport with certain monitors. Either way it's extremely convoluted. In fact it's vastly easier just to pirate the 4k version. I don't feel bad about doing this because I already pay for netflix.
  - michaelt 6y ago
    > How's it not 4K? Many people report Netflix's 4k is very heavily compressed compared to 4k blu-ray, making it high quality in name only. I wouldn't know, because Netflix won't serve 4k content to any of my devices.
    - cbg0 6y ago
      > Many people report Netflix's 4k is very heavily compressed compared to 4k blu-ray, making it high quality in name only. This is quite true, and it is to be expected from a streaming service that wants to deliver 4K video over most Internet connections. They stream 4K at around 15mbps to my LG TV, meanwhile a blu-ray's video stream would go above 50mbps typically. There is definitely also a cost component to take into account when you have over 160M paying accounts to deliver content to.
__s 6y ago
Can someone explain why this thread is full of people not caring about security? This article even goes over how TLS 1.3 is a perf improvement Have the anti privacy crowd come out in droves now that we have a public desire for contact tracing & there's a desire to scapegoat why netflix et al have reduced stream quality due to increased load? & HTTP is not an option. I for one enjoy my ISP not being able to inject ads into my streams
- solinent 6y ago
  We don't allow our mail to be seized and read without warrant, why should our other data be susceptible? Netflix already had a secure system with their original business model of mailing you the content. Finally, allowing advertisers to profile us based on our browsing data is one step before allowing the police to do so as well. Maybe it's the same step.
  - tedunangst 6y ago
    But what about interdiction? What if the CIA swaps the directors cut dvd for the theatrical version and you never learn deckard was a replicant?
    - solinent 6y ago
      Good point. I think there's always a certain element of trust here--the law is more important than the implementation sometimes.
- saurik 6y ago
  I don't really see many people on this post arguing against encryption entirely... and the article isn't vs HTTP, it is about an upgrade to a new version of TLS; so it kind of feels like you are just cherrypicking a few comments in order to hit at a rather fascinating strawman agenda / conspiracy theory: is there even a single comment here--other than yours ;P--poking at Netflix over reduced stream quality due to the pandemic?
  - 6y ago
stuff4ben 6y ago
While I really liked that they're moving to fully secure TLS 1.3, I was more impressed by their A/B testing. I find it fascinating they can segment their traffic by device AND then be able to deliver separate streams of content. It's something I've wanted to do at many companies I've worked at but we never had the data to begin with in order to do that. Always another fire to put out...
- deleted 6y ago
  [deleted]
gok 6y ago
This is cool, and I'm glad to see that A/B testing confirms that TLS 1.3 is actually faster at startup in the real world. That said, I find most annoying Netflix perf problems are in the middle of a stream, not the start. I wonder how much they're getting hit by TCP congestion control troubles, and if QUIC/HTTP3 might help.
- toast0 6y ago
  If TCP congestion control is an issue for Netflix, it would be addressable on the server side, and I'm guessing they would address it in the FreeBSD kernel. Looking through the congestion control directory, there's not a lot of changes coming from Netflix; although, the TCP_STATS one [1] looks interesting, and some of the others indicate they're probably doing some testing (bug fixes and code cleanup). Switching from TCP to TCP over UDP in order to run congestion control (and other things, such as segmentation) in userspace instead of kernel space makes more sense if you don't control the kernel; but Netflix controls the kernel on the content appliances. Netflix has also put a lot of effort into moving more of the processing into the kernel to avoid copying data and context switches between the kernel and userspace; moving congestion control into userspace would most likely have a large negative impact on their serving bandwidth per appliance. [1] https://reviews.freebsd.org/D20655 https://reviews.freebsd.org/D20655
vinay_ys 6y ago
It is interesting to see the experiment results with 7%-8% improvements in media rebuffers with just moving from 1-RTT to 0-RTT. Wonder if they ever considered having only one layer of encryption instead of two (TLS and DRM). Would that save more CPU and hence avoid media rebuffers lot more?
- anchpop 6y ago
  DRM is required by the people they licence video from. I imagine that they wouldn't do it if they didn't have to. (They might still do it to their own content but I don't see what they gain by doing it for someone else's)
  - Rebelgecko 6y ago
    IIRC, Amazon Prime actually uses less obnoxious DRM on their own content than for 3rd party stuff. At least, when I had a computer configuration that they said was unsupported for HD due to DRM reasons, I was still able to get Amazon's own videos in 1080p
  - fosefx 6y ago
    By now I have only heared about (and experienced) the downsites of DRM. Are there any good arguments for it / casestudies that show any reduction in piracy?
    - the_pwner224 6y ago
      https://news.ycombinator.com/item?id=15305476 https://news.ycombinator.com/item?id=15305476 https://news.ycombinator.com/item?id=15319476 https://news.ycombinator.com/item?id=15319476
- Thaxll 6y ago
  But those 2 layers are at different level, one is for content the other for the HTTP protocol.
dochtman 6y ago
TLS 1.3 has been out for a while now, and always promised performance improvements (for example, see this Cloudflare article from Sep 2016: https://blog.cloudflare.com/introducing-tls-1-3/ https://blog.cloudflare.com/introducing-tls-1-3/). I wonder why Netflix is only getting to it now? Even if some clients didn't support it yet, seems like they could still introduce support on the server side?
- shanemhansen 6y ago
  Just guessing: it could be related to the fact that they have been reported to use an unusual setup (kernel TLS on BSD). https://2019.eurobsdcon.org/slides/Kernel%20TLS%20and%20TLS%20hardware%20offload%20-%20Drew%20Gallatin.pdf https://2019.eurobsdcon.org/slides/Kernel%20TLS%20and%20TLS%...
ncmncm 6y ago
The elephant in the room is Youtube, which supports only (when last I checked) TLS 1.0. My OpenSSL is configured to fail connections below 1.3, but evidently Firefox bypasses that with its own TLS. If they support TLS at all, why not 1.3? Do they have some dodgy hack that avoids a performance cost in 1.3 that is not in 1.0? Or is it just sloth?
- syntheticcorp 6y ago
  That’s not the case now. I see a TLSv1.3 connection to the googlevideo.com server using Firefox, and a QUIC connection in Chrome
thanksforfish 6y ago
Is there a good website listing all the sites that don't use HTTPS or use really old TLS versions? I'm not seeing one. As an example, the NVIDIA driver download pages switch to http:// http:// for the actual downloads although the rest of the site is HTTPS... It's interesting to see which companies have difficulty keeping up.
- 1123581321 6y ago
  Keep in mind such a list wouldn’t necessarily be accurate, especially since proxies like Cloudflare can accept 1.0 and 1.1 connections and convert them to 1.2+ Between the proxy and the server.
  - thanksforfish 6y ago
    Oh sure, you'd need to do some research. But for sites that openly use http or old TLS, it would be obvious. I wonder what the top 1M would look like.

308 points by caution 6y ago

I noticed they didn't explicitly mention why they feel the need to ensure authentication+confidentiality+integrity for their streams, given that the data they're dealing with is films and TV shows, rather than, say, payment details. As I understand it, they use HTTPS to prevent spying and data-mining by unscrupulous ISPs. It doesn't affect their DRM at all, which would work just as well over plain HTTP.

daurnimator 6y ago
Also to prevent traffic shaping: ISPs throttle traffic that contains netflix.com in the SNI. ESNI (encrypted SNI) comes with TLS 1.3.
- cheschire 6y ago
  Ah, the good ol' radar detector reflector. There was a skit I saw a couple decades ago where a person was showing off his radar detector, then in order to combat that, the police had developed a radar detector reflector, so then he had made a radar detector reflector protector or something like that, then they made a protector detector, and so on. It was a couple minutes of explaining his best efforts to counter the police counters to his counters and on and on. But, you know, funny.
  - tbrock 6y ago
    You’re probably thinking of the movie “the big hit”. It’s trace busters: https://m.youtube.com/watch?v=Iw3G80bplTg https://m.youtube.com/watch?v=Iw3G80bplTg The trace busta busta busta.
- v7p1Qbt1im 6y ago
  Networking noob here. Can‘t the ISP see Netflix‘s ASN/commonly used IP‘s and shape that way?
  - gsnedders 6y ago
    Yes. The case where it's much more useful is traffic going in/out of major CDNs to shared IP addresses. For Netflix or someone else running their own CDN the gains are much smaller.
    - MaxBarraclough 6y ago
      So ISPs could still impose traffic shaping on Netflix if they wanted to, I take it? I can't see how Netflix could prevent this. Ultimately, their servers ('OCAs') have easily detectable IP addresses, right?
      - throwaway287391 6y ago
        Fast.com probably helps quite a bit. If ISPs throttle Netflix' traffic their fast.com measurements will look bad and customers will complain/sue. (AFAIK there's no way to distinguish between fast.com tests and actual Netflix video consumption since the former's traffic patterns are identical(?) to a Netflix video streaming client's.) Creating fast.com always seemed like a pretty brilliant move by Netflix to me.
        MaxBarraclough 6y ago
        Good point. Smaller players can't make the same move though.
        Tijdreiziger 6y ago
        I think that was the whole point of fast.com, right? Speeds to known 'speedtest' servers were manipulated by ISPs, and this was bad for Netflix's business because customers would ostensibly have good speeds yet have slow Netflix, so fast.com was created as a way to measure the 'real' speed to Netflix's servers.
- georgyo 6y ago
  ESNI was dropped from the TLS1.3 spec. It is currently a draft protocol. Almost nothing supports ESNI yet. Chrome does not have it yet. Firefox does but it very difficult to enable, there is a config flag but it does nothing on its own unless you also enable DNS over HTTP in Firefox. OpenSSL has no support for ESNI yet either. ESNI also never tells the user if it is working or not yet, making downgrades fairly easy. ESNI is a long way from being deployed, let alone useful.
  - moduspol 6y ago
    They're Netflix, though, so they've got a lot of use cases where they control both ends of the connection (e.g. they've got a native app for the end user). If it's important to them to have ESNI support, they presumably could do it for those use cases.
  - middleclick 6y ago
    CloudFlare has ESNI enabled (and so do websites hosted by it, like medium.com) so it's not as if no one is doing it.
    - georgyo 6y ago
      If no clients support it, and you can't tell as a user if it is working, then effectively no one is using it.
      - middleclick 6y ago
        Firefox supports it and Chrome has plans to support it in the future. I agree that it's not much right now but it's a start.
      - shawnz 6y ago
        Firefox supports it, although currently only as a configurable. Users can test their ESNI support online here: https://www.cloudflare.com/ssl/encrypted-sni/ https://www.cloudflare.com/ssl/encrypted-sni/
pedrocr 6y ago
> It doesn't affect their DRM at all, which would work just as well over plain HTTP. Does it work at all though? I does prevent paying customers from getting higher quality streams. But aren't all the shows still widely available in torrents at whatever quality you want?
baq 6y ago
from the summary: > From the field test, we are confident that TLS 1.3 provides us a better streaming experience. so they actually measured an improvement in performance metrics.
- MaxBarraclough 6y ago
  That's a performance comparison against TLS 1.2. It doesn't speak to why they're concerned about authentication+confidentiality+integrity for audio-video streams in the first place.
  - CydeWeys 6y ago
    Are you hinting at something here, some threat maybe, that you think Netflix is responding to? Or do you just not think that more security is a good thing generally and should be done for its own sake?
    - MaxBarraclough 6y ago
      It's possible to distribute large binary blobs over unencrypted HTTP, and to provide security using hash checks. I believe apt no longer does this due to a flaw in their implementation that would not have been an issue if they'd used HTTPS. I'm unsure if Steam still uses insecure HTTP, but they certainly used to. [0] [1] I'm all for HTTPS everywhere, but for Netflix and YouTube, that means encrypting exabytes of data. That's going to cost them quite a bit. Seems reasonable to go for more detail than It's good practice in general. > Are you hinting at something here No, as I put in my first comment, I think we know the specific reasons (untrustworthy ISPs), it's just interesting they didn't mention them. [0] https://whydoesaptnotusehttps.com/ https://whydoesaptnotusehttps.com/ [1] https://developer.valvesoftware.com/wiki/SteamPipe#Server_administrators https://developer.valvesoftware.com/wiki/SteamPipe#Server_ad...
      - CydeWeys 6y ago
        Given that hardware encrypting/decrypting is omnipresent, is it really that much more expensive to stream content securely? I'd like to see the numbers, but I just don't think it's actually a meaningful increase in cost. Network traffic itself tends to be the dominating factor in cost, with encryption just upping your electric bill slightly. It seems like the benefits of transmitting your streaming securely (e.g. defeating interloping ISPs) more than exceed the costs, hence why they're doing it.
Klathmon 6y ago
>I noticed they didn't explicitly mention why they feel the need to ensure authentication+confidentiality+integrity for their streams, given that the data they're dealing with is films and TV shows, rather than, say, payment details. Because secure should be the default where we can do it. If I'm watching a show, I don't necessarily want every single router along the way knowing what show i'm watching. Either because of advertising profiles or just because I don't want people knowing that I'm binge watching a specific show, or to go to an extreme, perhaps because what I'm watching isn't legal where I live. If I'm watching a documentary or news show, I don't want the possibility that someone could modify my stream and change parts of it. Misinformation is a big deal, and if we have a simple solution to get rid of whole attack vectors (especially before they become common), why not use it? And TLS gets rid of other attacks as well. token stealing is a lot harder if EVERYTHING is TLS encrypted vs if only some things are. It can also secure against vulnerabilities in the software that reads the streams. They no longer have to harden their systems as much against potentially hostile data, they can be behind well tested TLS termination that only allow verified data through to the "backend". Everything should be encrypted in transit, zero exceptions. Just because you don't think it's important to hide doesn't mean that others don't. Just because you think your parsing is secure doesn't mean that another layer isn't useful for defense in depth. We have the technology. It's free, cheap to run, and doesn't hurt the user experience in any significant way, so why not use it?
- saurik 6y ago
  > If I'm watching a show, I don't necessarily want every single router along the way knowing what show i'm watching. Unless Netflix has also started padding their compressed video segments to a deterministic size, TLS doesn't really help with the "I don't want people to know what video I'm watching" problem, as the encrypted media segments are first compressed using a variable bitrate encoding that causes a unique pattern in their resulting chunk sizes; and so, with a little database of signatures--apparently even just at the TCP level, without caring at all about the HTTP and TLS layers... which surprised even me, and I'm quite cynical about this stuff--you can pretty rapidly fingerprint what movie someone is watching from a passive traffic dump. https://dl.acm.org/doi/10.1145/3029806.3029821 https://dl.acm.org/doi/10.1145/3029806.3029821 > Identifying HTTPS-Protected Netflix Videos in Real-Time (2017) > After more than a year of research and development, Netflix recently upgraded their infrastructure to provide HTTPS encryption of video streams in order to protect the privacy of their viewers. Despite this upgrade, we demonstrate that it is possible to accurately identify Netflix videos from passive traffic capture in real-time with very limited hardware requirements. Specifically, we developed a system that can report the Netflix video being delivered by a TCP connection using only the information provided by TCP/IP headers. > To support our analysis, we created a fingerprint database comprised of 42,027 Netflix videos. Given this collection of fingerprints, we show that our system can differentiate between videos with greater than 99.99% accuracy. Moreover, when tested against 200 random 20-minute video streams, our system identified 99.5% of the videos with the majority of the identifications occurring less than two and a half minutes into the video stream. Note that this exact same kind of attack has been applied successfully to other places where a pattern of sizes can differentiate usage, including: Google Maps tiles (which are either variably-compressed fixed-dimension images or variably-dense fixed-region vectors), web pages (which are different sizes already but also link to images and other resources that are random sizes, which helps create the fingerprint), and non-personalized type-head search queries (where the result for each letter brings up some unknown set of titles and URLs; as you type each character the sequence of sizes for your intermediate results can expose the query, potentially even without needing any model of the likely keywords). Maps: https://ioactive.com/ssl-traffic-analysis-on-google-maps/ https://ioactive.com/ssl-traffic-analysis-on-google-maps/ Search: https://eprint.iacr.org/2014/959.pdf https://eprint.iacr.org/2014/959.pdf (edit: I just realized it might be interesting for me to note that I am the lead developer of a new VPN protocol called Orchid and have "attempting to solve as many of these issues as generically and yet with as little required overhead as I can" on my todo list; to be very clear: I don't handle any of this stuff yet at all, but should get to it in the next few months... while the company behind Orchid that is paying me to work on the protocol talks a lot about itself already, I'm really intending to do my own "launch" of the product once I solve some of the stability issues and finish my traffic analysis mitigations.)
tialaramex 6y ago
https://tools.ietf.org/html/bcp188 https://tools.ietf.org/html/bcp188 Pervasive Monitoring Is an Attack
londons_explore 6y ago
ISPs can still datamine which shows each user is watching simply based on the bursts of data as shows have fast moving or slow moving scenes. There are only a few tens of thousands of shows on Netflix, so it ought to be easy to identify.
- CydeWeys 6y ago
  Has anyone demonstrated that this attack is actually practical? It's super interesting. One way to defeat it, that is already happening to some extent (though not for this reason), is buffering. Maintain a large enough local buffer, and fill it at a constant bitrate, rather than runrate, and you lose this attack vector, no?
  - bostik 6y ago
    The attack is known from VoIP systems: https://www.researchgate.net/publication/321888273_Classification_of_VoIP_traffic_based_on_analysis_of_VBR_codecs https://www.researchgate.net/publication/321888273_Classific... Adapting the research to other non-CBR streams doesn't sound far-fetched.
    - CydeWeys 6y ago
      VoIP is a completely different use case though because it needs to be real-time. A latency of even 100ms is noticeable. By contrast, Netflix is streaming pre-recorded content, so any latency is not perceived at all so long as it is smaller than the buffer size. E.g. I could have a 5 minute buffer and suffer a 4 minute Internet outage and never so much as notice, whereas that would obliterate any VoIP use case.
      - bostik 6y ago
        I agree it's not exactly the same. I would also love to inhabit the parallel universe where Netflix hands out a 4-minute buffer, because in this one 20 seconds seems to be on the high end. However, I strongly suspect that Netflix's video and audio are sent in different streams. Occasionally video for some titles is missing but audio gets through, confusing our daughter to no end. So while you can't infer individual syllables from the audio stream (as you would from VoIP), the audio streams should have varying enough size and chunking characteristics that allow to identify them.
Spooky23 6y ago
ISPs compete with them. They need to think of Comcast wanting to degrade the experience of watching a show that is also available on Peacock.
derivativethrow 6y ago
To give you an answer focused on the crypto itself, because the way they explained the TLS properties doesn't make this clear: 1. Integrity is redundant with authentication, so really you could say they're ensuring confidentiality + authentication. You can't authenticate a thing without implicitly obtaining assurance of integrity. It's a strictly stronger property. 2. Confidentiality is (usually) insecure and unreliable without authentication. Without authentication you have no PKI for a key exchange to symmetric encryption, so you can't even do TLS in the first place. And if you don't have a carefully applied MAC or a native AEAD mode, your symmetric mode isn't that secure either. So really what you're asking reduces to the question of why they need the most sophisticated TLS scheme for encrypting their streams. If they want the most secure TLS scheme for confidentiality, TLS 1.3 is the way to do it. They explained one particular facet of why this is the case, re: perfect forward secrecy.
Ambroos 6y ago
In browsers at least, DRM doesn't work if your full page doesn't use HTTPS. The encrypted media extensions (EME) are only available on secure contexts, which means the page needs to be HTTPS. Segments are typically fetched with standard XHR/fetch, which means they need to use HTTPS on secure origins as well. You really don't want to have to deal with mixed content. Unrelated to DRM, Apple's ATS on iOS kinda requires developers to only use HTTPS origins in their apps. At that point, you might as well just do HTTPS everywhere.
- MaxBarraclough 6y ago
  > encrypted media extensions (EME) are only available on secure contexts, which means the page needs to be HTTPS I didn't know that, but you're right. [0] There was a time Netflix didn't use HTTPS. [1] I guess they must not have supported HTML5 at that time, and must have been using Silverlight for in-browser viewing. [0] https://www.w3.org/TR/encrypted-media/#privacy-secureorigin https://www.w3.org/TR/encrypted-media/#privacy-secureorigin [1] https://arstechnica.com/information-technology/2015/04/it-wasnt-easy-but-netflix-will-soon-use-https-to-secure-video-streams/ https://arstechnica.com/information-technology/2015/04/it-wa...
Rebelgecko 6y ago
>given that the data they're dealing with is films and TV shows, rather than, say, payment details. I definitely have had to enter payment details on Netflix.com before
- MaxBarraclough 6y ago
  We're talking about why they encrypt their media streams.
  - Rebelgecko 6y ago
    I'm not super familiar with how the TLS handshake and sessions work, is it straightforward to mix and match like that? e.g. tell the client that this particular webpage (like with payment info) requires TLS 1.3, but less important pages (like a movie) can fall back to TLS 1.0 or SSL or whatever?
elithrar 6y ago
> I noticed they didn't explicitly mention why they feel the need to ensure authentication+confidentiality+integrity for their streams, given that the data they're dealing with is films and TV shows, rather than, say, payment details. I can't speak for all of Netflix's motivations, but one of the biggest for OTT platforms is preventing MitM of your streams by middle-proxies - including ISPs. That MitM prevention encompasses data mining/user privacy, ad injection/replacement, and maintaining control over stream quality (Quality of Experience). There used to be [and still is, in some places] a lot of transcode-to-lower-bitrate-and-cache-inside-our-network behaviour, especially amongst mobile carriers. Reduced stream quality would reflect on Netflix, not the ISP who might be doing this transparently. A user switching up to 1080p or expecting 4K HDR (as appropriate) wants to get that, and Netflix [or others] want to deliver that experience as intended. The reason we see a LOT less of this now is due to how easy (and thus, widespsread) TLS became.
sedatk 6y ago
To prevent another Max Headroom incident? Joking aside, Netflix's web page runs on a vast variety of devices from phones to smart TV's which don't have the same security profiles. You don't want someone to be able to inject a packet to your Netflix stream to pwn your TV.
valleyjo 6y ago
“Defense in depth”. If you can protect an attack vector you should. Makes everything more difficult for attackers.
rmrfstar 6y ago
I don't think this is a new feature? I run an out-of-band network tap and have seen only TLS 1.3 traffic for some time now.
- deleted 6y ago
  [deleted]

saagarjha 6y ago

It’s interesting to see daily variation in play delay; is this just related to less network congestion during certain hours?

jakub_g 6y ago
You have different audience mix around the world (and even within each country) at different times of the day. At certain hours it's more mobile, other hours it's more desktop; more US or more Asia etc. (unclear what data is shown on the graph though)
barbegal 6y ago
Yes specifically queuing delay. It is unlikely that any significant proportion of packets are being retried but both servers and network infrastructure will have slightly longer queues.

erk__ 6y ago

I wonder if this is also done in-kernel on their edge machines. I know that they used to do it in kernel with ktls(?) on FreeBSD which they themselves are a developer of.

FpUser 6y ago

I am curious why would I care about what do they do with their streams as long as I get adequate playback quality.

jiggawatts 6y ago

Great. My cooking shows are encrypted to stop the communists learning my recipes. Now, can I watch the shows that I paid for in 4K on my PC? No? Come back when the advertised 4K is actual 4K.

mimsee 6y ago
How's it not 4K? You need to use Windows 10 Creator update or newer and the Edge browser (most likely the old one and not the newer Chromium based). You need an Intel 7th generation CPU (i3, i5, or i7) or an Nvidia GPU.[1][2] [1]: https://help.netflix.com/en/node/23931 https://help.netflix.com/en/node/23931 [2]: https://nvidia.custhelp.com/app/answers/detail/a_id/4583/~/4k-uhd-netflix-content-on-nvidia-gpus https://nvidia.custhelp.com/app/answers/detail/a_id/4583/~/4...
- Tomdarkness 6y ago
  Despite having all of those requirements I still can't get 4k streams on my PC. All I've managed to find are rumours from unofficial sources that there's some issue with HDCP 2.2 when using displayport with certain monitors. Either way it's extremely convoluted. In fact it's vastly easier just to pirate the 4k version. I don't feel bad about doing this because I already pay for netflix.
- michaelt 6y ago
  > How's it not 4K? Many people report Netflix's 4k is very heavily compressed compared to 4k blu-ray, making it high quality in name only. I wouldn't know, because Netflix won't serve 4k content to any of my devices.
  - cbg0 6y ago
    > Many people report Netflix's 4k is very heavily compressed compared to 4k blu-ray, making it high quality in name only. This is quite true, and it is to be expected from a streaming service that wants to deliver 4K video over most Internet connections. They stream 4K at around 15mbps to my LG TV, meanwhile a blu-ray's video stream would go above 50mbps typically. There is definitely also a cost component to take into account when you have over 160M paying accounts to deliver content to.

__s 6y ago

Can someone explain why this thread is full of people not caring about security? This article even goes over how TLS 1.3 is a perf improvement Have the anti privacy crowd come out in droves now that we have a public desire for contact tracing & there's a desire to scapegoat why netflix et al have reduced stream quality due to increased load? & HTTP is not an option. I for one enjoy my ISP not being able to inject ads into my streams

solinent 6y ago
We don't allow our mail to be seized and read without warrant, why should our other data be susceptible? Netflix already had a secure system with their original business model of mailing you the content. Finally, allowing advertisers to profile us based on our browsing data is one step before allowing the police to do so as well. Maybe it's the same step.
- tedunangst 6y ago
  But what about interdiction? What if the CIA swaps the directors cut dvd for the theatrical version and you never learn deckard was a replicant?
  - solinent 6y ago
    Good point. I think there's always a certain element of trust here--the law is more important than the implementation sometimes.
saurik 6y ago
I don't really see many people on this post arguing against encryption entirely... and the article isn't vs HTTP, it is about an upgrade to a new version of TLS; so it kind of feels like you are just cherrypicking a few comments in order to hit at a rather fascinating strawman agenda / conspiracy theory: is there even a single comment here--other than yours ;P--poking at Netflix over reduced stream quality due to the pandemic?
- 6y ago

stuff4ben 6y ago

While I really liked that they're moving to fully secure TLS 1.3, I was more impressed by their A/B testing. I find it fascinating they can segment their traffic by device AND then be able to deliver separate streams of content. It's something I've wanted to do at many companies I've worked at but we never had the data to begin with in order to do that. Always another fire to put out...

deleted 6y ago
[deleted]

gok 6y ago

This is cool, and I'm glad to see that A/B testing confirms that TLS 1.3 is actually faster at startup in the real world. That said, I find most annoying Netflix perf problems are in the middle of a stream, not the start. I wonder how much they're getting hit by TCP congestion control troubles, and if QUIC/HTTP3 might help.

toast0 6y ago
If TCP congestion control is an issue for Netflix, it would be addressable on the server side, and I'm guessing they would address it in the FreeBSD kernel. Looking through the congestion control directory, there's not a lot of changes coming from Netflix; although, the TCP_STATS one [1] looks interesting, and some of the others indicate they're probably doing some testing (bug fixes and code cleanup). Switching from TCP to TCP over UDP in order to run congestion control (and other things, such as segmentation) in userspace instead of kernel space makes more sense if you don't control the kernel; but Netflix controls the kernel on the content appliances. Netflix has also put a lot of effort into moving more of the processing into the kernel to avoid copying data and context switches between the kernel and userspace; moving congestion control into userspace would most likely have a large negative impact on their serving bandwidth per appliance. [1] https://reviews.freebsd.org/D20655 https://reviews.freebsd.org/D20655

vinay_ys 6y ago

It is interesting to see the experiment results with 7%-8% improvements in media rebuffers with just moving from 1-RTT to 0-RTT. Wonder if they ever considered having only one layer of encryption instead of two (TLS and DRM). Would that save more CPU and hence avoid media rebuffers lot more?

anchpop 6y ago
DRM is required by the people they licence video from. I imagine that they wouldn't do it if they didn't have to. (They might still do it to their own content but I don't see what they gain by doing it for someone else's)
- Rebelgecko 6y ago
  IIRC, Amazon Prime actually uses less obnoxious DRM on their own content than for 3rd party stuff. At least, when I had a computer configuration that they said was unsupported for HD due to DRM reasons, I was still able to get Amazon's own videos in 1080p
- fosefx 6y ago
  By now I have only heared about (and experienced) the downsites of DRM. Are there any good arguments for it / casestudies that show any reduction in piracy?
  - the_pwner224 6y ago
    https://news.ycombinator.com/item?id=15305476 https://news.ycombinator.com/item?id=15305476 https://news.ycombinator.com/item?id=15319476 https://news.ycombinator.com/item?id=15319476
Thaxll 6y ago
But those 2 layers are at different level, one is for content the other for the HTTP protocol.

dochtman 6y ago

TLS 1.3 has been out for a while now, and always promised performance improvements (for example, see this Cloudflare article from Sep 2016: https://blog.cloudflare.com/introducing-tls-1-3/ https://blog.cloudflare.com/introducing-tls-1-3/). I wonder why Netflix is only getting to it now? Even if some clients didn't support it yet, seems like they could still introduce support on the server side?

shanemhansen 6y ago
Just guessing: it could be related to the fact that they have been reported to use an unusual setup (kernel TLS on BSD). https://2019.eurobsdcon.org/slides/Kernel%20TLS%20and%20TLS%20hardware%20offload%20-%20Drew%20Gallatin.pdf https://2019.eurobsdcon.org/slides/Kernel%20TLS%20and%20TLS%...

ncmncm 6y ago

The elephant in the room is Youtube, which supports only (when last I checked) TLS 1.0. My OpenSSL is configured to fail connections below 1.3, but evidently Firefox bypasses that with its own TLS. If they support TLS at all, why not 1.3? Do they have some dodgy hack that avoids a performance cost in 1.3 that is not in 1.0? Or is it just sloth?

syntheticcorp 6y ago
That’s not the case now. I see a TLSv1.3 connection to the googlevideo.com server using Firefox, and a QUIC connection in Chrome

thanksforfish 6y ago

Is there a good website listing all the sites that don't use HTTPS or use really old TLS versions? I'm not seeing one. As an example, the NVIDIA driver download pages switch to http:// http:// for the actual downloads although the rest of the site is HTTPS... It's interesting to see which companies have difficulty keeping up.

1123581321 6y ago
Keep in mind such a list wouldn’t necessarily be accurate, especially since proxies like Cloudflare can accept 1.0 and 1.1 connections and convert them to 1.2+ Between the proxy and the server.
- thanksforfish 6y ago
  Oh sure, you'd need to do some research. But for sites that openly use http or old TLS, it would be obvious. I wonder what the top 1M would look like.