5 ms·

I see mention of Media Source Extensions (MSE) in these release notes. Good to see they're improving it, I guess. But my understanding [1] is that while Safari

I see mention of Media Source Extensions (MSE) in these release notes. Good to see they're improving it, I guess. But my understanding [1] is that while Safari on Desktop and iPad support MSE, Safari on iPhone does not. Does anyone know why this restriction exists and if it's likely to be removed? I'd expect these to be substantially the same codebase so the restriction confuses me. I saw folks speculating about business reasons for this restriction, which confuses me. It seems like HLS provides broadly similar functionality, so I'm not sure how many determined folks are entirely stopped from doing something by Apple's technology choice, or steered toward a native app, or whatever Apple supposedly wants instead. This just makes things unnecessarily painful for small projects. [1] backed up by https://caniuse.com/mediasource https://caniuse.com/mediasource

jsmith45 4y ago
I'd put down a lot of money on this being an artificial limitation, and not a technical issue. If I had to guess, it is that they have hyper-optimized the pipeline for HLS for battery consumption. Streaming video is one of the most power hungry things people do with their phone that laymen often don't think is power hungry. (Unlike say 3d games, or GPS). If the MSE path is less optimized (and it is, simply by nature of having JavaScript in the loop handling the video segments), which drains the battery faster, but sites like Twitch, Youtube, and Facebook switch to using MSE, then now the iPhone battery life numbers look worse.
- scottlamb 4y ago
  Fair point; I can see them making the choice for that reason. But I'm not sure it's actually true that HLS is more optimal: * On the one hand, the MSE path makes all the data flow through Javascript. At least that means chunks must be garbage-collected. I can see it meaning an extra copy or two also. * On the other hand, doesn't HLS require polling and extra round trips? You fetch the .m3u8 manifest repeatedly, and then fetch the media segments it indicates. If you want lowest possible latency, well, it's 1.5 trips higher than necessary, as well as requiring extra network requests (costing battery for wake-ups and a bit of bandwidth/radio time). As compared to a WebSocket where the server pushes segments as soon as they're available. In an application of mine, I've thought about writing web worker code to basically create a HLS API for my service from the real, WebSocket-based API. Then it'd work on iPhone. Of course, this means everything would flow through Javascript, totally defeating what advantage HLS has and requiring (device-internal, fortunately) polling.
  - lstamour 4y ago
    > As compared to a WebSocket where the server pushes segments as soon as they're available. This didn’t become a requirement for LL-HLS according to this blog post, but you could build the same push technique using HLS and HTTP/2 to push data to the client before it needs to ask for it: https://www.theoplayer.com/blog/impact-of-apple-ll-hls-update-2020 https://www.theoplayer.com/blog/impact-of-apple-ll-hls-updat... (or as suggested in the post, provide hint URLs for where the next block will be…) Apple’s official docs on LL-HLS: https://developer.apple.com/documentation/http_live_streaming/enabling_low-latency_http_live_streaming_hls https://developer.apple.com/documentation/http_live_streamin... Edit: the full spec is at https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-11 https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc82... and doesn’t appear to mention push directly but does require HTTP2. https://www.akamai.com/blog/performance/-using-ll-hls-with-byte-range-addressing-to-achieve-interoperabi https://www.akamai.com/blog/performance/-using-ll-hls-with-b... Mentions more details but I don’t have time to read them now. I expect folks are looking at this very closely, but the current protocol obviously lets the client be in full control over what gets sent, which would be somewhat lost if relying exclusively on HTTP2 server-initiated push… For example, clients can request separate caches of low and high bitrate media to always ensure they have something to play back in time…
    - scottlamb 4y ago
      Oh, I missed EXT-X-PRELOAD-HINT, [1] thanks. That seems like a significant improvement, in that the client can do a hanging GET and have the server respond when the media segment exists, avoiding that transit latency. Still has to send extra requests. (I guess the silver lining there is that this is a flow control mechanism, so the server won't keep sending more data if the client isn't keeping up with previous data. My custom WebSocket protocol doesn't really do flow control right now, beyond what TCP does.) [edit: another problem: multiple simultaneous hanging GETS are really awkward with HTTP/1.1 due to connection limits. You have to shard the domain name (which causes problems with cookie auth). Or you really need to use HTTP/2 or HTTP/3.] I'm really confused by the HTTP/2 push thing that article mentioned. My general understanding of HTTP/2 push is that the clients implement it by adding the pushed data to their cache and nothing more. (Or don't implement it at all. IIRC Chrome dropped support.) If the client later needs that URL, it loads more quickly. But there's no JS API for doing something as soon as a push is received. Maybe Safari has a special hook in their HLS implementation that bypasses the polling delay if the media segment is pushed, but if so, I don't know where that's documented, and it's a surprising behavior. Or maybe they gave up on the idea? Dunno. The whole HLS spec seems arcane compared to a protocol in which the server pushes data over a WebSocket. Even MSE is more complicated than I'd like. I'd rather try WebCodecs, but only Chrome supports it AFAIK. [2] I suppose the cool kids use WebRTC for live video instead, but my use case is an NVR, where I'd like a uniform way of handling playback and live. I don't think WebRTC really does playback. [1] https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis#section-4.4.5.3 https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc82... [2] https://github.com/Fyrd/caniuse/issues/5641 https://github.com/Fyrd/caniuse/issues/5641
robertoandred 4y ago
What specifically is not supported on iPhones?
- scottlamb 4y ago
  MSE is entirely unsupported on iPhones. window.MediaSource is undefined. https://github.com/scottlamb/moonfire-nvr/issues/121 https://github.com/scottlamb/moonfire-nvr/issues/121
jchw 4y ago
They also still don’t support WebM, VP8 or 9, or Opus on iOS, to my knowledge. I’m pretty sure there is hardware support, plus desktop Safari does, but for some unknown reason, they continue to force parent encumbered codecs only in Safari iOS. Not sure if that could theoretically be related to MSE being missing, but maybe it’s in a similar vein of control.

jsmith45 4y ago

I'd put down a lot of money on this being an artificial limitation, and not a technical issue. If I had to guess, it is that they have hyper-optimized the pipeline for HLS for battery consumption. Streaming video is one of the most power hungry things people do with their phone that laymen often don't think is power hungry. (Unlike say 3d games, or GPS). If the MSE path is less optimized (and it is, simply by nature of having JavaScript in the loop handling the video segments), which drains the battery faster, but sites like Twitch, Youtube, and Facebook switch to using MSE, then now the iPhone battery life numbers look worse.

scottlamb 4y ago
Fair point; I can see them making the choice for that reason. But I'm not sure it's actually true that HLS is more optimal: * On the one hand, the MSE path makes all the data flow through Javascript. At least that means chunks must be garbage-collected. I can see it meaning an extra copy or two also. * On the other hand, doesn't HLS require polling and extra round trips? You fetch the .m3u8 manifest repeatedly, and then fetch the media segments it indicates. If you want lowest possible latency, well, it's 1.5 trips higher than necessary, as well as requiring extra network requests (costing battery for wake-ups and a bit of bandwidth/radio time). As compared to a WebSocket where the server pushes segments as soon as they're available. In an application of mine, I've thought about writing web worker code to basically create a HLS API for my service from the real, WebSocket-based API. Then it'd work on iPhone. Of course, this means everything would flow through Javascript, totally defeating what advantage HLS has and requiring (device-internal, fortunately) polling.
- lstamour 4y ago
  > As compared to a WebSocket where the server pushes segments as soon as they're available. This didn’t become a requirement for LL-HLS according to this blog post, but you could build the same push technique using HLS and HTTP/2 to push data to the client before it needs to ask for it: https://www.theoplayer.com/blog/impact-of-apple-ll-hls-update-2020 https://www.theoplayer.com/blog/impact-of-apple-ll-hls-updat... (or as suggested in the post, provide hint URLs for where the next block will be…) Apple’s official docs on LL-HLS: https://developer.apple.com/documentation/http_live_streaming/enabling_low-latency_http_live_streaming_hls https://developer.apple.com/documentation/http_live_streamin... Edit: the full spec is at https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-11 https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc82... and doesn’t appear to mention push directly but does require HTTP2. https://www.akamai.com/blog/performance/-using-ll-hls-with-byte-range-addressing-to-achieve-interoperabi https://www.akamai.com/blog/performance/-using-ll-hls-with-b... Mentions more details but I don’t have time to read them now. I expect folks are looking at this very closely, but the current protocol obviously lets the client be in full control over what gets sent, which would be somewhat lost if relying exclusively on HTTP2 server-initiated push… For example, clients can request separate caches of low and high bitrate media to always ensure they have something to play back in time…
  - scottlamb 4y ago
    Oh, I missed EXT-X-PRELOAD-HINT, [1] thanks. That seems like a significant improvement, in that the client can do a hanging GET and have the server respond when the media segment exists, avoiding that transit latency. Still has to send extra requests. (I guess the silver lining there is that this is a flow control mechanism, so the server won't keep sending more data if the client isn't keeping up with previous data. My custom WebSocket protocol doesn't really do flow control right now, beyond what TCP does.) [edit: another problem: multiple simultaneous hanging GETS are really awkward with HTTP/1.1 due to connection limits. You have to shard the domain name (which causes problems with cookie auth). Or you really need to use HTTP/2 or HTTP/3.] I'm really confused by the HTTP/2 push thing that article mentioned. My general understanding of HTTP/2 push is that the clients implement it by adding the pushed data to their cache and nothing more. (Or don't implement it at all. IIRC Chrome dropped support.) If the client later needs that URL, it loads more quickly. But there's no JS API for doing something as soon as a push is received. Maybe Safari has a special hook in their HLS implementation that bypasses the polling delay if the media segment is pushed, but if so, I don't know where that's documented, and it's a surprising behavior. Or maybe they gave up on the idea? Dunno. The whole HLS spec seems arcane compared to a protocol in which the server pushes data over a WebSocket. Even MSE is more complicated than I'd like. I'd rather try WebCodecs, but only Chrome supports it AFAIK. [2] I suppose the cool kids use WebRTC for live video instead, but my use case is an NVR, where I'd like a uniform way of handling playback and live. I don't think WebRTC really does playback. [1] https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis#section-4.4.5.3 https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc82... [2] https://github.com/Fyrd/caniuse/issues/5641 https://github.com/Fyrd/caniuse/issues/5641

robertoandred 4y ago

What specifically is not supported on iPhones?

scottlamb 4y ago
MSE is entirely unsupported on iPhones. window.MediaSource is undefined. https://github.com/scottlamb/moonfire-nvr/issues/121 https://github.com/scottlamb/moonfire-nvr/issues/121

jchw 4y ago

They also still don’t support WebM, VP8 or 9, or Opus on iOS, to my knowledge. I’m pretty sure there is hardware support, plus desktop Safari does, but for some unknown reason, they continue to force parent encumbered codecs only in Safari iOS. Not sure if that could theoretically be related to MSE being missing, but maybe it’s in a similar vein of control.