5 ms·

>And finally, we encounter a large issue without a good solution. In encoded videos, a key frame is a frame in the video that contains all the visual informatio

by mandis 6y ago

>And finally, we encounter a large issue without a good solution. In encoded videos, a key frame is a frame in the video that contains all the visual information needed to render itself without any additional metadata. These are much larger than normal frames, and contribute greatly to the bitrate. Ideally, there would be as a few keyframes as possible. However, when a new user starts consuming a stream, they need at least one keyframe to view the video. WebRTC solves this problem using the RTP Control Protocl (RTCP). When a new user consumes a stream, they send a Full Intra Request (FIR) to the producer. When a producer receives this request, they insert a keyframe into the stream. This keeps the bitrate low while ensuring all the users can view the stream. FFmpeg does not support RTCP. This means that the default FFmpeg settings will produce output that won’t be viewable if consumed mid-stream, at least until a key frame is received. Therefore, the parameter -force_key_frames expr:gte(t,n_forced*4) is needed, which produces a key frame every 4 seconds. in case someone was wondering why it was a bad idea

tatersolid 6y ago
H.264 and most other modern codecs support “intra refresh” to avoid this problem, at the cost of a marginally higher bitrate overall. Think of this as a “rolling keyframe slice” which marches across the screen every few seconds. http://www.chaneru.com/Roku/HLS/X264_Settings.htm#intra-refresh http://www.chaneru.com/Roku/HLS/X264_Settings.htm#intra-refr...
- dimes 6y ago
  I was not aware of this at the time of writing, but it solves a large problem we've been having. Thank you so much for pointing that out. Edit: I've just tried using intra refresh, and it works pretty well, but the key frame interval is still required.
- Dylan16807 6y ago
  I would say intra refresh solves a different problem. You still have to wait for the intra refresh to cover the frame before you can start watching properly. That takes just as long as waiting for a keyframe, and needs slightly more bytes. The benefit of intra refresh is that you avoid having any particularly large frames. If you're using a sub-second buffer, then intra refresh makes your maximum frame size much smaller without sacrificing quality. It's a godsend for getting latency down to tiny amounts. But if you have 1 or 2 seconds of buffer then it's no big deal if a keyframe is an order of magnitude bigger than other frames, and intra refresh is pointless. Also it's not really a codec thing, it's a clever encoder trick that you can do on basically anything.
  - dimes 6y ago
    Yes, this is a good point. Intra refresh does reduce variability of the bitrate, but the bitrate is still higher than it would need to be if rtcp was supported.
  - beagle3 6y ago
    You could do on most things, but it has to be part of the codec to actually work. If you aren’t aware of this scheme, you can just wait until the next key frame. With IDR there is a lot more bookkeeping to do so you can figure out when every single needed pixel has been accounted for. And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something.
    - Dylan16807 6y ago
      > And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something. I think that supports my point. It doesn't matter if the technique is explicitly listed in the codec or not. You need clients that will tolerate infinite P-frames, and they will work equally well whether you're using intra refresh techniques on h.264 or h.262 The dumbest possible renderer would have full support; it takes extra logic to get in the way and break it.
amelius 6y ago
Ok, so only a problem in live streams? (And I suppose also when seeking inside a stream)
- SahAssar 6y ago
  It's a problem when playing from a non-start, non-keyframe point (which in practice means any arbitrary point). I'm guessing that's what you meant.
- gregoriol 6y ago
  It seems to me that you can't seek with a webrtc stream, as it is at least.
  - SahAssar 6y ago
    webrtc is just a stream but you can absolutely tell the sending side to seek to a certain point. If webrtc is your TV then the sending side is your VHS. You can tell the VHS to rewind or forward, but telling your TV to do the same is impossible. It just shows you what it gets.
    - deleted 6y ago
      [deleted]
    - gregoriol 6y ago
      I meant it's not part of webrtc, but you indeed can implement a lot of things around it.
wwweston 6y ago
Thanks for the easy summary. One thing to consider: for some IRL performances, it's not uncommon that if you arrive late, you might be seated at the timing discretion of an usher. I understand digital experiences may carry different expectations, but I could see building an experience around this, perhaps starting with audio-only and maybe even a countdown to a next keyframe event (every minute?) while a "please wait to be seated" is shown.
- SirSavary 6y ago
  Incredible -- "Our digital usher is finding you a seat in the cloud" slapped on a screen might just save us months of planning and millions on infrastructure.
  - wwweston 6y ago
    If you might be hiring for any product roles with the millions saved, feel free to reach out. :)
legohead 6y ago
Still sounds better than a FIR. If you consider a big streamer with thousands of users. Users are constantly arriving and leaving, so the keyframe requests are going to be so constant that I can see keyframes being generated much more often than 4 seconds (assuming I understand it all correctly).
- dimes 6y ago
  Usually you’ll have an SFU between the users and the streamer that can limit the number of requests to one every X seconds.
pantalaimon 6y ago
I'm sure this could be implemented if someone were to sit down and implement it.
1vuio0pswjnm7 6y ago
"In order for users to watch the video, they must be able to download it in real time, so the maximum bitrate has to be lower than the slowest connection among your users." Why not just (progressive) download, watch/save, delete? IOW, playback from saved file. Better for variety of conditions, e.g., connection might be slow.
- ngold 6y ago
  Streaming really is just downloading without the saving part.
scottlamb 6y ago
(The quoted paragraph is no longer in the article. But I'm still curious about it.) > Therefore, the parameter -force_key_frames expr:gte(t,n_forced*4) is needed, which produces a key frame every 4 seconds. How often would you like it to be producing key frames? My video experience is mostly with security cameras, and the ones I've used produce an I-frame every 2 seconds by default. Their encoders don't seem to be real high-quality; sometimes there's visible pulsing where the image will get worse until the next I-frame, so I wouldn't want to increase the interval much beyond that. > WebRTC solves this problem using the RTP Control Protocl (RTCP). When a new user consumes a stream, they send a Full Intra Request (FIR) to the producer. When a producer receives this request, they insert a keyframe into the stream. This keeps the bitrate low while ensuring all the users can view the stream. I'm writing an NVR that does live streaming with HTML5 Media Source Extensions, sending one-frame .mp4 fragments over a WebSocket connection. My approach to this problem is different: when a new client connects, I send them everything since the last key frame. IIRC there's more data in the I-frame than in the (up to 2 seconds of) P-frames since then, so this seems to work pretty well. If there were only an I-frame (say) every minute, I'd probably look at that inserting a keyframe approach...there is an ONVIF command to insert a key frame IIRC.
- dimes 6y ago
  As a reference, I think Google Chrome sends a key frame every 90 seconds by default
- banana_giraffe 6y ago
  A few random videos I pulled from youtube had a key frame on average every 4.5 seconds, ranging from 0.1 to 5.5 seconds apart, seems pretty consistent regardless of type of video, at least for the few I tried. The worst I've seen in production was a poorly configured encoder that insert a keyframe exactly every 30 seconds, along with segments every 10 seconds, which surprisingly, caused some players to crash trying to find a keyframe.
zaroth 6y ago
I guess it comes down to latency requirements? I would expect where latency isn’t a huge concern, the best user experience would be to start the new receiver back at the last keyframe and fill the buffer up to “present” so they can start watching instantly, and keep a few seconds in the buffer for stability. In more latency critical streams where you still want the perception of instant video startup I suppose you would have to start at the last keyframe and then as soon as the next key frame came through you could just jump ahead.

by mandis 6y ago

tatersolid 6y ago

H.264 and most other modern codecs support “intra refresh” to avoid this problem, at the cost of a marginally higher bitrate overall. Think of this as a “rolling keyframe slice” which marches across the screen every few seconds. http://www.chaneru.com/Roku/HLS/X264_Settings.htm#intra-refresh http://www.chaneru.com/Roku/HLS/X264_Settings.htm#intra-refr...

dimes 6y ago
I was not aware of this at the time of writing, but it solves a large problem we've been having. Thank you so much for pointing that out. Edit: I've just tried using intra refresh, and it works pretty well, but the key frame interval is still required.
Dylan16807 6y ago
I would say intra refresh solves a different problem. You still have to wait for the intra refresh to cover the frame before you can start watching properly. That takes just as long as waiting for a keyframe, and needs slightly more bytes. The benefit of intra refresh is that you avoid having any particularly large frames. If you're using a sub-second buffer, then intra refresh makes your maximum frame size much smaller without sacrificing quality. It's a godsend for getting latency down to tiny amounts. But if you have 1 or 2 seconds of buffer then it's no big deal if a keyframe is an order of magnitude bigger than other frames, and intra refresh is pointless. Also it's not really a codec thing, it's a clever encoder trick that you can do on basically anything.
- dimes 6y ago
  Yes, this is a good point. Intra refresh does reduce variability of the bitrate, but the bitrate is still higher than it would need to be if rtcp was supported.
- beagle3 6y ago
  You could do on most things, but it has to be part of the codec to actually work. If you aren’t aware of this scheme, you can just wait until the next key frame. With IDR there is a lot more bookkeeping to do so you can figure out when every single needed pixel has been accounted for. And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something.
  - Dylan16807 6y ago
    > And although it is part of the standard, I’ve encountered players that don’t support it even though they are based on ffmpeg - probably because they do look for key frames to seek to or something. I think that supports my point. It doesn't matter if the technique is explicitly listed in the codec or not. You need clients that will tolerate infinite P-frames, and they will work equally well whether you're using intra refresh techniques on h.264 or h.262 The dumbest possible renderer would have full support; it takes extra logic to get in the way and break it.

amelius 6y ago

Ok, so only a problem in live streams? (And I suppose also when seeking inside a stream)

SahAssar 6y ago
It's a problem when playing from a non-start, non-keyframe point (which in practice means any arbitrary point). I'm guessing that's what you meant.
gregoriol 6y ago
It seems to me that you can't seek with a webrtc stream, as it is at least.
- SahAssar 6y ago
  webrtc is just a stream but you can absolutely tell the sending side to seek to a certain point. If webrtc is your TV then the sending side is your VHS. You can tell the VHS to rewind or forward, but telling your TV to do the same is impossible. It just shows you what it gets.
  - deleted 6y ago
    [deleted]
  - gregoriol 6y ago
    I meant it's not part of webrtc, but you indeed can implement a lot of things around it.

wwweston 6y ago

Thanks for the easy summary. One thing to consider: for some IRL performances, it's not uncommon that if you arrive late, you might be seated at the timing discretion of an usher. I understand digital experiences may carry different expectations, but I could see building an experience around this, perhaps starting with audio-only and maybe even a countdown to a next keyframe event (every minute?) while a "please wait to be seated" is shown.

SirSavary 6y ago
Incredible -- "Our digital usher is finding you a seat in the cloud" slapped on a screen might just save us months of planning and millions on infrastructure.
- wwweston 6y ago
  If you might be hiring for any product roles with the millions saved, feel free to reach out. :)

legohead 6y ago

Still sounds better than a FIR. If you consider a big streamer with thousands of users. Users are constantly arriving and leaving, so the keyframe requests are going to be so constant that I can see keyframes being generated much more often than 4 seconds (assuming I understand it all correctly).

dimes 6y ago
Usually you’ll have an SFU between the users and the streamer that can limit the number of requests to one every X seconds.

pantalaimon 6y ago

I'm sure this could be implemented if someone were to sit down and implement it.

1vuio0pswjnm7 6y ago

"In order for users to watch the video, they must be able to download it in real time, so the maximum bitrate has to be lower than the slowest connection among your users." Why not just (progressive) download, watch/save, delete? IOW, playback from saved file. Better for variety of conditions, e.g., connection might be slow.

ngold 6y ago
Streaming really is just downloading without the saving part.

scottlamb 6y ago

(The quoted paragraph is no longer in the article. But I'm still curious about it.) > Therefore, the parameter -force_key_frames expr:gte(t,n_forced*4) is needed, which produces a key frame every 4 seconds. How often would you like it to be producing key frames? My video experience is mostly with security cameras, and the ones I've used produce an I-frame every 2 seconds by default. Their encoders don't seem to be real high-quality; sometimes there's visible pulsing where the image will get worse until the next I-frame, so I wouldn't want to increase the interval much beyond that. > WebRTC solves this problem using the RTP Control Protocl (RTCP). When a new user consumes a stream, they send a Full Intra Request (FIR) to the producer. When a producer receives this request, they insert a keyframe into the stream. This keeps the bitrate low while ensuring all the users can view the stream. I'm writing an NVR that does live streaming with HTML5 Media Source Extensions, sending one-frame .mp4 fragments over a WebSocket connection. My approach to this problem is different: when a new client connects, I send them everything since the last key frame. IIRC there's more data in the I-frame than in the (up to 2 seconds of) P-frames since then, so this seems to work pretty well. If there were only an I-frame (say) every minute, I'd probably look at that inserting a keyframe approach...there is an ONVIF command to insert a key frame IIRC.

dimes 6y ago
As a reference, I think Google Chrome sends a key frame every 90 seconds by default
banana_giraffe 6y ago
A few random videos I pulled from youtube had a key frame on average every 4.5 seconds, ranging from 0.1 to 5.5 seconds apart, seems pretty consistent regardless of type of video, at least for the few I tried. The worst I've seen in production was a poorly configured encoder that insert a keyframe exactly every 30 seconds, along with segments every 10 seconds, which surprisingly, caused some players to crash trying to find a keyframe.

zaroth 6y ago

I guess it comes down to latency requirements? I would expect where latency isn’t a huge concern, the best user experience would be to start the new receiver back at the last keyframe and fill the buffer up to “present” so they can start watching instantly, and keep a few seconds in the buffer for stability. In more latency critical streams where you still want the perception of instant video startup I suppose you would have to start at the last keyframe and then as soon as the next key frame came through you could just jump ahead.