4 ms·

Companies that caused HEVC patent mess had it coming. So I understand AV1 is great for streaming content when you have a lot of encoder computing resources. W

Companies that caused HEVC patent mess had it coming. So I understand AV1 is great for streaming content when you have a lot of encoder computing resources. What about other video codec application areas? Like real-time peer-to-peer low latency streaming (<50 ms codec latency)? Like for remote desktops, remote gaming, video conferencing, etc. What rules the roost in that department? More specifically, which codec would have the best image quality for low-latency 400-4000 kbit/s (== ~50-500 kB/s) 720p/1080p streaming for different computing resources (like say for these cases: ARM A53, high-end ARM (like phone flagship models), mobile GPU, modern x86 chip and discrete GPU)? h.265 with B-frames disabled? Or something else? What can achieve the lowest latency while maintaining low bandwidth requirements? (So not including obvious bandwidth-hogging techniques like streaming jpegs or [codec x] I-frames.)

incognito65423 9y ago
Streaming of computer screens is quite different compared to real life footage. UIs contain very shallow gradients (thus you want 10bit+), lots of flat areas which are easy to entropy-encode without any frequency domain quantization at all (so a modern codec that has passthrough blocks) and font rendering engines use subpixel anti-aliasing so you want 4:4:4 chroma. And that annoying effect where image quality refines gradually after large-scale changes on a screen (not enough bits allocated to an I-frame?). Screencasting is basically quite a different beast compared to live-action or even animation footage. At a minimum you need higher profiles of a modern codec and tweak the encoder settings for low latency AND different bit allocation compared to the usual.
- vardump 9y ago
  > Streaming of computer screens is quite different compared to real life footage. UIs contain very shallow gradients (thus you want 10bit+), lots of flat areas which are easy to entropy-encode without any frequency domain quantization at all... Well, you can get pretty good results with h.264 without B-frames. Modern composited desktops have caused traditional changed block (RLE, LZ) lossless codecs to perform badly. > font rendering engines use subpixel anti-aliasing so you want 4:4:4 chroma I think you can just gradually stream more (chroma) bits to stationary areas containing high detail (frequency) information, like text. In other words, tunable bit allocation between motion and stationary detail. No matter what, there's no way to know what's the subpixel layout on the receiving end. Luckily Retina/HiDPI displays are increasingly lessening importance of subpixel rendering.
  - incognito65423 9y ago
    The thing is that 4:2:0 chroma is the default in most codecs because the main profiles only support that. You couldn't stream full chroma video on twitch even if you wanted to. And subsampled chroma looks terrible on colored text, edges between flat colored areas and similar things.[0] So 4:4:4 is a must, only once you have that you can think about adaptive quantization for chroma. [0] https://upload.wikimedia.org/wikipedia/commons/1/1c/420-progressive-still.png https://upload.wikimedia.org/wikipedia/commons/1/1c/420-prog...
    - vardump 9y ago
      I agree with that, but 4:4:4 also means there's twice as much data. I think you should be able to get away with using chroma subsampling for moving portions while allocating bits in data stream for stationary areas, without subsampling (by using some heuristics, perhaps.) Chroma from luma techniques might be interesting in this context. https://people.xiph.org/~xiphmont/demo/daala/demo4.shtml https://people.xiph.org/~xiphmont/demo/daala/demo4.shtml
      - clouddrover 9y ago
        Chroma from luma happens to work particularly well for video game content. It's one of the reasons Twitch is interested in using AV1 as soon as possible. Here's a talk by Thomas Daede where he discusses chroma from luma: https://www.youtube.com/watch?v=yKEDf5-2sT4 https://www.youtube.com/watch?v=yKEDf5-2sT4 Slides: https://people.xiph.org/~tdaede/demuxed_av1_2017.pdf https://people.xiph.org/~tdaede/demuxed_av1_2017.pdf Twitch is also interested in AV1 for its s-frames feature which allows for better compression in low latency live streaming: https://www.youtube.com/watch?v=o5sJX6VA34o https://www.youtube.com/watch?v=o5sJX6VA34o https://www.youtube.com/watch?v=AeSNW2T7QwE https://www.youtube.com/watch?v=AeSNW2T7QwE
      - brigade 9y ago
        4:4:4 doesn’t mean twice the compressed bitrate, however. First, chroma residual already takes a minority of bits - luma residual and prediction info take way more bits and those are unchanged going to 4:4:4. Second, quantizing chroma more so there’s the same number of chroma bits as there would have been in 4:4:4 is approximately equivalent as subsampling chroma in the first place (okay, its actually slightly worse, so yeah you need maybe 1% greater bitrate to match quality with 4:2:0 with modern codecs on sequences for which 4:2:0 subsampling is transparent.)
genpfault 9y ago
x264 -tune zerolatency?
- vardump 9y ago
  Isn't that just h.264 with B-frames disabled plus some x264 implementation specific tweaks? h.264 with disabled B-frames is more or less what is typically currently used for low latency streaming.
  - sp332 9y ago
    x264 –slice-max-size <A> –vbv-maxrate <B> –vbv-bufsize <C> –crf <D> –intra-refresh –tune zerolatency There's some pretty crazy stuff going on under the hood. Here's an old blog post by the x264 lead developer: https://web.archive.org/web/20150507012544/http://x264dev.multimedia.cx:80/archives/249 https://web.archive.org/web/20150507012544/http://x264dev.mu... Periodic Intra Refresh completely eliminates the concept of keyframes
vardump 9y ago
Sorry for replying to my own post. AV1 encoder has a latency allowance parameter, which you can set to zero. Something like av1enc ... lag-in-frames=0 Obviously h.265 has same low latency option like h.264, in form of not using B-frames. The question remains, which one performs the best? Is AV1 with lag-in-frames=0 actually usable in practical applications with the available hardware? How much quality suffers from not being able to predict from future frames in comparison to other codecs? Anyways I really like the ability to specify balance between latency and quality in AV1 and VP9.
cbhl 9y ago
> So I understand AV1 is great for streaming content when you have a lot of encoder computing resources. > What about other video codec application areas? Having lots of compute means you can run the algorithm in a software encoder on general-purpose hardware. This is slow/expensive, but it means you can deploy the encoder relatively quickly - say, weeks - to start getting benefits of saved bandwidth. If you need to do encoding in a low-latency/real-time or low-power environment, basically you need to use hardware accelerated encoding/decoding. That means waiting years (sometimes decades, depending on the industry -- say, broadcast TV) to deploy the new codec. First the bitstream has to be frozen, then the hardware folks need to develop a hardware implementation, and then enough users need to buy devices with a new enough GPU or chip.
snvzz 9y ago
>when you have a lot of encoder computing resources. There's no reason to think that, after the final spec is published, nobody will work on an optimized encoder.

incognito65423 9y ago

Streaming of computer screens is quite different compared to real life footage. UIs contain very shallow gradients (thus you want 10bit+), lots of flat areas which are easy to entropy-encode without any frequency domain quantization at all (so a modern codec that has passthrough blocks) and font rendering engines use subpixel anti-aliasing so you want 4:4:4 chroma. And that annoying effect where image quality refines gradually after large-scale changes on a screen (not enough bits allocated to an I-frame?). Screencasting is basically quite a different beast compared to live-action or even animation footage. At a minimum you need higher profiles of a modern codec and tweak the encoder settings for low latency AND different bit allocation compared to the usual.

vardump 9y ago
> Streaming of computer screens is quite different compared to real life footage. UIs contain very shallow gradients (thus you want 10bit+), lots of flat areas which are easy to entropy-encode without any frequency domain quantization at all... Well, you can get pretty good results with h.264 without B-frames. Modern composited desktops have caused traditional changed block (RLE, LZ) lossless codecs to perform badly. > font rendering engines use subpixel anti-aliasing so you want 4:4:4 chroma I think you can just gradually stream more (chroma) bits to stationary areas containing high detail (frequency) information, like text. In other words, tunable bit allocation between motion and stationary detail. No matter what, there's no way to know what's the subpixel layout on the receiving end. Luckily Retina/HiDPI displays are increasingly lessening importance of subpixel rendering.
- incognito65423 9y ago
  The thing is that 4:2:0 chroma is the default in most codecs because the main profiles only support that. You couldn't stream full chroma video on twitch even if you wanted to. And subsampled chroma looks terrible on colored text, edges between flat colored areas and similar things.[0] So 4:4:4 is a must, only once you have that you can think about adaptive quantization for chroma. [0] https://upload.wikimedia.org/wikipedia/commons/1/1c/420-progressive-still.png https://upload.wikimedia.org/wikipedia/commons/1/1c/420-prog...
  - vardump 9y ago
    I agree with that, but 4:4:4 also means there's twice as much data. I think you should be able to get away with using chroma subsampling for moving portions while allocating bits in data stream for stationary areas, without subsampling (by using some heuristics, perhaps.) Chroma from luma techniques might be interesting in this context. https://people.xiph.org/~xiphmont/demo/daala/demo4.shtml https://people.xiph.org/~xiphmont/demo/daala/demo4.shtml
    - clouddrover 9y ago
      Chroma from luma happens to work particularly well for video game content. It's one of the reasons Twitch is interested in using AV1 as soon as possible. Here's a talk by Thomas Daede where he discusses chroma from luma: https://www.youtube.com/watch?v=yKEDf5-2sT4 https://www.youtube.com/watch?v=yKEDf5-2sT4 Slides: https://people.xiph.org/~tdaede/demuxed_av1_2017.pdf https://people.xiph.org/~tdaede/demuxed_av1_2017.pdf Twitch is also interested in AV1 for its s-frames feature which allows for better compression in low latency live streaming: https://www.youtube.com/watch?v=o5sJX6VA34o https://www.youtube.com/watch?v=o5sJX6VA34o https://www.youtube.com/watch?v=AeSNW2T7QwE https://www.youtube.com/watch?v=AeSNW2T7QwE
    - brigade 9y ago
      4:4:4 doesn’t mean twice the compressed bitrate, however. First, chroma residual already takes a minority of bits - luma residual and prediction info take way more bits and those are unchanged going to 4:4:4. Second, quantizing chroma more so there’s the same number of chroma bits as there would have been in 4:4:4 is approximately equivalent as subsampling chroma in the first place (okay, its actually slightly worse, so yeah you need maybe 1% greater bitrate to match quality with 4:2:0 with modern codecs on sequences for which 4:2:0 subsampling is transparent.)

genpfault 9y ago

x264 -tune zerolatency?

vardump 9y ago
Isn't that just h.264 with B-frames disabled plus some x264 implementation specific tweaks? h.264 with disabled B-frames is more or less what is typically currently used for low latency streaming.
- sp332 9y ago
  x264 –slice-max-size <A> –vbv-maxrate <B> –vbv-bufsize <C> –crf <D> –intra-refresh –tune zerolatency There's some pretty crazy stuff going on under the hood. Here's an old blog post by the x264 lead developer: https://web.archive.org/web/20150507012544/http://x264dev.multimedia.cx:80/archives/249 https://web.archive.org/web/20150507012544/http://x264dev.mu... Periodic Intra Refresh completely eliminates the concept of keyframes

vardump 9y ago

Sorry for replying to my own post. AV1 encoder has a latency allowance parameter, which you can set to zero. Something like av1enc ... lag-in-frames=0 Obviously h.265 has same low latency option like h.264, in form of not using B-frames. The question remains, which one performs the best? Is AV1 with lag-in-frames=0 actually usable in practical applications with the available hardware? How much quality suffers from not being able to predict from future frames in comparison to other codecs? Anyways I really like the ability to specify balance between latency and quality in AV1 and VP9.

cbhl 9y ago

> So I understand AV1 is great for streaming content when you have a lot of encoder computing resources. > What about other video codec application areas? Having lots of compute means you can run the algorithm in a software encoder on general-purpose hardware. This is slow/expensive, but it means you can deploy the encoder relatively quickly - say, weeks - to start getting benefits of saved bandwidth. If you need to do encoding in a low-latency/real-time or low-power environment, basically you need to use hardware accelerated encoding/decoding. That means waiting years (sometimes decades, depending on the industry -- say, broadcast TV) to deploy the new codec. First the bitstream has to be frozen, then the hardware folks need to develop a hardware implementation, and then enough users need to buy devices with a new enough GPU or chip.

snvzz 9y ago

>when you have a lot of encoder computing resources. There's no reason to think that, after the final spec is published, nobody will work on an optimized encoder.