5 ms·

The BBC article _is_ extremely flawed. Namely BBC bench-marked using this methodology^: The parameter “--end-usage=q” was set to force fixed QP encoding accord

by zbobet2012 7y ago

The BBC article _is_ extremely flawed. Namely BBC bench-marked using this methodology^: The parameter “--end-usage=q” was set to force fixed QP encoding according to the QPs in Table 1 and “--threads=1” was used to run the encoder in single-thread mode. The parameters “--passes=1” and “--lag-in-frames=0” were set to run AV1 in single pass mode without the possibility of looking ahead in the video sequence before encoding. Finally, the internal bit depth of the codec was set to 12 as typically used during the AV1 development. Finally, for all encoding technologies, each sequence was split into chunks of one Intra period (approximately 1 second, as defined in the RA configuration for HM and JEM), which allowed each chunk to be independently encoded in parallel. This coding configuration was adopted to reduce the overall time needed to encode with AV1, intead of encoding each 10 second sequence sequentially. The BBC essentially forced the GOP size to 1s for libaom. In every benchmark I've run where HM and libaom are _not_ run in ridiculous modes, libaom has been about 30% better bdrate. This is consistent with many, broadly reported, independent analysis such as MSU, Facebook, etc. The second BBC evaluation which at least enabled two pass (the configuration AOM was developed in) shows much larger gains^^. They still did not address the inequity in gop length. Even old codecs like h264 can see a 10 to 20% gain on sequences from longer GOPs. AV1 is far from perfect, as some of it's features are very computationally complex to implement today, while HEVC and VVC have "better known" complexity for their features. But the BBC benchmark analysis is simply not accurate. ^ https://zenodo.org/record/1494930#.XP7hdFVKgQ8 https://zenodo.org/record/1494930#.XP7hdFVKgQ8 ^^ https://www.bbc.co.uk/rd/blog/2019-05-av1-codec-streaming-processing-hevc-vvc https://www.bbc.co.uk/rd/blog/2019-05-av1-codec-streaming-pr... Comparisons: http://www.compression.ru/video/codec_comparison/hevc_2018/ http://www.compression.ru/video/codec_comparison/hevc_2018/ https://code.fb.com/video-engineering/facebook-video-adds-av1-support/ https://code.fb.com/video-engineering/facebook-video-adds-av... https://www.elecard.com/page/aom_av1_vs_hevc https://www.elecard.com/page/aom_av1_vs_hevc http://iphome.hhi.de/marpe/download/spie-2017.pdf http://iphome.hhi.de/marpe/download/spie-2017.pdf https://bitmovin.com/av1-multi-codec-dash-dataset/ https://bitmovin.com/av1-multi-codec-dash-dataset/ Edit: PS. The BBC evaluation is good for what they where benchmarking. Will AV1 today beat HM for live streaming? And the answer is, no. The Intel SVT encoder may change that soon. Libaom was not configured, or tested for that. But this is an odd benchmark to put forward as explicitly mentioned, speed is not what _either_ code base was built for.

motbob 7y ago
1) Good post, but there is no "inequity" in GOP length, is there? The paper says "for all encoding technologies, each sequence was split into chunks of one Intra period." 2) I also don't really see why this would inherently disadvantage AV1, though obviously it would dampen any efficiency gain. (Why would it dampen it? Because it's not like one encoder is going to draw keyframes way more efficiently than another encoder, and as you lower GOP size, way more of the filesize gets taken up by keyframes, proportionally). 3) One-second GOPs are pretty out there, but we do live in a world where two-second GOPs aren't that unusual, so it's not that crazy.
- zbobet2012 7y ago
  Apologies that should be "inequity caused by gop length" 1) AV1 has significant advantages in its inter compression vs HM. AV1 also has explicit tools for dealing with short GOPs that where not tested. See: https://jmvalin.ca/papers/AV1_tools.pdf https://jmvalin.ca/papers/AV1_tools.pdf for inter prediction. 2) AV1 and HM both have a different set of tools for intra prediction and as such different coding efficiency on key frames. 3) Yeah and two second gops are really bad for video quality. 4-6s is far more common except on latency sensitive applications. I'll also point out that the reported MOS scores have several oddities, like an 8mbps av1 stream being rates lower than it's own 6mbps stream.
  - motbob 7y ago
    >I'll also point out that the reported MOS scores have several oddities, like an 8mbps av1 stream being rates lower than it's own 6mbps stream. That reminds me of the old, hilariously broken ffmpeg AAC encoder where increasing the bitrate decreased the quality after a certain point, at least in my tests. So although my guess at what would be causing weird stuff like this would certainly be bad testing methodology by BBC rather than a quirk in the encoder, there's a non-zero chance that I'd be wrong.
  - microcolonel 7y ago
    Not to mention 1-second GOPs are ludicrous for a broadcast/ondemand publisher like the BBC. However, I think libaom performance was the deciding factor for this. They say right there that they chose this arrangement because it would allow them to deliver 10s chunks with reasonable latency. With SVT-AV1 and appropriate hardware, there's no reason they couldn't do 10s like they should, and in that case AV1 would probably shine.

motbob 7y ago

1) Good post, but there is no "inequity" in GOP length, is there? The paper says "for all encoding technologies, each sequence was split into chunks of one Intra period." 2) I also don't really see why this would inherently disadvantage AV1, though obviously it would dampen any efficiency gain. (Why would it dampen it? Because it's not like one encoder is going to draw keyframes way more efficiently than another encoder, and as you lower GOP size, way more of the filesize gets taken up by keyframes, proportionally). 3) One-second GOPs are pretty out there, but we do live in a world where two-second GOPs aren't that unusual, so it's not that crazy.

zbobet2012 7y ago
Apologies that should be "inequity caused by gop length" 1) AV1 has significant advantages in its inter compression vs HM. AV1 also has explicit tools for dealing with short GOPs that where not tested. See: https://jmvalin.ca/papers/AV1_tools.pdf https://jmvalin.ca/papers/AV1_tools.pdf for inter prediction. 2) AV1 and HM both have a different set of tools for intra prediction and as such different coding efficiency on key frames. 3) Yeah and two second gops are really bad for video quality. 4-6s is far more common except on latency sensitive applications. I'll also point out that the reported MOS scores have several oddities, like an 8mbps av1 stream being rates lower than it's own 6mbps stream.
- motbob 7y ago
  >I'll also point out that the reported MOS scores have several oddities, like an 8mbps av1 stream being rates lower than it's own 6mbps stream. That reminds me of the old, hilariously broken ffmpeg AAC encoder where increasing the bitrate decreased the quality after a certain point, at least in my tests. So although my guess at what would be causing weird stuff like this would certainly be bad testing methodology by BBC rather than a quirk in the encoder, there's a non-zero chance that I'd be wrong.
- microcolonel 7y ago
  Not to mention 1-second GOPs are ludicrous for a broadcast/ondemand publisher like the BBC. However, I think libaom performance was the deciding factor for this. They say right there that they chose this arrangement because it would allow them to deliver 10s chunks with reasonable latency. With SVT-AV1 and appropriate hardware, there's no reason they couldn't do 10s like they should, and in that case AV1 would probably shine.