3 ms·

This paper is staggeringly awful. Beyond the misleading stuff in the actual meat of the paper, even just the results at the end seem to be completely made up!

This paper is staggeringly awful. Beyond the misleading stuff in the actual meat of the paper, even just the results at the end seem to be completely made up! I attempted to replicate the results and found the following to be true: 1. They used a different keyframe interval for the two encoders. 2. They cherry-picked four test clips out of the hundreds available to find the ones where libvpx did best. They did not show results for any of the most widely used test clips, as libvpx loses badly on all of them. 3. The numbers on their graphs are simply wrong. Running libvpx with their exact settings, on the exact same source, gives almost 1db worse PSNR than they reported (note: I only tested Deadline CIF). The x264 results match. Either the parameters they used don't match the ones they actually listed, or the numbers are fudged. The worst thing about all this? According to VP8 developers, this paper was already accepted into a major journal. (If anyone's interested, I can post more detail on a few of the other things wrong with this paper, such as how they compared a known-to-be-fast VP8 decoder with a known-to-be-slow H.264 decoder, or how they fudged the scale on their graphs...)

haberman 16y ago
Those are some pretty serious accusations without a whole lot of evidence or specifics to back them up. You are claiming the paper's authors lied about their numbers and "show absolutely no interest in correcting even the obvious errors." Have you actually contacted them to be sure you didn't make a mistake? If so, what did they say? I find it hard to believe that they would respond to an accusation of fudging their numbers by saying they don't care. You also have a financial interest in the continued dominance of h.264 due to the commercial licensing program for x.264 (http://mailman.videolan.org/pipermail/x264-devel/2010-July/007508.html http://mailman.videolan.org/pipermail/x264-devel/2010-July/0...). This is a conflict of interest that is relevant to your opinion about other video formats. You also have a history of making unfounded accusations. "Well, this time, it looks like [Tandberg] ran out of ideas, so they had to go use the cheat sheet: open source." (http://x264.nl/developers/Dark_Shikari/tandberg.html http://x264.nl/developers/Dark_Shikari/tandberg.html) Later: "Update: Tandberg claims they came up with the algorithm independently: to be fair, I can actually believe this to some extent." (http://x264dev.multimedia.cx/archives/589 http://x264dev.multimedia.cx/archives/589) There's no doubt that you are an expert in video encoders. That's not a license to make unsubstantiated accusations about them.
- DarkShikari 16y ago
  You are claiming the paper's authors lied about their numbers and "show absolutely no interest in correcting even the obvious errors." Have you actually contacted them to be sure you didn't make a mistake? If so, what did they say? I find it hard to believe that they would respond to an accusation of fudging their numbers by saying they don't care. I've been on IRC with them since I made that post. I have changed that part in the post though -- as even though they appear to have shown no interest, I have received information from an off-the-record source that they are actually looking into it. This is a conflict of interest that is relevant to your opinion about other video formats. We're developing xvp8 too, which we will also license. Personally, I don't care who wins, as long as the encoders are good. xvp8 has the bonus benefit that MPEG-LA doesn't demand license fees on it, so we can aim it at some markets we wouldn't otherwise be able to target. Now, yes, this does create an incentive for me to want to dislike libvpx, as they are now a competitor. You also have a history of making unfounded accusations. Is filing for a patent that covers existing open source software suddenly "not bad"? The accusation seems pretty founded to me! The only reason I retracted any of it was because people SENT DEATH THREATS to the programmer whose name was on the patent. Those are some pretty serious accusations without a whole lot of evidence or specifics to back them up. Specifics: Video tested: Deadline CIF Bitrate: 700kbps libvpx version: latest git x264 version: latest git PSNR achieved by x264 with their settings: 44.507db (roughly matches the graph). PSNR achieved by libvpx with the settings they referred to: 43.625db (definitely doesn't match the graph) There are a few possibilities here: 1) libvpx has gotten about ~0.8db worse since they did the test, since I don't have the revision number. Unlikely. 2) The settings they referred to aren't the actual settings they used. Highly likely -- but even enabling the ARNR options, as they mentioned on IRC, only brought libvpx up to about 44.0db. 3) They tweaked the settings specifically for the test. Possible, and not quite as bad as literally making up numbers, but still cheating. 4) They made up the numbers. But even if I was completely wrong on this, cherry-picking results in such an extreme way is something you do in fluff marketing pieces, not in an academic paper. The entire paper, from top to bottom, reads like marketing: it doesn't mention a single downside of any of their features.
  - haberman 16y ago
ZeroGravitas 16y ago
I'd be interested in more detail on why you consider comparing the speed of the ffmpeg H.264 and VP8 decoders to be unfair. Don't they share a lot of code, given the similarities of the formats?
- astrange 16y ago
  The H.264 decoder contains serious added complexity because H.264 supports multiple reference frames and interlacing. It was also based on the shared MPEG-1/2/4 code, even though many of those features aren't used. That results in poor cache locality and data structures with a lot of unused stuff allocated. (Also, the author is very single-minded about optimization and seems to have completely ignored all this for most of its development…) The VP8 decoder is based on the VP5/6 decoders and isn't as different from them, so it doesn't have that problem. That said, I believe they're both pretty efficient and competitive with others. By "known to be slow", he means "known to be a bit slower with the same # decoding threads than CoreAVC".

haberman 16y ago

Those are some pretty serious accusations without a whole lot of evidence or specifics to back them up. You are claiming the paper's authors lied about their numbers and "show absolutely no interest in correcting even the obvious errors." Have you actually contacted them to be sure you didn't make a mistake? If so, what did they say? I find it hard to believe that they would respond to an accusation of fudging their numbers by saying they don't care. You also have a financial interest in the continued dominance of h.264 due to the commercial licensing program for x.264 (http://mailman.videolan.org/pipermail/x264-devel/2010-July/007508.html http://mailman.videolan.org/pipermail/x264-devel/2010-July/0...). This is a conflict of interest that is relevant to your opinion about other video formats. You also have a history of making unfounded accusations. "Well, this time, it looks like [Tandberg] ran out of ideas, so they had to go use the cheat sheet: open source." (http://x264.nl/developers/Dark_Shikari/tandberg.html http://x264.nl/developers/Dark_Shikari/tandberg.html) Later: "Update: Tandberg claims they came up with the algorithm independently: to be fair, I can actually believe this to some extent." (http://x264dev.multimedia.cx/archives/589 http://x264dev.multimedia.cx/archives/589) There's no doubt that you are an expert in video encoders. That's not a license to make unsubstantiated accusations about them.

DarkShikari 16y ago
You are claiming the paper's authors lied about their numbers and "show absolutely no interest in correcting even the obvious errors." Have you actually contacted them to be sure you didn't make a mistake? If so, what did they say? I find it hard to believe that they would respond to an accusation of fudging their numbers by saying they don't care. I've been on IRC with them since I made that post. I have changed that part in the post though -- as even though they appear to have shown no interest, I have received information from an off-the-record source that they are actually looking into it. This is a conflict of interest that is relevant to your opinion about other video formats. We're developing xvp8 too, which we will also license. Personally, I don't care who wins, as long as the encoders are good. xvp8 has the bonus benefit that MPEG-LA doesn't demand license fees on it, so we can aim it at some markets we wouldn't otherwise be able to target. Now, yes, this does create an incentive for me to want to dislike libvpx, as they are now a competitor. You also have a history of making unfounded accusations. Is filing for a patent that covers existing open source software suddenly "not bad"? The accusation seems pretty founded to me! The only reason I retracted any of it was because people SENT DEATH THREATS to the programmer whose name was on the patent. Those are some pretty serious accusations without a whole lot of evidence or specifics to back them up. Specifics: Video tested: Deadline CIF Bitrate: 700kbps libvpx version: latest git x264 version: latest git PSNR achieved by x264 with their settings: 44.507db (roughly matches the graph). PSNR achieved by libvpx with the settings they referred to: 43.625db (definitely doesn't match the graph) There are a few possibilities here: 1) libvpx has gotten about ~0.8db worse since they did the test, since I don't have the revision number. Unlikely. 2) The settings they referred to aren't the actual settings they used. Highly likely -- but even enabling the ARNR options, as they mentioned on IRC, only brought libvpx up to about 44.0db. 3) They tweaked the settings specifically for the test. Possible, and not quite as bad as literally making up numbers, but still cheating. 4) They made up the numbers. But even if I was completely wrong on this, cherry-picking results in such an extreme way is something you do in fluff marketing pieces, not in an academic paper. The entire paper, from top to bottom, reads like marketing: it doesn't mention a single downside of any of their features.
- haberman 16y ago

ZeroGravitas 16y ago

I'd be interested in more detail on why you consider comparing the speed of the ffmpeg H.264 and VP8 decoders to be unfair. Don't they share a lot of code, given the similarities of the formats?

astrange 16y ago
The H.264 decoder contains serious added complexity because H.264 supports multiple reference frames and interlacing. It was also based on the shared MPEG-1/2/4 code, even though many of those features aren't used. That results in poor cache locality and data structures with a lot of unused stuff allocated. (Also, the author is very single-minded about optimization and seems to have completely ignored all this for most of its development…) The VP8 decoder is based on the VP5/6 decoders and isn't as different from them, so it doesn't have that problem. That said, I believe they're both pretty efficient and competitive with others. By "known to be slow", he means "known to be a bit slower with the same # decoding threads than CoreAVC".