3 ms·

In general, the legacy hardware codec deployments are more important than what some ambitious software vendors think is "better". The primary inertia of media

In general, the legacy hardware codec deployments are more important than what some ambitious software vendors think is "better". The primary inertia of media publishing markets, is content that will deliver properly on all platforms with legacy compatibility. Initially, a new software codec will grind the cpu and battery-life like its on a 20 year old phone. Then often becomes pipelined into premium GPUs for fringe users, and finally mainstreamed by mobile publishers to save quality/bandwidth when the market is viable (i.e. above 80% of users). If anyone thinks they can shortcut this process, or repeat a lock-down of the market with 1990s licensing models... than it will end badly for the project. There are decades of media content and free codecs keeping the distribution standards firmly anchored in compatibility mode. These popular choices become entrenched as old Patents expire on "good-enough" popular formats. Best of luck, =)

tambre 3y ago
Doesn't seem to much too relevant for image codecs though, no? Decoding 10s of still images on a CPU for a webpage that'll be used for minutes versus 10s of delta images in a much more complicated video coded aren't quite comparable. I don't think we have much deployment of WebP or AVIF hardware decoders yet the formats have widespread use and adoption.
- Joel_Mckay 3y ago
  WebP has problems (several unreported CVEs too), and is essentially another tga disaster under the hood. Anyone smart keeps those little monsters in a box. =) What I am seeing is the high-water-mark for WebP was 3 years ago... It is already dead as a format, but may find a niche use-case for some users. Consider built-in web-cams that have hardware h264 codecs built-in, as the main cpu just has to stream the data with 1 core. Better battery life for both the sender and receiver. Keep in mind a single web page may have hundreds of images, and mjpeg streams are still popular in machine-vision use-cases. As most media/gpu hardware is now integrated into most modern browsers, the inertia of "good-enough" will likely remain.. and become permanent as patents expire. =)
brooksbp 3y ago
How much better do you think a new codec needs to be to make it all the way to mainstream? 2x? 10x?
- Joel_Mckay 3y ago
  Better aesthetics and even 18% reduction in file sizes refuses to move the sleepy elephant (h265 is likely still fringe stage). Even a trivial codec licensing fee of $2.50 for Pi users was not very successful for many media formats (i.e. 10% of retail price kills most of the market). However, h264 support was important enough to wrap into the pi4 retail price, and even at 11m/pcs a month there is still no stock available. https://www.youtube.com/watch?v=ygU2bCx2Z3g https://www.youtube.com/watch?v=ygU2bCx2Z3g
  - brooksbp 3y ago
    I would like to think that integrating reconfigurable logic into chips will help. But, no idea if the economics makes sense. And, the ecosystem around managing that pretty much does not exist.
    - Joel_Mckay 3y ago
      ASIC are very space/energy efficient compared to fpga. The choice a chip maker has is to include popular legacy/free codecs like mp3, or pay some IP holder that won't even pick up a phone unless there is over $3m on the table. h264 was easy by comparison, but hardly ideal. NVIDIA was a miracle when you consider what they likely had to endure. =)
brucethemoose2 3y ago
You are thinking of video codecs. Is hardware avif decoding done anywhere? The only example I can think of where this is done is HEIF on iOS devices, maybe. Some cloud GPUs have jpeg decoding blocks for ingesting tons of images, but that'd not really the same thing.
- Joel_Mckay 3y ago
  Actually, right now I'm thinking about whether cold pizza is still edible. mjpeg streams are still popular in machine-vision use-cases. And you can be fairly certain your webcam has that codec built-in if the device was made in the past 15 years. Note most media/gpu hardware is now integrated into modern browsers. =)

tambre 3y ago

Doesn't seem to much too relevant for image codecs though, no? Decoding 10s of still images on a CPU for a webpage that'll be used for minutes versus 10s of delta images in a much more complicated video coded aren't quite comparable. I don't think we have much deployment of WebP or AVIF hardware decoders yet the formats have widespread use and adoption.

Joel_Mckay 3y ago
WebP has problems (several unreported CVEs too), and is essentially another tga disaster under the hood. Anyone smart keeps those little monsters in a box. =) What I am seeing is the high-water-mark for WebP was 3 years ago... It is already dead as a format, but may find a niche use-case for some users. Consider built-in web-cams that have hardware h264 codecs built-in, as the main cpu just has to stream the data with 1 core. Better battery life for both the sender and receiver. Keep in mind a single web page may have hundreds of images, and mjpeg streams are still popular in machine-vision use-cases. As most media/gpu hardware is now integrated into most modern browsers, the inertia of "good-enough" will likely remain.. and become permanent as patents expire. =)

brooksbp 3y ago

How much better do you think a new codec needs to be to make it all the way to mainstream? 2x? 10x?

Joel_Mckay 3y ago
Better aesthetics and even 18% reduction in file sizes refuses to move the sleepy elephant (h265 is likely still fringe stage). Even a trivial codec licensing fee of $2.50 for Pi users was not very successful for many media formats (i.e. 10% of retail price kills most of the market). However, h264 support was important enough to wrap into the pi4 retail price, and even at 11m/pcs a month there is still no stock available. https://www.youtube.com/watch?v=ygU2bCx2Z3g https://www.youtube.com/watch?v=ygU2bCx2Z3g
- brooksbp 3y ago
  I would like to think that integrating reconfigurable logic into chips will help. But, no idea if the economics makes sense. And, the ecosystem around managing that pretty much does not exist.
  - Joel_Mckay 3y ago
    ASIC are very space/energy efficient compared to fpga. The choice a chip maker has is to include popular legacy/free codecs like mp3, or pay some IP holder that won't even pick up a phone unless there is over $3m on the table. h264 was easy by comparison, but hardly ideal. NVIDIA was a miracle when you consider what they likely had to endure. =)

brucethemoose2 3y ago

You are thinking of video codecs. Is hardware avif decoding done anywhere? The only example I can think of where this is done is HEIF on iOS devices, maybe. Some cloud GPUs have jpeg decoding blocks for ingesting tons of images, but that'd not really the same thing.

Joel_Mckay 3y ago
Actually, right now I'm thinking about whether cold pizza is still edible. mjpeg streams are still popular in machine-vision use-cases. And you can be fairly certain your webcam has that codec built-in if the device was made in the past 15 years. Note most media/gpu hardware is now integrated into modern browsers. =)