7 ms·

How useful are hardware decoders? My understanding is that they are not necessarily faster than software decoders (in my experience it is the opposite), so the

by cm2187 3y ago

How useful are hardware decoders? My understanding is that they are not necessarily faster than software decoders (in my experience it is the opposite), so the only benefit I can see is less CPU consumption. I see the benefit when you run on a battery (laptop / smartphone) but on a desktop or server, unless you critically need the CPU at that time, I am not sure how useful it is.

fulafel 3y ago
Yes, real-time decoding on a desktop is just nice-to-have, with a couple of exceptions (1) if your decoder software runs too slow for real-time decoding. I think it historically has happened every once in a while until CPUs and decoder optimizations catch up again. Eg 4k H.265 was too much for many desktops for a while. (2) you do a lot of it, in which low power decoding->saving electricity is still good eg Linux web browsers for ages had sw-only decoding at it worked mostly fine, just laptops having battery eaten in browser apps & electron based apps (videoconferencing). As codecs are historically a top source of remote code execution vulnerabilities that get our device pwned, the poor sandboxability of hw decoders makes them a security problem as well. eg as a pathway to exploit the flaky and high-privileged GPU driver code paths.
Vogtinator 3y ago
That's exactly what makes them useful. Energy consumption is vastly reduced.
FirmwareBurner 3y ago
>I see the benefit when you run on a battery (laptop / smartphone) but on a desktop or server, unless you critically need the CPU at that time, I am not sure how useful it is. Yes it is. Very useful. The vast majority of people use laptops now, not desktop computers with 120mm tower coolers anymore, and if you want to switch to Linux then not having video decode nuke your battery life and turn your tiny fans into a Pratt & Whitney, then this is another welcome change to closing the gap to Windows. People, especially on the Linux side, need to stop viewing the world as if everyone still has desktop towers PCs tucked under their desks while disregarding the software issues and deficiencies that only plague laptops, which are the majority of the marketshare and have vastly different constrains than desktops.
- Dalewyn 3y ago
  Software video decoding isn't some monstrous workload, though. We're not in the era of Pentium 4 and h.264 decoding drowning every CPU out there to a screeching halt anymore, the CPUs of today hardly acknowledge the load even in the low-end tiers. The benefits of hardware decoding are lower power consumption (read: electricity bills) and offloading the work to dedicated co-processors freeing up resources on the CPU proper, which are indeed far less relevant on desktops and even laptops which can just pack big batteries. The biggest beneficiaries are by far mobile devices and really low power hardware. Computers which either cannot practically carry more batteries or which use hardware that simply aren't performant for more important reasons. Desktops and laptops meanwhile are better geared to benefit from the perks of software decoding: Better quality and versatility, and fine-tuning of filters and rendering paths.
  - FirmwareBurner 3y ago
    With all due respect, I think you must be out of touch if you don't acknowledge the negative effects of lacking HW decode on video playback and content consumption on regular notebooks. People do notice this even if they don't know where the fault comes from. Even on a modern Ryzen laptop APU, if I run H265 or AV1 4k decode in software the CPU usage is vastly noticeable. On my older Intel i5 6300HQ they're pegging it at 100% .
    - Dalewyn 3y ago
      I watch software decoded video all the time because I like watching anime. I have never, ever had any problems for quite literally over a decade thanks to how powerful CPUs have gotten. Way back 20 years ago, you would have had a point. But today? CPUs are several orders of magnitudes of overpowered for the task.
      - FirmwareBurner 3y ago
        >. I have never, ever had any problems for quite literally over a decade thanks to how powerful CPUs have gotten. I wasn't talking about having problems decoding, I was talking about software decode screwing your battery life due to high CPU usage on certain streams. Which is a problem, but not something most people who don't have laptops care about. Do you watch your anime on a desktop or a laptop on battery?
ddalex 3y ago
I can tell you from my Chromebook and Macbook experience, HW decoders and encoders are the difference between a productive remote video meetup with good collaboration, and a frustrating lagged glitchy suttering meet with no work done, if CPU only.
Havoc 3y ago
They're usually substantially faster. Many low end devices can't do decode on cpu at all well enough to say play YT. Add hw decoder and magic. Re desktop use - even there the energy efficiency is a win, especially for continuous use. e.g. I've always got YT open
kiririn 3y ago
Software decoding 4K x265 requires insane resources. You could live transcode it to 1080p quite reasonably on 1 average core, but simply decoding it requires ~3 average cores
- ddalex 3y ago
  how come decoding is more work intensive vs transcoding ?
  - magicalhippo 3y ago
    With JPEG you can scale down in power-of-two factors by just not processing the higher-frequency components. You're still left with the non-lossy compression step, so one could do something like "unzip", repackage data throwing away the high-frequency stuff, "zip". Wouldn't surprise me if you can do something similar with h264/h265.
    - zamadatix 3y ago
      I'd be surprised if that worked for interframe compression like video but even if it still did, and extremely well, encoding to the resulting format still has to fit in and that's where the majority of the work should be even when you just decoded the source normally.
  - smolder 3y ago
    I think they mean transcoding as an additional step upon already decoded frames.
rollcat 3y ago
For encoding, it's like riding a bus vs pushing a bus. AWS sells a $100k 1u encoder box to broadcast companies. They buy that because there's hardly a 2nd best to speak of. Encoding 15min of 8k source material on i9 9900k (that was 2019 and THE fastest CPU on the market) took me 19h. Same for decoding, any mobile device will drink battery like it's water on Sahara. Video is unusable without HW acceleration.
- cm2187 3y ago
  Encoding is very different. Most codecs are designed to be slow to encode, fast to decode. HW encoding is day and night vs software. But I see ffmpeg mentioned in the post. Switching to hardware decoding is likely to slow down your pipeline.
  - bdhcuidbebe 3y ago
    > Switching to hardware decoding is likely to slow down your pipeline. If thats your experience, then something is off with your setup. Hardware decoding on a modern computer might seem equally fast as software until you look at power usage where hw decode is a hard win. Even so, the resources used to decode modern high end codecs like x265 or av1 will be major and hard to miss. but anyway discussed are encoding &decoding. And video encoding goes from like 3 fps in software to 100s of fps in hardware, making it possible at all with any resolution and quality to speak of. You need encoding to stream your video, like in a video meeting.
    - cm2187 3y ago
      Maybe I am doing it wrong. But I tried the following commands to test it on h264/h265 and 1080p/4k sources: Soft: ffmpeg -i {src} -map 0:v -f null - Hard: ffmpeg -hwaccel cuda -i {src} -map 0:v -f null - the result I get is: Source Hard Soft 1080p-h264 25.1x 37.1x 1080p-h265 26.4x 36.5x 4k-h264 5.5x 7.1x 4k-h265 5.5x 6.5x this is on an old CPU (i7-6700K) and a somewhat recent GPU (RTX 2060) and with ffmpeg 6
      - 3y ago
BLanen 3y ago
Sorry if I'm misunderstanding your comment's implication. But this is not hardware decoding. This is software decoding on the GPU. NVIDIA for example has specialized (en/de)-coding hardware on their GPUs, which is different. And even that works together with their GPGPU Cuda cores.
- bno1 3y ago
  No, the entire point of those vulkan extensions is to use the hardware-accelerated video processing units. You can use computer shaders for software decoding on the GPU, there is no need for special extensions to do that. AMD and Intel have specialized encoding/decoding hw too.
- vetinari 3y ago
  All three big vendors (Nvidia, AMD, Intel) and also bunch of ARM SoC vendors do have specialized blocks in their graphic solution whose only purpose is to decode and encode specific codec profiles. It is hardware decoding, not decoding on GPU.

by cm2187 3y ago

fulafel 3y ago

Yes, real-time decoding on a desktop is just nice-to-have, with a couple of exceptions (1) if your decoder software runs too slow for real-time decoding. I think it historically has happened every once in a while until CPUs and decoder optimizations catch up again. Eg 4k H.265 was too much for many desktops for a while. (2) you do a lot of it, in which low power decoding->saving electricity is still good eg Linux web browsers for ages had sw-only decoding at it worked mostly fine, just laptops having battery eaten in browser apps & electron based apps (videoconferencing). As codecs are historically a top source of remote code execution vulnerabilities that get our device pwned, the poor sandboxability of hw decoders makes them a security problem as well. eg as a pathway to exploit the flaky and high-privileged GPU driver code paths.

Vogtinator 3y ago

That's exactly what makes them useful. Energy consumption is vastly reduced.

FirmwareBurner 3y ago

>I see the benefit when you run on a battery (laptop / smartphone) but on a desktop or server, unless you critically need the CPU at that time, I am not sure how useful it is. Yes it is. Very useful. The vast majority of people use laptops now, not desktop computers with 120mm tower coolers anymore, and if you want to switch to Linux then not having video decode nuke your battery life and turn your tiny fans into a Pratt & Whitney, then this is another welcome change to closing the gap to Windows. People, especially on the Linux side, need to stop viewing the world as if everyone still has desktop towers PCs tucked under their desks while disregarding the software issues and deficiencies that only plague laptops, which are the majority of the marketshare and have vastly different constrains than desktops.

Dalewyn 3y ago
Software video decoding isn't some monstrous workload, though. We're not in the era of Pentium 4 and h.264 decoding drowning every CPU out there to a screeching halt anymore, the CPUs of today hardly acknowledge the load even in the low-end tiers. The benefits of hardware decoding are lower power consumption (read: electricity bills) and offloading the work to dedicated co-processors freeing up resources on the CPU proper, which are indeed far less relevant on desktops and even laptops which can just pack big batteries. The biggest beneficiaries are by far mobile devices and really low power hardware. Computers which either cannot practically carry more batteries or which use hardware that simply aren't performant for more important reasons. Desktops and laptops meanwhile are better geared to benefit from the perks of software decoding: Better quality and versatility, and fine-tuning of filters and rendering paths.
- FirmwareBurner 3y ago
  With all due respect, I think you must be out of touch if you don't acknowledge the negative effects of lacking HW decode on video playback and content consumption on regular notebooks. People do notice this even if they don't know where the fault comes from. Even on a modern Ryzen laptop APU, if I run H265 or AV1 4k decode in software the CPU usage is vastly noticeable. On my older Intel i5 6300HQ they're pegging it at 100% .
  - Dalewyn 3y ago
    I watch software decoded video all the time because I like watching anime. I have never, ever had any problems for quite literally over a decade thanks to how powerful CPUs have gotten. Way back 20 years ago, you would have had a point. But today? CPUs are several orders of magnitudes of overpowered for the task.
    - FirmwareBurner 3y ago
      >. I have never, ever had any problems for quite literally over a decade thanks to how powerful CPUs have gotten. I wasn't talking about having problems decoding, I was talking about software decode screwing your battery life due to high CPU usage on certain streams. Which is a problem, but not something most people who don't have laptops care about. Do you watch your anime on a desktop or a laptop on battery?

ddalex 3y ago

I can tell you from my Chromebook and Macbook experience, HW decoders and encoders are the difference between a productive remote video meetup with good collaboration, and a frustrating lagged glitchy suttering meet with no work done, if CPU only.

Havoc 3y ago

They're usually substantially faster. Many low end devices can't do decode on cpu at all well enough to say play YT. Add hw decoder and magic. Re desktop use - even there the energy efficiency is a win, especially for continuous use. e.g. I've always got YT open

kiririn 3y ago

Software decoding 4K x265 requires insane resources. You could live transcode it to 1080p quite reasonably on 1 average core, but simply decoding it requires ~3 average cores

ddalex 3y ago
how come decoding is more work intensive vs transcoding ?
- magicalhippo 3y ago
  With JPEG you can scale down in power-of-two factors by just not processing the higher-frequency components. You're still left with the non-lossy compression step, so one could do something like "unzip", repackage data throwing away the high-frequency stuff, "zip". Wouldn't surprise me if you can do something similar with h264/h265.
  - zamadatix 3y ago
    I'd be surprised if that worked for interframe compression like video but even if it still did, and extremely well, encoding to the resulting format still has to fit in and that's where the majority of the work should be even when you just decoded the source normally.
- smolder 3y ago
  I think they mean transcoding as an additional step upon already decoded frames.

rollcat 3y ago

For encoding, it's like riding a bus vs pushing a bus. AWS sells a $100k 1u encoder box to broadcast companies. They buy that because there's hardly a 2nd best to speak of. Encoding 15min of 8k source material on i9 9900k (that was 2019 and THE fastest CPU on the market) took me 19h. Same for decoding, any mobile device will drink battery like it's water on Sahara. Video is unusable without HW acceleration.

cm2187 3y ago
Encoding is very different. Most codecs are designed to be slow to encode, fast to decode. HW encoding is day and night vs software. But I see ffmpeg mentioned in the post. Switching to hardware decoding is likely to slow down your pipeline.
- bdhcuidbebe 3y ago
  > Switching to hardware decoding is likely to slow down your pipeline. If thats your experience, then something is off with your setup. Hardware decoding on a modern computer might seem equally fast as software until you look at power usage where hw decode is a hard win. Even so, the resources used to decode modern high end codecs like x265 or av1 will be major and hard to miss. but anyway discussed are encoding &decoding. And video encoding goes from like 3 fps in software to 100s of fps in hardware, making it possible at all with any resolution and quality to speak of. You need encoding to stream your video, like in a video meeting.
  - cm2187 3y ago
    Maybe I am doing it wrong. But I tried the following commands to test it on h264/h265 and 1080p/4k sources: Soft: ffmpeg -i {src} -map 0:v -f null - Hard: ffmpeg -hwaccel cuda -i {src} -map 0:v -f null - the result I get is: Source Hard Soft 1080p-h264 25.1x 37.1x 1080p-h265 26.4x 36.5x 4k-h264 5.5x 7.1x 4k-h265 5.5x 6.5x this is on an old CPU (i7-6700K) and a somewhat recent GPU (RTX 2060) and with ffmpeg 6
    - 3y ago

BLanen 3y ago

Sorry if I'm misunderstanding your comment's implication. But this is not hardware decoding. This is software decoding on the GPU. NVIDIA for example has specialized (en/de)-coding hardware on their GPUs, which is different. And even that works together with their GPGPU Cuda cores.

bno1 3y ago
No, the entire point of those vulkan extensions is to use the hardware-accelerated video processing units. You can use computer shaders for software decoding on the GPU, there is no need for special extensions to do that. AMD and Intel have specialized encoding/decoding hw too.
vetinari 3y ago
All three big vendors (Nvidia, AMD, Intel) and also bunch of ARM SoC vendors do have specialized blocks in their graphic solution whose only purpose is to decode and encode specific codec profiles. It is hardware decoding, not decoding on GPU.