3 ms·

What is the use case? Okay, ultra low latency streaming. That is good. But. If you are sending the frames via some protocol over the network, like WebRTC, it wi

What is the use case? Okay, ultra low latency streaming. That is good. But. If you are sending the frames via some protocol over the network, like WebRTC, it will be touching the CPU anyway. Software encoding of 4K h264 is real time on a single thread on 65w, decade old CPUs, with low latency. The CPU encoders are much better quality and more flexible. So it's very difficult to justify the level of complexity needed for hardware video encoding. Absolutely no need for it for TV streaming for example. But people keep being obsessed with it who have no need for it. IMO vendors should stop reinventing hardware video encoding and instead assign the programmer time to making libwebrtc and libvpx better suit their particular use case.

eptcyka 7mo ago
It will be more energy efficient. And the CPU is free to jit half a gig of javascript in the mean time.
- temp0826 7mo ago
  It's hugely more efficient, if you're on a battery powered device it could mean hours more of play time. It's pretty insane just how much better it is (I go through a bit of extra effort to make sure it's working for me, hw decoding isn't includes in some distros).
xattt 7mo ago
It’s a leftover mindset from the mid-2000s when GPGPU became possible, and additional performance was “unlocked” from an otherwise under-utilized silicon.
jpc0 7mo ago
I'm not entirely sure that this is true. I haven't actually looked into this but it might not be the realm of possibility. But you are generating a frame on GPU, if you can also encode it there, either with nvenc or vulkan doesn't matter. Then DMA the to the nic while just using the CPU to process the packet headers, assuming that cannot also be handled in the GPU/nic
- nerdsniper 7mo ago
  You can also often DMA video coming in through peripherals to get it straight into the GPU, skipping the CPU.
chillfox 7mo ago
The article explains it. This is not for streaming over the web, but for editing professional grade video on consumer hardware.
- doctorpangloss 7mo ago
  davinci resolve is the only commercial NLE with any kind of vulkan support, and it is experimental prores decodes faster than realtime single threaded on a decade old CPU too it doesn't make sense. it's much different with say, a video game, where a texture will be loaded once into VRAM, and then yes, all the work will be done on the GPU. a video will have CPU IO every frame, you are still doing a ton of CPU work. i don't know why people are talking about power efficiency, in a pro editing context, your CPU will be very, very busy with these IO threads, including and especially in ffmpeg with hardware encoding/decoding nonetheless. it doesn't look anything like a video game workload which is what this stack is designed for.
  - lostmsu 7mo ago
    That reduces power consumption. So should improve battery life of laptops and help environment a little.
    - doctorpangloss 7mo ago
      How do you figure? Power consumption will be higher. Decompressing on GPU with shaders uses the highest power state and every engine (the copy, compute and graphics for your NLE). The CPU will be running at full speed for IO and all the other NLE tasks. Using the GPU might be more efficient, but it will not matter to decode faster than realtime, which is my whole point.
      - lostmsu 7mo ago
        > Decompressing on GPU with shaders uses the highest power state Huh?
  - pandaforce 7mo ago
pandaforce 7mo ago
The article explicitly mentions that mainstream codecs like H264 are not the target. This is for very high bitrate high resolution professional codecs.
nerdsniper 7mo ago
> If you are sending the frames via some protocol over the network, like WebRTC, it will be touching the CPU anyway. Software encoding of 4K h264 is real time on a single thread on 65w, decade old CPUs, with low latency. This is valid for a single stream, but the equation changes when you're trying to squeeze the highest # of simultaneous streams into the least amount of CapEx possible. Sure, you still have to transfer it to the CPU cache just before you send it over WebRTC/HTTP/whatever, but you unlock a lot of capacity by using all the rest of the silicon as much as you can. Being able to use a budget/midrange GPU instead of a high-end ultra-high-core-count CPU could make a big difference to a business with the right use-case. That said, TFA doesn't seem to be targeting that kind of high stream density use-case either. I don't think e.g. Frigate NVR users are going to switch to any of the mentioned technologies in this blog post.
- doctorpangloss 7mo ago
  > This is valid for a single stream, but the equation changes when you're trying to squeeze the highest # of simultaneous streams into the least amount of CapEx possible... Being able to use a budget/midrange GPU instead of a high-end ultra-high-core-count CPU could make a big difference to a business with the right use-case. okay well, the $5, 9 year old i5-9500T i benchmarked has 6 cores, and 1080p H264 decode is 2x realtime per thread, it would support 12 realtime streams. the scenario you are talking about, with frigate NVR users, well that thing supports a lot of ways to display many thumbnails simultaneously very cheaply - a human being can't even see more than ~3K resolution on a typical display at typical viewing distance - and it is pretty acceptable to drop frames, because the user isn't going to be able to comprehend the simultaneous, realtime video of more than 6 camera feeds. i think you are proving my point that, this is all a bunch of overengineering. the ancient intel chip has QSV which will work just fine. nobody needs GPUs to be doing this work.
hrmtst93837 7mo ago
[flagged]

eptcyka 7mo ago

It will be more energy efficient. And the CPU is free to jit half a gig of javascript in the mean time.

temp0826 7mo ago
It's hugely more efficient, if you're on a battery powered device it could mean hours more of play time. It's pretty insane just how much better it is (I go through a bit of extra effort to make sure it's working for me, hw decoding isn't includes in some distros).

xattt 7mo ago

It’s a leftover mindset from the mid-2000s when GPGPU became possible, and additional performance was “unlocked” from an otherwise under-utilized silicon.

jpc0 7mo ago

I'm not entirely sure that this is true. I haven't actually looked into this but it might not be the realm of possibility. But you are generating a frame on GPU, if you can also encode it there, either with nvenc or vulkan doesn't matter. Then DMA the to the nic while just using the CPU to process the packet headers, assuming that cannot also be handled in the GPU/nic

nerdsniper 7mo ago
You can also often DMA video coming in through peripherals to get it straight into the GPU, skipping the CPU.

chillfox 7mo ago

The article explains it. This is not for streaming over the web, but for editing professional grade video on consumer hardware.

doctorpangloss 7mo ago
davinci resolve is the only commercial NLE with any kind of vulkan support, and it is experimental prores decodes faster than realtime single threaded on a decade old CPU too it doesn't make sense. it's much different with say, a video game, where a texture will be loaded once into VRAM, and then yes, all the work will be done on the GPU. a video will have CPU IO every frame, you are still doing a ton of CPU work. i don't know why people are talking about power efficiency, in a pro editing context, your CPU will be very, very busy with these IO threads, including and especially in ffmpeg with hardware encoding/decoding nonetheless. it doesn't look anything like a video game workload which is what this stack is designed for.
- lostmsu 7mo ago
  That reduces power consumption. So should improve battery life of laptops and help environment a little.
  - doctorpangloss 7mo ago
    How do you figure? Power consumption will be higher. Decompressing on GPU with shaders uses the highest power state and every engine (the copy, compute and graphics for your NLE). The CPU will be running at full speed for IO and all the other NLE tasks. Using the GPU might be more efficient, but it will not matter to decode faster than realtime, which is my whole point.
    - lostmsu 7mo ago
      > Decompressing on GPU with shaders uses the highest power state Huh?
- pandaforce 7mo ago

pandaforce 7mo ago

The article explicitly mentions that mainstream codecs like H264 are not the target. This is for very high bitrate high resolution professional codecs.

nerdsniper 7mo ago

> If you are sending the frames via some protocol over the network, like WebRTC, it will be touching the CPU anyway. Software encoding of 4K h264 is real time on a single thread on 65w, decade old CPUs, with low latency. This is valid for a single stream, but the equation changes when you're trying to squeeze the highest # of simultaneous streams into the least amount of CapEx possible. Sure, you still have to transfer it to the CPU cache just before you send it over WebRTC/HTTP/whatever, but you unlock a lot of capacity by using all the rest of the silicon as much as you can. Being able to use a budget/midrange GPU instead of a high-end ultra-high-core-count CPU could make a big difference to a business with the right use-case. That said, TFA doesn't seem to be targeting that kind of high stream density use-case either. I don't think e.g. Frigate NVR users are going to switch to any of the mentioned technologies in this blog post.

doctorpangloss 7mo ago
> This is valid for a single stream, but the equation changes when you're trying to squeeze the highest # of simultaneous streams into the least amount of CapEx possible... Being able to use a budget/midrange GPU instead of a high-end ultra-high-core-count CPU could make a big difference to a business with the right use-case. okay well, the $5, 9 year old i5-9500T i benchmarked has 6 cores, and 1080p H264 decode is 2x realtime per thread, it would support 12 realtime streams. the scenario you are talking about, with frigate NVR users, well that thing supports a lot of ways to display many thumbnails simultaneously very cheaply - a human being can't even see more than ~3K resolution on a typical display at typical viewing distance - and it is pretty acceptable to drop frames, because the user isn't going to be able to comprehend the simultaneous, realtime video of more than 6 camera feeds. i think you are proving my point that, this is all a bunch of overengineering. the ancient intel chip has QSV which will work just fine. nobody needs GPUs to be doing this work.

hrmtst93837 7mo ago

[flagged]