4 ms·

This is great. It brings it in line with DirectX ( https://learn.microsoft.com/en-us/windows/win32/medfound/direct3d-12-video-overview https://learn.microsoft.c

by dagmx 4y ago

This is great. It brings it in line with DirectX ( https://learn.microsoft.com/en-us/windows/win32/medfound/direct3d-12-video-overview https://learn.microsoft.com/en-us/windows/win32/medfound/dir... ) It’s quite handy to be able to render a video on to spatial objects

vlovich123 4y ago
Yeah. I was trying to do something like render a scene and encode it on-GPU before sending it to disk. Unfortunately, there’s no way to give FFMPEG video textures afaict so even if it uses hw decode, you’re copying the full pixel buffer data twice instead of the drastically smaller encoded frame once (or at least there’s no mechanism exposed that the JS ecosystem can make use of which relies on command-line ffmpeg)
- averne_ 4y ago
  There are OpenGL extensions which can import a provided GPU buffer as a texture, using those you can achieve zero-copy. For instance, with VAAPI->OpenGL you would use vaExportSurfaceHandle in conjunction with glEGLImageTargetTexture2DOES. Check out the "hwdec" mechanism in MPV: https://github.com/mpv-player/mpv/blob/master/video/out/hwdec/hwdec_vaapi.c#L264 https://github.com/mpv-player/mpv/blob/master/video/out/hwde... https://github.com/mpv-player/mpv/blob/master/video/out/hwdec/dmabuf_interop_gl.c#L253 https://github.com/mpv-player/mpv/blob/master/video/out/hwde...
  - vlovich123 4y ago
    Sure. I think the part that’s missing is that FFMPEG runs out of process and doesn’t deal in GPU buffers/textures.
    - cillian64 4y ago
      If you use ffmpeg’s libavcodec interface then you can get it to give you the decoded framebuffers as exported DRM-prime descriptors which you can turn into textures. This is how Firefox does video decode with VAAPI, using libavcodec as a wrapper. Edit: missed the part about JS ecosystem. You can move DRM prime descriptors between processes, but I assume you can’t do this from the ffmpeg CLI and would need to write your own little C wrapper around libavcodec
- goeiedaggoeie 4y ago
  As someone else said you can achieve this with openGL, I also think gstreamer can do this.
djmips 4y ago
I had a professional project I wanted to do in Linux but ended up going with Windows because of the 3D / video performance.
Const-me 4y ago
The API you have linked is too low level to my taste. When I wanted to play video and render them inside D3D12 scenes, I have used a higher-level one, IMFMediaEngine https://learn.microsoft.com/en-us/windows/win32/api/mfmediaengine/nn-mfmediaengine-imfmediaengine https://learn.microsoft.com/en-us/windows/win32/api/mfmediae... That OS-supplied high-level video player object can demux containers, decode, play audio, and copy uncompressed frames to D3D11 textures. Then, it’s relatively easy to use DXGI surface sharing API to share the resulting RGBA8 frames from D3D11 into D3D12.

by dagmx 4y ago

vlovich123 4y ago

Yeah. I was trying to do something like render a scene and encode it on-GPU before sending it to disk. Unfortunately, there’s no way to give FFMPEG video textures afaict so even if it uses hw decode, you’re copying the full pixel buffer data twice instead of the drastically smaller encoded frame once (or at least there’s no mechanism exposed that the JS ecosystem can make use of which relies on command-line ffmpeg)

averne_ 4y ago
There are OpenGL extensions which can import a provided GPU buffer as a texture, using those you can achieve zero-copy. For instance, with VAAPI->OpenGL you would use vaExportSurfaceHandle in conjunction with glEGLImageTargetTexture2DOES. Check out the "hwdec" mechanism in MPV: https://github.com/mpv-player/mpv/blob/master/video/out/hwdec/hwdec_vaapi.c#L264 https://github.com/mpv-player/mpv/blob/master/video/out/hwde... https://github.com/mpv-player/mpv/blob/master/video/out/hwdec/dmabuf_interop_gl.c#L253 https://github.com/mpv-player/mpv/blob/master/video/out/hwde...
- vlovich123 4y ago
  Sure. I think the part that’s missing is that FFMPEG runs out of process and doesn’t deal in GPU buffers/textures.
  - cillian64 4y ago
    If you use ffmpeg’s libavcodec interface then you can get it to give you the decoded framebuffers as exported DRM-prime descriptors which you can turn into textures. This is how Firefox does video decode with VAAPI, using libavcodec as a wrapper. Edit: missed the part about JS ecosystem. You can move DRM prime descriptors between processes, but I assume you can’t do this from the ffmpeg CLI and would need to write your own little C wrapper around libavcodec
goeiedaggoeie 4y ago
As someone else said you can achieve this with openGL, I also think gstreamer can do this.

djmips 4y ago

I had a professional project I wanted to do in Linux but ended up going with Windows because of the 3D / video performance.

Const-me 4y ago

The API you have linked is too low level to my taste. When I wanted to play video and render them inside D3D12 scenes, I have used a higher-level one, IMFMediaEngine https://learn.microsoft.com/en-us/windows/win32/api/mfmediaengine/nn-mfmediaengine-imfmediaengine https://learn.microsoft.com/en-us/windows/win32/api/mfmediae... That OS-supplied high-level video player object can demux containers, decode, play audio, and copy uncompressed frames to D3D11 textures. Then, it’s relatively easy to use DXGI surface sharing API to share the resulting RGBA8 frames from D3D11 into D3D12.