4 ms·

> From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other w

> From what I understand by reading the article, GTK uses GL to render in the GPU then copies the pixels into main memory for the compositor to mix with other windows. This seems very strange to me. It’s how things would work with wl_shm, which is the baseline pixel-pushing interface in Wayland, but AFAIU Gtk uses EGL / Mesa, which in turn uses Linux dmabufs, which is how you do hardware-accelerated rendering / DRI on Linux today in general. However, how precisely Linux dmabufs work in a DRI context is not clear to me, because the documentation is lacking, to say the least. It seems that you can ask map to dmabufs into memory, and you can create EGLSurfaces from them, but are they always mapped into CPU memory (if only kernel-side), or can they be bare GPU memory handles until the user asks to map them? I’d hope for the latter, and if so, the only thing the work discussed in the article avoids is extra GPU-side blits (video decoding buffer to window buffer to screen), which is non-negligible but not necessarily the end of the world.

rjsw 3y ago
Linux on ARM SoCs with HW video decoders that are separate to the GPU can use the V4L2 API to avoid some copying. The decoder writes a frame to a buffer that the GPU can see then you use GL to get the GPU to merge it into the framebuffer.
DonHopkins 3y ago
I misunderstood the article saying "exports the resulting texture" as meaning it exported it to CPU memory, but audidude explained how it actually works. I believe GL_TEXTURE_EXTERNAL_OES is an Android-only OpenGL extension that takes the place of some uses of DMABUF but is not as flexible and general. ChatGPT seems to know more about them, but I can't guarantee how accurate and up-to-date it is: https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0a0d8f5 https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0... The tricky bit may be teaching pytorch to accept dmabuf handles and read and write dmabuf GPU buffers. (And ffmpeg too!)

rjsw 3y ago

Linux on ARM SoCs with HW video decoders that are separate to the GPU can use the V4L2 API to avoid some copying. The decoder writes a frame to a buffer that the GPU can see then you use GL to get the GPU to merge it into the framebuffer.

DonHopkins 3y ago

I misunderstood the article saying "exports the resulting texture" as meaning it exported it to CPU memory, but audidude explained how it actually works. I believe GL_TEXTURE_EXTERNAL_OES is an Android-only OpenGL extension that takes the place of some uses of DMABUF but is not as flexible and general. ChatGPT seems to know more about them, but I can't guarantee how accurate and up-to-date it is: https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0a0d8f5 https://chat.openai.com/share/abff036b-3020-4093-a13b-86cbf0... The tricky bit may be teaching pytorch to accept dmabuf handles and read and write dmabuf GPU buffers. (And ffmpeg too!)