133 ms·

Woah - a huge flash to the past when I read this. I ported Wayland to VideoCore4 (the multimedia engine in first Pi chip) back in 2011 - it was part of the Mel

Woah - a huge flash to the past when I read this. I ported Wayland to VideoCore4 (the multimedia engine in first Pi chip) back in 2011 - it was part of the Meltemi Nokia project that got cancelled the follow year - shame as it was pretty cool and had half a chance IMO. We worked with the team in Oslo on QT acceleration over GL ES and used EGL below Wayland (coupled with some magic APIs to get the VideoCore HVS to work well). Ported this to an ARM combo chip that had just the VideoCore GPU in it as well (no HVS) - it worked pretty well. Prior to this however, I made a VideoCore demo that used a PCI bridge chip (from Broadcom) and you could plug it into a Dell laptop running Ubuntu and get accelerated video decode and also X11 window scaling working at 60fps. We nearly sold this into Apple for accelerating their macbook's but IIRC, getting the macbook into low power mode whilst the video was playing on the external chip was going to be soo much work that they gave up. A even further back, I remember validating the HVS hardware block and writing the initial scaler driver for it (IIRC, scaler.c...) and made a dispman2 port for the driver. Circa 2006! Great team - one of the most enjoyable set of people to work with I've ever come across.

mycall 7y ago
If you connected with Eric Anhold and worked on this, you would help millions of Pi students and users. It is an opportunity.
- mrlambchop 7y ago
  I checked out Eric's work just - he's doing an amazing job! Much more thought put into the subject (security for GLSL being a huge topic on its own). The lack of MMU in the VC4 architecture is a huge pain point that probably sucks up most engineering cycles when it comes to arbitrary application environments using GPU resources like X11 or Wayland - when memory runs out, what to do? You can throw engineering resources at it, but engineering talent really needs access to VMCS on the VideoCore side todo any worth while work. When we ported Android to the VC4 architecture the first time (~2010), the low memory killer in Linux was subverted to understand to kill Android applications based on their use of the VideoCore GPU memory and it worked pretty well, yet it would still close the primary running app once in a while. Run monkey over Android and all hell broke loose - really tough situations to defensively code fore. For example, for CX reasons, you had to ignore some GLES errors in the primary client due to low memory, then the system had to kill another application that was using the memory, then it would kill the EGL context for the primary application so it would refresh the entire app lifecycle using an almost suspect code path inside Android. Good times! Imagine Wayload has very similar challenges for normal desktop use.
  - stefan_ 7y ago
    The open source driver doesn't use VMCS, it instead puts aside a fixed block of memory (typically 256 MiB, which I think is also a limit due to some hardware bugs) for use through the Linux Contiguous Memory Allocator (CMA) that it then draws from. VMCS only comes into the picture if you use video decode, but I think Dave Stevenson from the foundation hacked the firmware side to support importing Linux allocated memory blocks into VMCS so that you can do zero-copy decode and import into EGL (or more likely HVS, the EGL support for the formats is pretty limited). (I really liked the design of the HVS - having pretty much scripted planes is a fresh approach over similar hardware blocks that have a fixed number of planes each with own idiosyncracies and limitations)
    - mrlambchop 7y ago
      The HVS was a cool design - the only real issue if I recall was its limited line buffers meaning it was hard to determine what could be composited in real time, so we ended up always rendering to a triple buffer. It also did some amazing scaling in realtime, but this came with some crazy artifacts that were super painful to triage. The occasional "flash of bad pixels" in a live composition screen was really painful. I just remember I wrote the DPI and CPI peripherals that we brought up the HVS with - on a 4MHz FPGA complex, we had YUV camera input scaling up to a WVGA DPI screen running at 60fps on a distributed FPGA platform through the HVS. Fun times.

mycall 7y ago

If you connected with Eric Anhold and worked on this, you would help millions of Pi students and users. It is an opportunity.

mrlambchop 7y ago
I checked out Eric's work just - he's doing an amazing job! Much more thought put into the subject (security for GLSL being a huge topic on its own). The lack of MMU in the VC4 architecture is a huge pain point that probably sucks up most engineering cycles when it comes to arbitrary application environments using GPU resources like X11 or Wayland - when memory runs out, what to do? You can throw engineering resources at it, but engineering talent really needs access to VMCS on the VideoCore side todo any worth while work. When we ported Android to the VC4 architecture the first time (~2010), the low memory killer in Linux was subverted to understand to kill Android applications based on their use of the VideoCore GPU memory and it worked pretty well, yet it would still close the primary running app once in a while. Run monkey over Android and all hell broke loose - really tough situations to defensively code fore. For example, for CX reasons, you had to ignore some GLES errors in the primary client due to low memory, then the system had to kill another application that was using the memory, then it would kill the EGL context for the primary application so it would refresh the entire app lifecycle using an almost suspect code path inside Android. Good times! Imagine Wayload has very similar challenges for normal desktop use.
- stefan_ 7y ago
  The open source driver doesn't use VMCS, it instead puts aside a fixed block of memory (typically 256 MiB, which I think is also a limit due to some hardware bugs) for use through the Linux Contiguous Memory Allocator (CMA) that it then draws from. VMCS only comes into the picture if you use video decode, but I think Dave Stevenson from the foundation hacked the firmware side to support importing Linux allocated memory blocks into VMCS so that you can do zero-copy decode and import into EGL (or more likely HVS, the EGL support for the formats is pretty limited). (I really liked the design of the HVS - having pretty much scripted planes is a fresh approach over similar hardware blocks that have a fixed number of planes each with own idiosyncracies and limitations)
  - mrlambchop 7y ago
    The HVS was a cool design - the only real issue if I recall was its limited line buffers meaning it was hard to determine what could be composited in real time, so we ended up always rendering to a triple buffer. It also did some amazing scaling in realtime, but this came with some crazy artifacts that were super painful to triage. The occasional "flash of bad pixels" in a live composition screen was really painful. I just remember I wrote the DPI and CPI peripherals that we brought up the HVS with - on a 4MHz FPGA complex, we had YUV camera input scaling up to a WVGA DPI screen running at 60fps on a distributed FPGA platform through the HVS. Fun times.