11 ms·

Drawing Graphics on Apple Vision with the Metal Rendering API

142 points by georginikolov 2y ago

georginikolov 2y ago
I've put everything I've learned while developing graphics for Apple Vision with the Metal API and Compositor Services in writing. It covers foveation, vertex amplification, stereoscopic rendering, and more techniques that extend beyond this particular device.
- fsloth 2y ago
  Thanks! Looks very interesting and usefull.
tomovo 2y ago
I'm curious about the autoreleasepool. What's the reason to have it and how much of a difference does it make?
- flohofwoe 2y ago
  It's a lifetime management system for short-lived objects, same idea as memory arenas but for manually refcounted objects you got out of an API. It keeps the object alive until the end of the autorelease scope without anybody having to explicitly call the release method (the autoreleasepool does this instead). Not sure if the concept still makes sense with ARC though, it's probably just a no-op there - or probably not seeing that Swift seems to have inherited autoreleasepools from ObjC).
  - tomovo 2y ago
    Hi! Yeah I know the concept from non-arc ObjC times but trying to understand the point in this context.
    - flohofwoe 2y ago
      There's an autoreleasepool around the per-frame code - which seems to be standard procedure for Metal code - so any object within a render frame that has been created with autorelease will be released at the end of a frame. Metal has a couple of 'frame transient' objects (like MTLRenderPassDescriptor and MTLRenderCommandEncoder) which have a new instance created in each frame, and I guess the main job of the autoreleasepool is to clean up those transient objects (along with any other 'short-lived-junk' that might be returned from Metal API methods). And my guess for why this is still needed in the age of ARC: I guess that ARC has a 'autorelease-blindness', e.g. it cannot figure out at compile time what objects are registered with autorelease pools (especially when the objects are passed across DLL boundaries) - it can only add retain/release calls on top. Just speculation on my part though.
    - meindnoch 2y ago
      The render thread is running an infinite loop. You need to manually drain the autorelease pool, otherwise autoreleased objects created during rendering won't be released until the thread exits (there's an implicit top-level pool in each NSThread which is drained on exit). In UIKit/AppKit, the autorelease pool is drained at the end of each NSRunLoop iteration by the framework, so you don't typically drain it yourself. Here they created their own runloop - an infinite `while` loop that calls `onRender()` - so they must drain the pool themselves.
nelup20 2y ago
Looks great, thanks so much for writing this!
lapcat 2y ago
After finally releasing a Vision Pro version of my app last week, I've sold a total of 7 units. That's by far my worst software launch ever, even worse than my past failures. Don't bother developing for this platform. The customer base is miniscule.
- that_lurker 2y ago
  You should still get familiar with it in the event that the next one will be cheaper as that will increase the customer base a lot.
  - closewith 2y ago
    On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM), production deliveries slashed, and the reported shareholder near-revolts, I think it's clear that the Vision Pro has flopped and there won't be a sequel any time soon.
    - lapcat 2y ago
      > On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM) This is a fictitious number. Apple never had the manufacturing capacity to reach that so-called target. > the reported shareholder near-revolts Citation needed.
      - andsoitis 2y ago
        GP is exaggerating (response below); at the same time many would argue that their overall argument that the Vision Pro is most probably a flop is in the ballpark. There's been much to say about price, the utility of VR, and most recently, the news that companies simply aren't bringing their apps to Vision Pro[1]. > On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM) All the reports online reflect actual vs. expected more around 450-500k vs. 800k - 1M, not 3 million. > the reported shareholder near-revolts The stock market's reaction to the Vision Pro announcement saw Apple's shares dip by 3%, indicating a mix of investor optimism and concern, not revolt. [1] https://www.wsj.com/tech/personal-tech/apple-vision-pro-software-sales-fec324c0 https://www.wsj.com/tech/personal-tech/apple-vision-pro-soft...
dagmx 2y ago
This is a pretty great reference repo in general for optimized mobile rendering, and most of it other than compositor services, can be generalized to other headsets in terms of approach (not API). Thanks for putting this up!
neomantra 2y ago
This is a great read, thank you for sharing! I learned a lot from reading it and I'm pretty experienced in this space! I enjoyed checking out your other work too. OPs article heavily covers Vertex Amplification. I didn't realize VisionOS used Vertex Amplification for this, but it makes sense since the hardware supports it. For those interested in Mesh Shading (a few good videos released this week [1][2][3]), Vertex Amplification is a key tech for Mesh Shading, where one writes Object/Mesh functions (names in Metal API, called Task/Mesh/Amplification shaders in other APIs). Introduced by Nvidia in 2018 and only really available en masse for the last couple years, Vertex Amplification was the first time a GPU could create vertices (or "destroy" them by not emitting anything), versus the fixed mesh inputs. It's so cool and powerful and a different way of thinking about the pipeline. This article shows the same concept, but in Vertex Shaders for multiple render targets. While you might not make a Vision app, it could be worthwhile read to further understand this architecture. I've spent a few months in Metal Mesh Shading and hadn't realized this application of it at all. [1] https://news.ycombinator.com/item?id=41839190 https://news.ycombinator.com/item?id=41839190 [2] https://www.youtube.com/watch?v=3EMdMD1PsgY https://www.youtube.com/watch?v=3EMdMD1PsgY [3] https://www.youtube.com/watch?v=EtX7WnFhxtQ https://www.youtube.com/watch?v=EtX7WnFhxtQ (good explanation and demo)
- dagmx 2y ago
  Nit: amplification shaders are not the same as mesh shaders (though in many cases they can be seen as abstractions over one) Your description is of a mesh shaders, but an amplification shader is able to basically reuse vertex data from a vertex shader pass without use of a mesh shader.
  - cubefox 2y ago
    Nit: Mesh shading, or the mesh shader pipeline, has two unique parts: optional task/amplification shaders, and mandatory mesh shaders.
    - neomantra 2y ago
      Thanks both for pulling on this point. In drafting, I was both torn and fuzzy about the nomenclature. Also trying to be distinct about hardware features (dynamic vertex data?) versus language/API constructs. But you are right, it's important to be clear because they are different and hardware might not support both. I was trying to see if the other graphics APIs (Vulkan, DirectX) had this Vertex Amplification in Vertex Shader feature, but it doesn't seem so? Maybe it was easier for Apple to inject the concept into Metal (advantage of controlling of the whole stack).
tasoeur 2y ago
For people who want to play with metal shaders on Apple Vision Pro I recommend this live coding app: https://shader.vision https://shader.vision
Animats 2y ago
Metal is the main reason many games are never ported to MacOS. Metal is roughly equivalent to Vulkan, which is on Windows and Linux. But Apple just had to Think Different. So all the major rendering libraries either have a layer which abstracts over Vulkan and Metal, or they don't support MacOS at all. That's what WGPU does. If it were not for Apple ignoring standards, much of that would be unnecessary. It adds so much excess baggage that the consensus is to go direct to Vulkan or DX12, and blow off MacOS support.
- spease 2y ago
  Does Metal allow for better performance with Apple hardware than if Vulcan were the “direct” interface, if you optimize with it?
- GeekyBear 2y ago
  There is no such thing as "the one true gaming graphics API" Windows/Xbox uses Direct3D, Mac/iDevices use Metal, PlayStation uses GNM/GNMX, Nintendo Switch uses NVN. Compared to DirectX, Vulkan is barely used at all in gaming, except on Linux and the most recent subset of Android devices. If it weren't for Proton's DirectX emulation running on top on Vulkan, Vulkan would be largely irrelevant, from a "widely used in real world gaming" perspective.
- aseipp 2y ago
  If Apple actually wanted to bring games to MacOS, they'd give Valve and CodeWeavers a billion dollars and get the Steam Client working, put tons of effort into a d3dvk-for-Metal solution, and have it working with as little overhead as possible. They don't do this because they can't funnel them through the App Store. The graphics API they use for the operating isn't relevant (Vulkan isn't used very much for most games.) The funny thing is that with the rise of Proton and Steam Deck driving its development -- there's literally no reason anymore for a game developer to ever target any "Desktop PC" platform except x86_64 Portable Executable files, using DirectX. Because Proton will just handle the port with nearly zero effort. Even on ARM64 devices like Snapdragon, the CPU is often not the limiting factor (and binary translation is only in the realm of 15%) so emulation for games is totally viable if the GPU can handle the load.
- 2y ago

142 points by georginikolov 2y ago

I've put everything I've learned while developing graphics for Apple Vision with the Metal API and Compositor Services in writing. It covers foveation, vertex amplification, stereoscopic rendering, and more techniques that extend beyond this particular device.

fsloth 2y ago
Thanks! Looks very interesting and usefull.

tomovo 2y ago

I'm curious about the autoreleasepool. What's the reason to have it and how much of a difference does it make?

flohofwoe 2y ago
It's a lifetime management system for short-lived objects, same idea as memory arenas but for manually refcounted objects you got out of an API. It keeps the object alive until the end of the autorelease scope without anybody having to explicitly call the release method (the autoreleasepool does this instead). Not sure if the concept still makes sense with ARC though, it's probably just a no-op there - or probably not seeing that Swift seems to have inherited autoreleasepools from ObjC).
- tomovo 2y ago
  Hi! Yeah I know the concept from non-arc ObjC times but trying to understand the point in this context.
  - flohofwoe 2y ago
    There's an autoreleasepool around the per-frame code - which seems to be standard procedure for Metal code - so any object within a render frame that has been created with autorelease will be released at the end of a frame. Metal has a couple of 'frame transient' objects (like MTLRenderPassDescriptor and MTLRenderCommandEncoder) which have a new instance created in each frame, and I guess the main job of the autoreleasepool is to clean up those transient objects (along with any other 'short-lived-junk' that might be returned from Metal API methods). And my guess for why this is still needed in the age of ARC: I guess that ARC has a 'autorelease-blindness', e.g. it cannot figure out at compile time what objects are registered with autorelease pools (especially when the objects are passed across DLL boundaries) - it can only add retain/release calls on top. Just speculation on my part though.
  - meindnoch 2y ago
    The render thread is running an infinite loop. You need to manually drain the autorelease pool, otherwise autoreleased objects created during rendering won't be released until the thread exits (there's an implicit top-level pool in each NSThread which is drained on exit). In UIKit/AppKit, the autorelease pool is drained at the end of each NSRunLoop iteration by the framework, so you don't typically drain it yourself. Here they created their own runloop - an infinite `while` loop that calls `onRender()` - so they must drain the pool themselves.

nelup20 2y ago

Looks great, thanks so much for writing this!

lapcat 2y ago

After finally releasing a Vision Pro version of my app last week, I've sold a total of 7 units. That's by far my worst software launch ever, even worse than my past failures. Don't bother developing for this platform. The customer base is miniscule.

that_lurker 2y ago
You should still get familiar with it in the event that the next one will be cheaper as that will increase the customer base a lot.
- closewith 2y ago
  On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM), production deliveries slashed, and the reported shareholder near-revolts, I think it's clear that the Vision Pro has flopped and there won't be a sequel any time soon.
  - lapcat 2y ago
    > On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM) This is a fictitious number. Apple never had the manufacturing capacity to reach that so-called target. > the reported shareholder near-revolts Citation needed.
    - andsoitis 2y ago
      GP is exaggerating (response below); at the same time many would argue that their overall argument that the Vision Pro is most probably a flop is in the ballpark. There's been much to say about price, the utility of VR, and most recently, the news that companies simply aren't bringing their apps to Vision Pro[1]. > On track to having sold less than 15% of their Y1 target (400,000 vs 3 MM) All the reports online reflect actual vs. expected more around 450-500k vs. 800k - 1M, not 3 million. > the reported shareholder near-revolts The stock market's reaction to the Vision Pro announcement saw Apple's shares dip by 3%, indicating a mix of investor optimism and concern, not revolt. [1] https://www.wsj.com/tech/personal-tech/apple-vision-pro-software-sales-fec324c0 https://www.wsj.com/tech/personal-tech/apple-vision-pro-soft...

dagmx 2y ago

This is a pretty great reference repo in general for optimized mobile rendering, and most of it other than compositor services, can be generalized to other headsets in terms of approach (not API). Thanks for putting this up!

neomantra 2y ago

This is a great read, thank you for sharing! I learned a lot from reading it and I'm pretty experienced in this space! I enjoyed checking out your other work too. OPs article heavily covers Vertex Amplification. I didn't realize VisionOS used Vertex Amplification for this, but it makes sense since the hardware supports it. For those interested in Mesh Shading (a few good videos released this week [1][2][3]), Vertex Amplification is a key tech for Mesh Shading, where one writes Object/Mesh functions (names in Metal API, called Task/Mesh/Amplification shaders in other APIs). Introduced by Nvidia in 2018 and only really available en masse for the last couple years, Vertex Amplification was the first time a GPU could create vertices (or "destroy" them by not emitting anything), versus the fixed mesh inputs. It's so cool and powerful and a different way of thinking about the pipeline. This article shows the same concept, but in Vertex Shaders for multiple render targets. While you might not make a Vision app, it could be worthwhile read to further understand this architecture. I've spent a few months in Metal Mesh Shading and hadn't realized this application of it at all. [1] https://news.ycombinator.com/item?id=41839190 https://news.ycombinator.com/item?id=41839190 [2] https://www.youtube.com/watch?v=3EMdMD1PsgY https://www.youtube.com/watch?v=3EMdMD1PsgY [3] https://www.youtube.com/watch?v=EtX7WnFhxtQ https://www.youtube.com/watch?v=EtX7WnFhxtQ (good explanation and demo)

dagmx 2y ago
Nit: amplification shaders are not the same as mesh shaders (though in many cases they can be seen as abstractions over one) Your description is of a mesh shaders, but an amplification shader is able to basically reuse vertex data from a vertex shader pass without use of a mesh shader.
- cubefox 2y ago
  Nit: Mesh shading, or the mesh shader pipeline, has two unique parts: optional task/amplification shaders, and mandatory mesh shaders.
  - neomantra 2y ago
    Thanks both for pulling on this point. In drafting, I was both torn and fuzzy about the nomenclature. Also trying to be distinct about hardware features (dynamic vertex data?) versus language/API constructs. But you are right, it's important to be clear because they are different and hardware might not support both. I was trying to see if the other graphics APIs (Vulkan, DirectX) had this Vertex Amplification in Vertex Shader feature, but it doesn't seem so? Maybe it was easier for Apple to inject the concept into Metal (advantage of controlling of the whole stack).

tasoeur 2y ago

For people who want to play with metal shaders on Apple Vision Pro I recommend this live coding app: https://shader.vision https://shader.vision

Animats 2y ago

Metal is the main reason many games are never ported to MacOS. Metal is roughly equivalent to Vulkan, which is on Windows and Linux. But Apple just had to Think Different. So all the major rendering libraries either have a layer which abstracts over Vulkan and Metal, or they don't support MacOS at all. That's what WGPU does. If it were not for Apple ignoring standards, much of that would be unnecessary. It adds so much excess baggage that the consensus is to go direct to Vulkan or DX12, and blow off MacOS support.

spease 2y ago
Does Metal allow for better performance with Apple hardware than if Vulcan were the “direct” interface, if you optimize with it?
GeekyBear 2y ago
There is no such thing as "the one true gaming graphics API" Windows/Xbox uses Direct3D, Mac/iDevices use Metal, PlayStation uses GNM/GNMX, Nintendo Switch uses NVN. Compared to DirectX, Vulkan is barely used at all in gaming, except on Linux and the most recent subset of Android devices. If it weren't for Proton's DirectX emulation running on top on Vulkan, Vulkan would be largely irrelevant, from a "widely used in real world gaming" perspective.
aseipp 2y ago
If Apple actually wanted to bring games to MacOS, they'd give Valve and CodeWeavers a billion dollars and get the Steam Client working, put tons of effort into a d3dvk-for-Metal solution, and have it working with as little overhead as possible. They don't do this because they can't funnel them through the App Store. The graphics API they use for the operating isn't relevant (Vulkan isn't used very much for most games.) The funny thing is that with the rise of Proton and Steam Deck driving its development -- there's literally no reason anymore for a game developer to ever target any "Desktop PC" platform except x86_64 Portable Executable files, using DirectX. Because Proton will just handle the port with nearly zero effort. Even on ARM64 devices like Snapdragon, the CPU is often not the limiting factor (and binary translation is only in the realm of 15%) so emulation for games is totally viable if the GPU can handle the load.
2y ago