5 ms·

> If a frame is 16ms, and the best consumer SSDs are around 2GB/s, You can only load 33MB of assets in a one frame in a best-case scenario. 33 MB of assets per

by exDM69 10y ago

> If a frame is 16ms, and the best consumer SSDs are around 2GB/s, You can only load 33MB of assets in a one frame in a best-case scenario. 33 MB of assets per frame is more than enough for a lot of practical tasks. Modern texture compression achieves 2 bits per pixel (e.g. ASTC 8x8) with very little percievable loss of quality (when you apply texture mapping, lighting, effects, etc. ASTC 12x12 gets 0.89 bits per pixel, but noticeable loss of quality). At this rate, the 33 MB/frame is 16k x 8k worth of texture data, every frame. Vastly more than there are pixels on screen. And this figure does not take into account the block cache at all. If the data has been used recently (several minutes depending on RAM size), a copy of it might be still around in RAM which would make it nearly instantaneous and almost a gigabyte per frame of data (assuming all memory bw is put to this use). As for the framerate issues, it would be imperative that this technique is implemented in a stall-free way (which is possible with Vulkan/D3D12) so that a steady 60 fps is maintained. All textures should have at least a few mip map levels resident at all times to fall back on. A single "standard" GPU sparse page (64k) is 512x512 texels at ASTC 8x8, which is quite a large texture already. In other words: 33 MB per frame is a lot more than what we get today.

dogma1138 10y ago
The GPU doesn't have DMA to the disk or RAM it has to go through the CPU to access it which causes quite a delay. Even if you read from RAM during normal operations you'll have very low framerate unless you are using pretty damn good latency masking a good example will be texture streaming where you hold low res textures in your GPU memory and load the higher res ones from RAM or disk and even on very high end systems it causes allot of texture popins which people find rather annoying.
- vardump 10y ago
  GPU can just queue page faults and raise an IRQ. Those faults can then be handled in any size of group on the CPU side. By using a queue, latency can be effectively hidden. Perhaps disk I/O DMA can even be directly connected to GPU DMA, bypassing system RAM entirely. Even currently, ethernet and disk DMA can bypass memory and go directly to CPU L3 cache. With existing flexibility like that, it's not far-fetched to be able to connect DMA between two devices. Pretty much same happens when you touch a page that's not present from user mode. CPU will get interrupted and queue disk I/O to get the faulting page. Even disks themselves have their internal queues (like SATA NCQ). Disks also have CPUs to serve queued I/O requests.
  - exDM69 10y ago
    > GPU can just raise IRQ and have a queue for the page faults. Handling GPU IRQ's is possible but has unnecessarily high latency and pipeline stalling problems. However, the kernel mode drivers do a bit of this behind the scenes (mainly when switching from app to app). But using sparse textures (ie. GL_EXT_sparse_texture2), the GPU can detect if a fault would occur and react to it. Instead of IRQ'ing for every missed page, a list of all missed pages can be extracted at the end of a frame. > Perhaps disk I/O DMA can even be connected to GPU DMA, bypassing system RAM entirely. Afaik Nvidia's NVLink does this but it's aimed at super computers, not graphics.
    - vardump 10y ago
      > Handling GPU IRQ's is possible but has unnecessarily high latency and pipeline stalling problems. However, the kernel mode drivers do a bit of this behind the scenes (mainly when switching from app to app). Time it takes for GPU to access data present in its local internal GDDR RAM: about 1 microsecond. Time it takes from GPU asserting (pulls up) [1] IRQ until CPU handles IRQ: 5-50 microseconds. Time it takes for CPU to insert I/O request into OS internal queue: unknown, but it's almost certainly a small fraction of a microsecond. Hundreds of faults can be handled with one IRQ request. So there's not really that much more latency than in a normal CPU page fault. Actually probably less on average, because these faults can be grouped. GPU memory access latency is very slow and they are already built to handle high latency with a massive number of hardware threads. IRQ can for example be edge triggered by the first GPU page fault. Each GPU page fault doesn't need to trigger CPU side interrupt. That'd be pointless, it'd just cause high CPU load for no benefit. I think some sort of page fault FIFO is simpler hardware wise than to wait for something arbitrary event like "end of a frame". GPU can just keep pushing new faults in a CPU-visible FIFO. That way it's possible to amortize latency and to avoid IRQ storm (=excessive number of IRQ requests). CPU side can just pull faults from the FIFO at whatever rate I/O system can support. Fault data from GPU could also include priority, so that more visually important data can be fetched first even if it wasn't encountered first. For example geometry or textures covering major parts of screen could have high priority. (I've co-designed a HW mechanism for avoiding an excessive number of IRQs without sacrificing latency and designed and implemented a kernel mode driver to support it. Of course GPUs are quite a bit more complicated than that simple case.) [1]: Well, technically it's sending an MSI(-X) message. Just using traditional terminology for clarity. https://en.wikipedia.org/wiki/Message_Signaled_Interrupts#MSI-X https://en.wikipedia.org/wiki/Message_Signaled_Interrupts#MS...
- exDM69 10y ago
  Latency hiding is indeed required, but it's a solvable problem. There's no way that this would work synchronously. See my other comment above. I implemented a technique like this (on a <10 watt mobile device!) and I had no issues whatsoever with framerate, even my suboptimal technique (using OpenGL sparse textures, which are quite restrictive) that might stall. These stalls can be avoided with Vulkan/D3D12 techniques (using aliased sparse pages, ie. one physical page mapped on several textures) so maintaining a stable framerate simply isn't an issue. This was done with the CPU orchestrating the whole deal. The GPU isn't required to have access to disk or initiate the DMA transfer. Latency, on the other hand, is an issue but it doesn't seem to be that bad in practice (warning: anecdotal evidence). In my simple demo, practically every page uploaded was resident on the GPU by the next frame after it was required. A small number of pages (less than 1%) had 2 frames of latency. None had 3 or more. Hiding the visual artifacts from texture popping is a real issue too, but can be mitigated by speculatively uploading pages and applying filtering between mip map levels. All of this can be done today using Vulkan or D3D12. What Carmack is suggesting (if I understood him correctly) is to make the kernel driver on the CPU initiate asynchronous upload without userspace intervention, which would improve latency and bandwidth.
  - snaky 10y ago
    > Latency hiding is indeed required, but it's a solvable problem Is it so regarding VR? How can you predict a direction and angle where a user turn his head to?
    - vardump 10y ago
      Latency hiding for asset loading. You're talking about rendering latency. Those two are different issues. Asset loading latency doesn't affect rendering latency, put simply, it just means you get to play sooner, with less wait.
- the8472 10y ago
  > The GPU doesn't have DMA to the disk or RAM But does it have to be this way?
  - dogma1138 10y ago
    We've barely started touching the security of things like WebGL, I really don't want to see DMA attacks coming to me via the internet just yet.
- Hondor 10y ago
  What if the GPU had its own private SSD with textures installed to it when you install the game? A 100GB SSD is about 20% of the price of a high end video card.
  - dogma1138 10y ago
    Inb4 the next GTX titan comes with 24GB of HBM and an M.2 slot. I don't think it's necessary for applications that require so much memory you can probably develop some proprietary tech to access storage (NVIDIA does it). JC isn't talking about compute tho, he's talking about gaming he really likes megatextures but he's in a minority and I honestly can't claim to have enough expertise to judge if he's right or not. GPU memory management is a mixed bag there are many cards with different speed of RAM, different bandwidths all doing the magic voodoo in the background regarding low level memory access and compression way beyond what you are exposed to on the driver and API layers (e.g. NVIDIA's incard memory compression which is enabled regardless of how you compress or load your textures in the first place). WDDM allows lower end system to take advantage of more video memory to improve performance in less demanding applications which are most apps today while still enabling all the nice graphics we come to expect from out window manager and applications. I'm not sure if allowing the GPU to read directly from the SSD or RAM (outside of the current scope of virtualized GPU memory and asset loading) has actually enough benefit to justify both the engineering costs and the potential security pitfall that can happen when you run multiple applications that all of a sudden get DMA access to your RAM and local storage. This is more so the case for gaming considering the amount of RAM that graphic cards come with today and this is only increasing I don't know if anyone but JC actually wants/needs this. I am wondering how do consoles handle it, IIRC from playing a bit with the UDK (UE3) at the time I recall that it supported streaming textures directly from the optical media on the Xbox 360, so if anyone knows how it works under the hood and is willing to share I think it would add to this topic.
  - CyberDildonics 10y ago
    That could happen with Intel's optane SSDs but I don't think it is necessary. The latency between the CPU and GPU is not measured in any sort of human perceivable time, so it doesn't matter unless the GPU is making millions of cpu memory accesses, and that isn't how games or GPUs work for obvious reasons. The bandwidth is also not a problem, the entire memory of a 16GB GPU can be filled in under a second.
- Unklejoe 10y ago
  [The GPU doesn't have DMA to the disk or RAM it has to go through the CPU to access it which causes quite a delay.] How come? If it's on the PCI bus it should be able to bus master its way into the system RAM, no? Or was this capability disabled to prevent bad graphics code from being able to exploit the system or something?
  - vardump 10y ago
    How would GPU or GPU drivers know how to issue DMA transfer and know when DMA is ready on another piece of hardware? Those things are specific to a piece of hardware. Besides there's a small detail called IOMMU that prevents PCI(-e) devices from writing or reading wherever they please.
    - dogma1138 10y ago
      On windows GPU memory is already virtualized. You can pull assets from RAM somewhat transparently already (if you want to do it well you need to optimize it further, but to some extent NVIDIA and AMD do quite a bit of optimization in the driver too), shared page table between the CPU and GPU is also already in place under WDDM. So adding a disk based page file to this won't be so hard, it could more or less work the same way as the page file currently works. I'm still not sure if this is actually needed, GPU's already come with stupid amount of RAM today, i have 24GB of GDDR5 in my system, and even mid range cards today will not come with less than 6GB of RAM.
      - vardump 10y ago
        > You can pull assets from RAM somewhat transparently already (if you want to do it well you need to optimize it further, but to some extent NVIDIA and AMD do quite a bit of optimization in the driver too), shared page table between the CPU and GPU is also already in place under WDDM. Can GPU page table entry point to non-present page(s)? Or does it only work for "pinned pages" [1], that cannot paged out of RAM? If it doesn't require pinning, then what prevents from mmapping assets on the disk even today? If, however, it does require pinning, then John Carmack has a damn good point. [1]: https://en.wikipedia.org/wiki/Virtual_memory#Pinned_pages https://en.wikipedia.org/wiki/Virtual_memory#Pinned_pages
- valarauca1 10y ago
  >The GPU doesn't have DMA to the disk or RAM The GPU does have DMA to RAM References: https://stackoverflow.com/questions/19242711/how-can-i-use-gpu-dma-from-gpu-cuda-code-to-copying-data https://stackoverflow.com/questions/19242711/how-can-i-use-g... http://docs.nvidia.com/cuda/gpudirect-rdma/ http://docs.nvidia.com/cuda/gpudirect-rdma/ https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect
  - dogma1138 10y ago
    GPUDirect is very specific it require implementation on the PCIe switch (some modification to IOMMU), UEFI/BIOS and card, as well as the OS Kernel and the display drivers. Because it requires both very specific software and hardware configurations only some features work on some systems. http://docs.nvidia.com/cuda/gpudirect-rdma/index.html#supported-systems http://docs.nvidia.com/cuda/gpudirect-rdma/index.html#suppor... Also calling this DMA is kind of misleading it does require the CUDA software layer to work, and for the most part it's a lot of hacks coupled together to form a feature set. So I would still currently stand by what I said that there isn't a standard and generic way for GPU's to have DMA access.

by exDM69 10y ago

dogma1138 10y ago

The GPU doesn't have DMA to the disk or RAM it has to go through the CPU to access it which causes quite a delay. Even if you read from RAM during normal operations you'll have very low framerate unless you are using pretty damn good latency masking a good example will be texture streaming where you hold low res textures in your GPU memory and load the higher res ones from RAM or disk and even on very high end systems it causes allot of texture popins which people find rather annoying.

vardump 10y ago
GPU can just queue page faults and raise an IRQ. Those faults can then be handled in any size of group on the CPU side. By using a queue, latency can be effectively hidden. Perhaps disk I/O DMA can even be directly connected to GPU DMA, bypassing system RAM entirely. Even currently, ethernet and disk DMA can bypass memory and go directly to CPU L3 cache. With existing flexibility like that, it's not far-fetched to be able to connect DMA between two devices. Pretty much same happens when you touch a page that's not present from user mode. CPU will get interrupted and queue disk I/O to get the faulting page. Even disks themselves have their internal queues (like SATA NCQ). Disks also have CPUs to serve queued I/O requests.
- exDM69 10y ago
  > GPU can just raise IRQ and have a queue for the page faults. Handling GPU IRQ's is possible but has unnecessarily high latency and pipeline stalling problems. However, the kernel mode drivers do a bit of this behind the scenes (mainly when switching from app to app). But using sparse textures (ie. GL_EXT_sparse_texture2), the GPU can detect if a fault would occur and react to it. Instead of IRQ'ing for every missed page, a list of all missed pages can be extracted at the end of a frame. > Perhaps disk I/O DMA can even be connected to GPU DMA, bypassing system RAM entirely. Afaik Nvidia's NVLink does this but it's aimed at super computers, not graphics.
  - vardump 10y ago
    > Handling GPU IRQ's is possible but has unnecessarily high latency and pipeline stalling problems. However, the kernel mode drivers do a bit of this behind the scenes (mainly when switching from app to app). Time it takes for GPU to access data present in its local internal GDDR RAM: about 1 microsecond. Time it takes from GPU asserting (pulls up) [1] IRQ until CPU handles IRQ: 5-50 microseconds. Time it takes for CPU to insert I/O request into OS internal queue: unknown, but it's almost certainly a small fraction of a microsecond. Hundreds of faults can be handled with one IRQ request. So there's not really that much more latency than in a normal CPU page fault. Actually probably less on average, because these faults can be grouped. GPU memory access latency is very slow and they are already built to handle high latency with a massive number of hardware threads. IRQ can for example be edge triggered by the first GPU page fault. Each GPU page fault doesn't need to trigger CPU side interrupt. That'd be pointless, it'd just cause high CPU load for no benefit. I think some sort of page fault FIFO is simpler hardware wise than to wait for something arbitrary event like "end of a frame". GPU can just keep pushing new faults in a CPU-visible FIFO. That way it's possible to amortize latency and to avoid IRQ storm (=excessive number of IRQ requests). CPU side can just pull faults from the FIFO at whatever rate I/O system can support. Fault data from GPU could also include priority, so that more visually important data can be fetched first even if it wasn't encountered first. For example geometry or textures covering major parts of screen could have high priority. (I've co-designed a HW mechanism for avoiding an excessive number of IRQs without sacrificing latency and designed and implemented a kernel mode driver to support it. Of course GPUs are quite a bit more complicated than that simple case.) [1]: Well, technically it's sending an MSI(-X) message. Just using traditional terminology for clarity. https://en.wikipedia.org/wiki/Message_Signaled_Interrupts#MSI-X https://en.wikipedia.org/wiki/Message_Signaled_Interrupts#MS...
exDM69 10y ago
Latency hiding is indeed required, but it's a solvable problem. There's no way that this would work synchronously. See my other comment above. I implemented a technique like this (on a <10 watt mobile device!) and I had no issues whatsoever with framerate, even my suboptimal technique (using OpenGL sparse textures, which are quite restrictive) that might stall. These stalls can be avoided with Vulkan/D3D12 techniques (using aliased sparse pages, ie. one physical page mapped on several textures) so maintaining a stable framerate simply isn't an issue. This was done with the CPU orchestrating the whole deal. The GPU isn't required to have access to disk or initiate the DMA transfer. Latency, on the other hand, is an issue but it doesn't seem to be that bad in practice (warning: anecdotal evidence). In my simple demo, practically every page uploaded was resident on the GPU by the next frame after it was required. A small number of pages (less than 1%) had 2 frames of latency. None had 3 or more. Hiding the visual artifacts from texture popping is a real issue too, but can be mitigated by speculatively uploading pages and applying filtering between mip map levels. All of this can be done today using Vulkan or D3D12. What Carmack is suggesting (if I understood him correctly) is to make the kernel driver on the CPU initiate asynchronous upload without userspace intervention, which would improve latency and bandwidth.
- snaky 10y ago
  > Latency hiding is indeed required, but it's a solvable problem Is it so regarding VR? How can you predict a direction and angle where a user turn his head to?
  - vardump 10y ago
    Latency hiding for asset loading. You're talking about rendering latency. Those two are different issues. Asset loading latency doesn't affect rendering latency, put simply, it just means you get to play sooner, with less wait.
the8472 10y ago
> The GPU doesn't have DMA to the disk or RAM But does it have to be this way?
- dogma1138 10y ago
  We've barely started touching the security of things like WebGL, I really don't want to see DMA attacks coming to me via the internet just yet.
Hondor 10y ago
What if the GPU had its own private SSD with textures installed to it when you install the game? A 100GB SSD is about 20% of the price of a high end video card.
- dogma1138 10y ago
  Inb4 the next GTX titan comes with 24GB of HBM and an M.2 slot. I don't think it's necessary for applications that require so much memory you can probably develop some proprietary tech to access storage (NVIDIA does it). JC isn't talking about compute tho, he's talking about gaming he really likes megatextures but he's in a minority and I honestly can't claim to have enough expertise to judge if he's right or not. GPU memory management is a mixed bag there are many cards with different speed of RAM, different bandwidths all doing the magic voodoo in the background regarding low level memory access and compression way beyond what you are exposed to on the driver and API layers (e.g. NVIDIA's incard memory compression which is enabled regardless of how you compress or load your textures in the first place). WDDM allows lower end system to take advantage of more video memory to improve performance in less demanding applications which are most apps today while still enabling all the nice graphics we come to expect from out window manager and applications. I'm not sure if allowing the GPU to read directly from the SSD or RAM (outside of the current scope of virtualized GPU memory and asset loading) has actually enough benefit to justify both the engineering costs and the potential security pitfall that can happen when you run multiple applications that all of a sudden get DMA access to your RAM and local storage. This is more so the case for gaming considering the amount of RAM that graphic cards come with today and this is only increasing I don't know if anyone but JC actually wants/needs this. I am wondering how do consoles handle it, IIRC from playing a bit with the UDK (UE3) at the time I recall that it supported streaming textures directly from the optical media on the Xbox 360, so if anyone knows how it works under the hood and is willing to share I think it would add to this topic.
- CyberDildonics 10y ago
  That could happen with Intel's optane SSDs but I don't think it is necessary. The latency between the CPU and GPU is not measured in any sort of human perceivable time, so it doesn't matter unless the GPU is making millions of cpu memory accesses, and that isn't how games or GPUs work for obvious reasons. The bandwidth is also not a problem, the entire memory of a 16GB GPU can be filled in under a second.
Unklejoe 10y ago
[The GPU doesn't have DMA to the disk or RAM it has to go through the CPU to access it which causes quite a delay.] How come? If it's on the PCI bus it should be able to bus master its way into the system RAM, no? Or was this capability disabled to prevent bad graphics code from being able to exploit the system or something?
- vardump 10y ago
  How would GPU or GPU drivers know how to issue DMA transfer and know when DMA is ready on another piece of hardware? Those things are specific to a piece of hardware. Besides there's a small detail called IOMMU that prevents PCI(-e) devices from writing or reading wherever they please.
  - dogma1138 10y ago
    On windows GPU memory is already virtualized. You can pull assets from RAM somewhat transparently already (if you want to do it well you need to optimize it further, but to some extent NVIDIA and AMD do quite a bit of optimization in the driver too), shared page table between the CPU and GPU is also already in place under WDDM. So adding a disk based page file to this won't be so hard, it could more or less work the same way as the page file currently works. I'm still not sure if this is actually needed, GPU's already come with stupid amount of RAM today, i have 24GB of GDDR5 in my system, and even mid range cards today will not come with less than 6GB of RAM.
    - vardump 10y ago
      > You can pull assets from RAM somewhat transparently already (if you want to do it well you need to optimize it further, but to some extent NVIDIA and AMD do quite a bit of optimization in the driver too), shared page table between the CPU and GPU is also already in place under WDDM. Can GPU page table entry point to non-present page(s)? Or does it only work for "pinned pages" [1], that cannot paged out of RAM? If it doesn't require pinning, then what prevents from mmapping assets on the disk even today? If, however, it does require pinning, then John Carmack has a damn good point. [1]: https://en.wikipedia.org/wiki/Virtual_memory#Pinned_pages https://en.wikipedia.org/wiki/Virtual_memory#Pinned_pages
valarauca1 10y ago
>The GPU doesn't have DMA to the disk or RAM The GPU does have DMA to RAM References: https://stackoverflow.com/questions/19242711/how-can-i-use-gpu-dma-from-gpu-cuda-code-to-copying-data https://stackoverflow.com/questions/19242711/how-can-i-use-g... http://docs.nvidia.com/cuda/gpudirect-rdma/ http://docs.nvidia.com/cuda/gpudirect-rdma/ https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect
- dogma1138 10y ago
  GPUDirect is very specific it require implementation on the PCIe switch (some modification to IOMMU), UEFI/BIOS and card, as well as the OS Kernel and the display drivers. Because it requires both very specific software and hardware configurations only some features work on some systems. http://docs.nvidia.com/cuda/gpudirect-rdma/index.html#supported-systems http://docs.nvidia.com/cuda/gpudirect-rdma/index.html#suppor... Also calling this DMA is kind of misleading it does require the CUDA software layer to work, and for the most part it's a lot of hacks coupled together to form a feature set. So I would still currently stand by what I said that there isn't a standard and generic way for GPU's to have DMA access.