4 ms·

CUDA was a great technology before 2018. Then in 2018, they prohibited usage of GeForce GPUs in data centers: https://www.datacenterdynamics.com/en/news/nvidia

CUDA was a great technology before 2018. Then in 2018, they prohibited usage of GeForce GPUs in data centers: https://www.datacenterdynamics.com/en/news/nvidia-updates-geforce-eula-to-prohibit-data-center-use/ https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... And during 5 years which followed, they have reached the point where for many use cases nVidia is dramatically worse value compared to competitors. On my day job I do CAM/CAE, we compute a lot of FP64 numbers. Because we use D3D tech for GPGPU, we recently bought some computers with AMD 7900 XTX. These GPUs were sold for about $1000. An nVidia equivalent is L40: the AMD is 1.459 TFlops and 0.96 TB/sec memory bandwidth, the nVidia is 1.414 TFlops and 0.864 TB/sec memory bandwidth, but the nVidia is sold for about $9000. That’s an order of magnitude difference in cost efficiency.

braza 3y ago
It’s really disturbing for me the possibility that a that a hardware vendor could enforce that on court. Does someone knows what’s the rationale for that from Nvidia?
- Const-me 3y ago
  I think it’s money. nVidia is selling same chips on different boards for different prices. An example from the current generation, $800 GeForce 4070 Ti, and $2500 L4 are based on the same chip. The difference between these two cards is amount of VRAM, some settings which control clock frequencies, and these legal/software limitations. Without the legal limitations, people who need to compute stuff which fits in 12GB VRAM would only pay them $800 for the GeForce model.
P_I_Staker 3y ago
I agree the cost difference being substantial, but something seems off here. I think you can buy nVidia cores that can beat anything AMD has for ~$1500-2000. That is gaming focused, but it makes me extremely skeptical of the overall numbers. If the AMD chip really is better for your job, that's not that crazy a claim, but it make no sense that nVidia wouldn't have something for 1.5-2x as much. This seems to be the going rate, currently. If you're getting comparable speed for 1/9 the cost, I don't think that product would exist for very long. I've been looking at graphics cards and nVidia clearly beats AMD in every category with around a 50-100% markup.
- Const-me 3y ago
  Gamers don’t care about FP64 performance, and it seems nVidia is using that for market segmentation. The FP64 performance for RTX 4090 is 1.142 TFlops, for RTX 3090 Ti 0.524 TFlops. AMD doesn’t do that, FP64 performance is consistently better there, and have been this way for quite a few years. For example, the figure for 3090 Ti (a $2000 card from 2022) is similar to Radeon Vega 56, a $400 card from 2017 which can do 0.518 TFlops. And another thing: nVidia forbids usage of GeForce cards in data centers, while AMD allows that. I don’t know how specifically they define datacenter, whether it’s enforceable, or whether it’s tested in courts of various jurisdictions. I just don’t want to find out answers to these questions at the legal expenses of my employer. I believe they would prefer to not cut corners like that. I think nVidia only beats AMD due to the ecosystem: for GPGPU that’s CUDA (and especially the included first-party libraries like BLAS, FFT, DNN and others), also due to the support in popular libraries like TensorFlow. However, it’s not that hard to ignore the ecosystem, and instead write some compute shaders in HLSL. Here’s a non-trivial open-source project unrelated to CAE, where I managed to do just that with decent results: https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper That software even works on Linux, probably due to Valve’s work on DXVK 2.0 (a compatibility layer which implements D3D11 on top of Vulkan).
  - aseipp 3y ago
    > However, it’s not that hard to ignore the ecosystem I'd say this will only work if you have stable results or models to target like Whisper, that's one thing, and "going your own way" can help improve portability and stuff; Llama.cpp is another good example. But a lot of the software demand is not driven by that, it's driven by continuously evolving models and needs; a lot of the bloat or whatever you want to call it is a result of that. Besides that, the programming models are moving on. The open source Nvidia Linux driver now enables fully heterogeneous memory management on x86 across the CPU and GPU. This means the GPU and CPU do not need the programmer to enforce memory coherency, perform device-specific allocations, or copy memory; migrations, page table/TLB flushes, etc all work out of the box with no modifications to userspace software. So now your io_uring asynchronous loop can write training data to memory that is implicitly available to the GPU, no matter what memory allocator you're using. It basically means arbitrary CPU compute and arbitrary GPU compute is now composable using the memory substrate (and OS kernel) as a coherent transport/storage layer. On x86/Nvidia, this works on the granularity of a page, but on the Grace Hopper superchip, this is going to take place at the level of a cache line. Multiple Hopper Superchips can be NVLink'd together over infiniband so this works across the cluster. You can drive an entire rack of systems this way and it works. For people actually doing a lot of GPU-specific programming, or deploying models on servers (e.g. for API usage), this is going to be a big deal in the long run, and it started way back when they first introduced unified virtual memory. AMD is moving this way too for their compute stacks, I assume. The compute shader model just isn't evolving for these kinds of needs and it isn't clear it's going to anytime soon.

braza 3y ago

It’s really disturbing for me the possibility that a that a hardware vendor could enforce that on court. Does someone knows what’s the rationale for that from Nvidia?

Const-me 3y ago
I think it’s money. nVidia is selling same chips on different boards for different prices. An example from the current generation, $800 GeForce 4070 Ti, and $2500 L4 are based on the same chip. The difference between these two cards is amount of VRAM, some settings which control clock frequencies, and these legal/software limitations. Without the legal limitations, people who need to compute stuff which fits in 12GB VRAM would only pay them $800 for the GeForce model.

P_I_Staker 3y ago

I agree the cost difference being substantial, but something seems off here. I think you can buy nVidia cores that can beat anything AMD has for ~$1500-2000. That is gaming focused, but it makes me extremely skeptical of the overall numbers. If the AMD chip really is better for your job, that's not that crazy a claim, but it make no sense that nVidia wouldn't have something for 1.5-2x as much. This seems to be the going rate, currently. If you're getting comparable speed for 1/9 the cost, I don't think that product would exist for very long. I've been looking at graphics cards and nVidia clearly beats AMD in every category with around a 50-100% markup.

Const-me 3y ago
Gamers don’t care about FP64 performance, and it seems nVidia is using that for market segmentation. The FP64 performance for RTX 4090 is 1.142 TFlops, for RTX 3090 Ti 0.524 TFlops. AMD doesn’t do that, FP64 performance is consistently better there, and have been this way for quite a few years. For example, the figure for 3090 Ti (a $2000 card from 2022) is similar to Radeon Vega 56, a $400 card from 2017 which can do 0.518 TFlops. And another thing: nVidia forbids usage of GeForce cards in data centers, while AMD allows that. I don’t know how specifically they define datacenter, whether it’s enforceable, or whether it’s tested in courts of various jurisdictions. I just don’t want to find out answers to these questions at the legal expenses of my employer. I believe they would prefer to not cut corners like that. I think nVidia only beats AMD due to the ecosystem: for GPGPU that’s CUDA (and especially the included first-party libraries like BLAS, FFT, DNN and others), also due to the support in popular libraries like TensorFlow. However, it’s not that hard to ignore the ecosystem, and instead write some compute shaders in HLSL. Here’s a non-trivial open-source project unrelated to CAE, where I managed to do just that with decent results: https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper That software even works on Linux, probably due to Valve’s work on DXVK 2.0 (a compatibility layer which implements D3D11 on top of Vulkan).
- aseipp 3y ago
  > However, it’s not that hard to ignore the ecosystem I'd say this will only work if you have stable results or models to target like Whisper, that's one thing, and "going your own way" can help improve portability and stuff; Llama.cpp is another good example. But a lot of the software demand is not driven by that, it's driven by continuously evolving models and needs; a lot of the bloat or whatever you want to call it is a result of that. Besides that, the programming models are moving on. The open source Nvidia Linux driver now enables fully heterogeneous memory management on x86 across the CPU and GPU. This means the GPU and CPU do not need the programmer to enforce memory coherency, perform device-specific allocations, or copy memory; migrations, page table/TLB flushes, etc all work out of the box with no modifications to userspace software. So now your io_uring asynchronous loop can write training data to memory that is implicitly available to the GPU, no matter what memory allocator you're using. It basically means arbitrary CPU compute and arbitrary GPU compute is now composable using the memory substrate (and OS kernel) as a coherent transport/storage layer. On x86/Nvidia, this works on the granularity of a page, but on the Grace Hopper superchip, this is going to take place at the level of a cache line. Multiple Hopper Superchips can be NVLink'd together over infiniband so this works across the cluster. You can drive an entire rack of systems this way and it works. For people actually doing a lot of GPU-specific programming, or deploying models on servers (e.g. for API usage), this is going to be a big deal in the long run, and it started way back when they first introduced unified virtual memory. AMD is moving this way too for their compute stacks, I assume. The compute shader model just isn't evolving for these kinds of needs and it isn't clear it's going to anytime soon.