5 ms·
Windows equivalent: https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Runs on GP
by Const-me 3y ago
Windows equivalent: https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral...
Runs on GPUs, uses about 5GB VRAM. On integrated GPUs generates 1-2 tokens/second, on discrete ones often over 20 tokens/second.
- rahimnathwani 3y agoWow this sounds cool: Replacement of CUDA with Direct3D 11
- Const-me 3y agoThe insane complexity of these ML models is limited to the data being handled, billions of numbers in hundreds of tensors. The GPU-running compute kernels are surprisingly simple, they just called many times with different input tensors. This makes these compute kernels relatively portable across GPU APIs and languages. I think on Windows, Direct3D is often a better tech choice for GPGPU. (1) It runs on GPUs of all vendors (2) Direct3D is an essential Windows component; the OS even composes windows on desktop with D3D11. Unlike CUDA, D3D runtime libraries are already available in the OS by default, which simplifies software installation and support.