Const-me

38 ms·

451.
by Const-me 3y ago
Couple times in the past I wanted to port open source ML models from CUDA/Python to a better technology stack. I have ported Whisper https://github.com/Const-me/Whisper/ and Mistral https://github.
452.
by Const-me 3y ago
> the consumer cards don't do compute? Typically, software developers only support a single GPGPU API, and that API is nVidia CUDA. Technically, consumer AMD cards are awesome at compute. For example, UE5 renders triangle meshes wit
453.
by Const-me 3y ago
The insane complexity of these ML models is limited to the data being handled, billions of numbers in hundreds of tensors. The GPU-running compute kernels are surprisingly simple, they just called many times with different input tensors. Th
454.
by Const-me 3y ago
Windows equivalent: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Runs on GPUs, uses about 5GB VRAM. On integrated GPUs generates 1-2 tokens/second, on discrete ones often ove
455.
by Const-me 3y ago
> ie most ML inferencing Most ML inferencing is throttled with memory, not compute. This certainly applies to both Whisper and Mistral models. > it certainly doesn't spell the end of CUDA No, because traditional HPC. Some people
456.
by Const-me 3y ago
> no subgroups Indeed, in D3D they are called “wave intrinsics” and require D3D12. But that’s IMO a reasonable price to pay for hardware compatibility. > no cooperative matrix multiplication Matrix multiplication compute shader which
457.
by Const-me 3y ago
I have recently published an AI-related open-source project entirely based on compute shaders https://github.com/Const-me/Cgml and I’m super happy with the workflow. Possible to implement very complicated things withou
458.
by Const-me 3y ago
No, I have not considered ILGPU, it’s the first time I see that thing. Very interesting, but I’m not sure I would have used it for that project. In my experience, using any non-native GPU API is asking for troubles. On Windows, the native o
459.
by Const-me 3y ago
I don’t work for Microsoft, and I don’t care whether people are using Microsoft’s technologies. As a consumer, I’m not happy with nVidia’s monopoly in the field. I find it disappointing that Microsoft chose to play along, instead of disrupt
460.
by Const-me 3y ago
Fans have limited lifespan due to bearing and dust accumulation. Are you saying I can replace fan and heatsink without also replacing the liquid metal?
461.
by Const-me 3y ago
> will run only on NVIDIA GPUs for the preview I wonder why Microsoft helps nVidia, instead of using their own technology? Here’s an example: https://github.com/Const-me/Cgml
462.
by Const-me 3y ago
Intel Core Duo CPU was released in 2006. By then it was obvious computationally intensive programs need multithreading, these Unix-style processes are no longer adequate. I wonder why did it took so long for FFmpeg? BTW, MS Media foundation
463.
by Const-me 3y ago
The repository contains the library, an implementation of Mistral 7B model on top of that library, and an offline chat program to interact with that model. Here’s a guide for the offline chat program: https://github.com/Cons
464.
Show HN: Cgml, GPU-targeted vendor-agnostic AI library for Windows(github.com)
2 points by Const-me 3y ago|1 comments
465.
by Const-me 3y ago
> _mm256_cvtps_epu32 that represent a low-level operation in a specific instruction set (this is a float to int cast from AVX2) That instruction is from AVX-512. There’s no AVX2 float to int cast. There’s in AVX1, the int is signed and t
466.
by Const-me 3y ago
> How did we overcome the HBM <-> SRAM bottleneck? Because every number we load from the model through that bottleneck gets reused, to compute different requests within the batch.
467.
by Const-me 3y ago
It seems the getenv_s and GetEnvironmentVariable API functions are 100% equivalent; the API is only slightly different. When you don’t know the length of the variable you query, you guessed wrong on the first attempt, and other threads are
468.
by Const-me 3y ago
Interestingly, Windows developers made much better choices back in the 1990-s. GetEnvironmentStrings() API comes with the FreeEnvironmentStrings() counterpart. Whoever calls GetEnvironmentStrings() to get the entire environment is then resp
469.
by Const-me 3y ago
It’s not just the language. That code is impossible to directly translate to a pixel shader because GPUs only implement fixed-function blending. Render target pixels (and depth values) are write-only in the graphics pipeline, they can be on
470.
by Const-me 3y ago
NanoVG is a 2D rendering library built directly on top of OpenGL or GLES. Modern rich GUI needs much more than a renderer, but amount and development cost of that “much more” varies depending on GUI complexity. The device I developed is rat
471.
by Const-me 3y ago
> cross-platform GUI that still doesn't have a good, officially supported story Yeah, but in the meantime there’re decent unofficial options. Once I compiled NanoVG into a DLL, consumed with C#, and implemented rich GUI on top of th
472.
by Const-me 3y ago
> Attending a conference in Russia is collaborating with fascists? There’s no private business in Russia, almost all economy is state owned in practice. That particular conference is organized by Sberbank. By speaking at the conference,
473.
by Const-me 3y ago
Musk is collaborating with fascists. Here’s an example: https://news.yahoo.com/elon-musk-speak-russian-conference-17...
474.
by Const-me 3y ago
I think one possible reason for that, ideally these things need different memory. If you use high-bandwidth high-latency GDDR memory, CPU cores will underperform due to high latency, like there: https://www.tomshardware.com/
475.
by Const-me 3y ago
> are simply unmatched by anything in the PC world at that price - sometimes any price Technically, comparable Windows laptops do exist, but hardware vendors don’t want to sell them. For some reason, hardware vendors really want to sell
476.
by Const-me 3y ago
Not quite. These floating-point EUs are shared between both threads of the physical core. I would rather say 64 CUDA cores.
477.
by Const-me 3y ago
That’s incorrect. The 3rd Geneva convention of 1949 says only combatant are prisoners of war: members of the armed forces, members of militia, etc. Here’s a link: https://www.un.org/en/genocideprevention/documents&
478.
by Const-me 3y ago
An old one: https://en.wikipedia.org/wiki/Vampire:_The_Masquerade_%E2%80... There were many technical issues with the release version, but GOG sells the game with unofficial patches integrated, which worked great for m
479.
by Const-me 3y ago
> How is that relevant to the vast majority of the code targeted by LINQ? The people I responded to were discussing applicability of LINQ. I think very fast sum of List<int> collections doesn’t compensate for suboptimal performance
480.
by Const-me 3y ago
My point is the support is very limited, only works for a few trivial use cases, like sum/min/max/average of arrays of a few selected types. I believe it’s technically possible to vectorize more complicated stuff in C#, just

More ›