Const-me

31 ms·

421.
by Const-me 3y ago
> The amount of unsafe code used to implement C# vastly outweighs the amount in Rust's standard library. According to bing.com chat, https://github.com/dotnet/runtime has 3.5M LOC, and https://github
422.
by Const-me 3y ago
Don’t create new objects, instead use stack and/or unmanaged heap. This reduces the expressive power of the language, for example LINQ from the standard library is probably out because based on the delegates which require memory alloca
423.
by Const-me 3y ago
> C# is not more safe than Rust By design, Rust requires unsafe code to implement any non-trivial data structures (except trivial POD types). This applies to both Rust standard library, and third-party crates. The issue is not a theory,
424.
by Const-me 3y ago
> What kind of solution would you propose to folks like your boss? Rewrite it in C#. It’s safer than Rust, because VM. Both standard and third party libraries are often way better. With modern versions of the language, GC allocations ar
425.
by Const-me 3y ago
> the API was high-friction due to the shader language, and the glue between shader and CPU Direct3D 11 compute shaders share these things with Vulkan, yet D3D11 is relatively easy to use. For example, see that library which implements M
426.
by Const-me 3y ago
People use batching on servers to optimize throughput for the complete server, not for a single session. See “throughput (tokens/s) versus concurrency” graph in that article: https://www.predera.com/blog/mistral-7b
427.
by Const-me 3y ago
I wonder are you using a quantized version of Mistral? NVidia 3090 has 936 GB/second memory bandwidth, so 150 tokens/second = 7.2 GB per token. In the original 16 bits format, the model takes about 13GB. Anyway, while these datace
428.
by Const-me 3y ago
These data center targeted GPUs can only output that many tokens per second for large batches. These tokens are shared between hundreds or even thousands of users concurrently accessing the same server. That’s why despite these GPUs deliver
429.
by Const-me 3y ago
Minor update https://github.com/Const-me/Cgml/releases/tag/1.1a Can’t edit that comment anymore, too late.
430.
by Const-me 3y ago
Is that explanation better? https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... Same Mistral Instruct 0.2 model, different implementation.
431.
by Const-me 3y ago
The model is on BitTorrent, see readme for the frontend app: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... The torrent file is also inside the MistralChat.zip archive.
432.
by Const-me 3y ago
I made a free and open source Windows equivalent: https://github.com/Const-me/Cgml/releases/tag/1.1
433.
by Const-me 3y ago
Sadly, WPF is Windows-only. There’s an equivalent cross-platform GUI framework called Avalonia https://www.avaloniaui.net/ I don’t have a hands-on experience with it, but based on the internets I have an impression the tech
434.
by Const-me 3y ago
On Windows, WPF is better: https://en.wikipedia.org/wiki/Windows_Presentation_Foundatio... The critical features missing from HTML are data binding and data templates. Last time I checked, many modern frontend framewor
435.
by Const-me 3y ago
For implementing stuff from scratch, if you use Windows you could try my C# based library for that: https://github.com/Const-me/Cgml/ It’s vendor agnostic, so HLSL instead of CUDA or Triton. Here’s the compute sha
436.
by Const-me 3y ago
From the context I assumed these are the things I knew as “non-functional requirements”. Basically, random stuff which doesn’t have specific associated use cases. Depending on the project, they might specify environment, setup and deploymen
437.
by Const-me 3y ago
The only important numbers are processing power TFlops, and memory bandwidth GB/s. If your compute shader doesn’t approach the theoretical limit of either computations or memory, it doesn’t mean there’s anything wrong with the GPU. Her
438.
by Const-me 3y ago
The performance on integrated GPUs is not stellar, but it should work. On my AMD Ryzen 5 5600u with dual-channel DDR4, I’m getting 2 tokens/second. My friend with Intel Core i3 and single-channel memory was getting 1 token/second.
439.
by Const-me 3y ago
Mobile games are only harmful to a relatively tiny group of addicted gamers, while internet ads have very serious consequences acting on society as a whole. I don’t think mobile gaming companies have a potential to destroy free press, or ne
440.
by Const-me 3y ago
Since they mentioned they’re working on Mistral-7B, I’d like to note that my GPU-only implementation of Mistral uses slightly over 5GB of VRAM: https://github.com/Const-me/Cgml Runs pretty good on most consumer-grade G
441.
by Const-me 3y ago
> that is not going to happen It’s even worse. They happily introduce breaking changes for some things, but not others. I have once wasted days of work updating a codebase to build with C++/20 because the new standard suddenly chang
442.
by Const-me 3y ago
For some problems, sticking to the SIMD subset exposed to Java might be slower than no SIMD at all. Example: https://news.ycombinator.com/item?id=36624240 The linked C++ code is easily portable to C# because hardware intrin
443.
by Const-me 3y ago
RDNA3 FLOPS numbers are real. Shader compiler happily uses VOPD for quite a few things including FP32 FMA. An independent researcher measured 62.9 FP32 TFlops (boost) in Vulkan on 7900XTX, see “Full GPU Throughput – Vulkan” section on that
444.
by Const-me 3y ago
Vector API is very far from being hardware intrinsics. Unfortunately for Java programmers, it’s merely a least common denominator of SIMD instructions across different ISAs. This makes the feature very limited by design, IMO borderline usel
445.
by Const-me 3y ago
You’re comparing general-purpose computations to a proprietary feature with limited applicability. For example, in my Mistral implementation the most expensive compute shader is handling matrices compressed with a custom codec unknown to te
446.
by Const-me 3y ago
My laptop has a single GPU inside Ryzen 5 5600U i.e. unified memory, all consoles also have unified memory. These devices are fine with traditional GPU programming model, where shaders only have access to well-shaped pieces of memory access
447.
by Const-me 3y ago
> Are the weights encoded on every startup? Not really, that code only runs while importing the PyTorch format. See readme for the frontend app: https://github.com/Const-me/Cgml/tree/master/Mistral&#x2
448.
by Const-me 3y ago
Why would you want OpenCL? Pretty sure D3D11 compute shaders gonna be adequate for a Torch backend, and they even work on Linux with Wine: https://github.com/Const-me/Whisper/issues/42 Native Vulkan compute s
449.
by Const-me 3y ago
> Vulkan Compute backends for numerical compute (as typified by both OpenCL and SYCL) are challenging Microsoft has an offline dxc.exe compiler which compiles HLSL to Spir-V. Also, DXVK has a JIT compiler which recompiles DXBC byte codes
450.
by Const-me 3y ago
> Can you link to benchmarks or other analysis supporting this claim? Current generation nVidia: https://en.wikipedia.org/wiki/GeForce_40_series#Desktop Current generation AMD: https://en.wikipedia.org&#x

More ›