throwdbaaway

6 ms·

1.
by throwdbaaway 12d ago
Back to the topic about cuda moat, in the slide with title "Problems with Coopmat1", the current frontier open models have absolutely no issue with: * manual pipelining * shared memory staging * tiling * bounds checking
2.
by throwdbaaway 12d ago
So when I said "a couple of nvidia engineers", I indeed meant Jeff. VK_KHR_cooperative_matrix - embrace? VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com&#
3.
by throwdbaaway 13d ago
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with t
4.
by throwdbaaway 13d ago
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks li
5.
by throwdbaaway 13d ago
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX. Few problems I noticed: * coopmat2 from nvidia is the
6.
by throwdbaaway 29d ago
Hold on.. the routed experts are in FP8 now? Previously they were in BF16. Nice, this shall cut my download time by half!
7.
by throwdbaaway 1mo ago
Sounds like that's what z.ai did to get GLM-5.3-Flash running on Huawei chips.
8.
by throwdbaaway 1mo ago
Exactly. Coding for inference is solved. CUDA is no longer a moat.
9.
by throwdbaaway 1mo ago
> $20k workstation, best case: $15k M5 Ultra 512GB, 36-month amortization, ~$440/mo. Runs a GLM-5.3-class model at ~30 tok/s. Saturated 24/7 it produces roughly 58M output tokens/month. For agentic coding, ~90% of the
10.
by throwdbaaway 1mo ago
As for KV cache quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme... ).
11.
by throwdbaaway 1mo ago
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ... If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once
12.
by throwdbaaway 1mo ago
Reproduced on the CUDA stack right? Let's say DeepSeek is being forced to use the CANN stack, and the new pricing reflects the cost when 100% of inference is done with Huawei chips. Then, I suppose we can infer that: * CANN stack is 1.
13.
by throwdbaaway 2mo ago
Objectively speaking, the 2 bit quant from antirez has very low accuracy. Meanwhile, his 4 bit quant does have decent accuracy, but is a bit pointless by being bigger than the full precision MXFP4 quant. Anyway, they all work fine in practi
14.
by throwdbaaway 2mo ago
They need to get a license from moonshot to provide inference for K3. Probably have to follow the pricing set by moonshot as well.
15.
by throwdbaaway 2mo ago
It works, thanks to https://github.com/ikawrakow/ik_llama.cpp/pull/1911 , which got merged in early June. However, there might still be some issue with the chat template.
16.
by throwdbaaway 2mo ago
I suspect this is why DeepSeek had to introduce the 2x peak hours pricing. The price would be too low otherwise.
17.
by throwdbaaway 2mo ago
Yeah antirez made a lot of big claims in that paragraph. Sounds like a case of AI psychosis.
18.
by throwdbaaway 3mo ago
If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.
19.
by throwdbaaway 3mo ago
That's exactly what I said. They do care when FLOPs are involved. Restoring an old session with 900k tokens will require a lot of FLOPs to reprocess the 900k token. Meanwhile, they don't really care if you use hundreds of millions
20.
by throwdbaaway 3mo ago
Different sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4 , 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take less than 9GB of VRAM for 1
21.
by throwdbaaway 3mo ago
Well I wouldn't call it a low bar, since some of the edits were quite complex. And 1M context in less than 6GB of VRAM is truly impressive, but somehow this gets way less attention than the crappy turbo quant from Google.
22.
by throwdbaaway 3mo ago
While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089 > The challenge is: when you let a session idle for >1 hour, when you come back to it and send a promp
23.
by throwdbaaway 3mo ago
Indeed they are all lossy. Not sure how much they contribute to the quality loss in long context though. I got a 700k session with DSV4 Pro (official API), and the model was still coherent and didn't make any tool call error.
24.
by throwdbaaway 3mo ago
The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion: > That is, for your $100/month fee,
25.
by throwdbaaway 3mo ago
Seems like a pretty pointless post that still centers around output tokens. In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require GPU compute, and DeepSeek has shown that it can be done 50~100x c
26.
by throwdbaaway 3mo ago
And somehow they claimed that it is "lossless".
27.
by throwdbaaway 3mo ago
On ZFS with zstd compression, I am getting 1.34x compressratio for the BF16 weights (across multiple models). Here's the du output for GLM-5.2: $ du -s -BG /cube/models/zai-org/GLM-5.2/ 1099G /
28.
by throwdbaaway 4mo ago
And their disk-based caching is amazing. I got a long 700k context session spanning more than a week, with pauses in between that was longer than a day, and some rewinds mixed in as well. Stats from pi: ↑400k ↓438k R432M 71.9%/1.0M Hal
29.
by throwdbaaway 5mo ago
Hah, that's because the prompt itself was only about 30 tokens. We need a much bigger prompt to properly test PP.
30.
by throwdbaaway 5mo ago
Huh that's not what I gathered from the tweet at all. If I am going to write a five why's analysis, the immediate cause is the LLM wrongly decided to delete a volume, while the root cause is the bad design to co-locate staging and

More ›