petu

6 ms·

31.
by petu 2mo ago
Maybe some were fixed, but: 1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just c
32.
by petu 2mo ago
There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that &qu
33.
by petu 2mo ago
They show that CS-4 can't really do batching (or rather it can't properly benefit from it), total throughput barely changes (25%?): https://cdn.sanity.io/images/e4qjo92p/production/6a132331880... Wh
34.
by petu 2mo ago
Single 3090 under llama.cpp: | model | size | test | t/s | | ------------------- | ------- | ------ | ---- | | gemma4 31B Q4_0 | 16.1 GB | pp2048 | 1248 | | gemma4 31B Q4_0 | 16.1 GB | tg512 | 4
35.
by petu 2mo ago
> and it needs 10 times more RAM. More like 3-6. Qwen 27B full quality is FP16. So 54GB. In practice most inference providers would serve FP8, so 27GB. DeepSeek V4 Flash in full quality is mostly FP4. ~167GB official release. So Deepseek
36.
by petu 2mo ago
> Qwen models are slower in tokens/s, compared to similarly sized gemma4 and others No? Gemma 31B and Qwen 27B are about the same speed. Gemma 26B-A4B and Qwen 35B-A3B are about the same speed.
37.
by petu 2mo ago
I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other. As I understand (useful) concurrency for MoE require
38.
by petu 2mo ago
What was your prompt length? It's possible it was just processing it and it's likely not fast on your setup.
39.
by petu 2mo ago
You need ~24-26GB for basic setup (17-19GB model + 128K 8 bit context), so you can, but not much memory would be left for doing anything else on that machine. And even then it would run at like 5-10 t/s (due to relatively low memory ba
40.
by petu 2mo ago
> and more memory/computationally expensive to run any way I look at it. Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference no
41.
by petu 2mo ago
Deepseek's DSpark does dynamically adjust speculated token count per user/completion. https://arxiv.org/abs/2607.05147 But they do so to maximise total throughput, I don't think there's reason to do
42.
by petu 2mo ago
Unsloth one is gguf for llama.cpp (and some other on-device engines). So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
43.
by petu 2mo ago
I have no idea, but I've assumed that batching can't work on Cerebras. Batching works because of severe memory bottleneck, but Cerebras whole thing is serving models out of "L1 cache" (?).
44.
by petu 2mo ago
Two Chinese models are open weight. What interesting going for Grok that it would overshadow all bad PR?
45.
by petu 2mo ago
Whoever runs their Twitter wouldn't confirm / promise anything beyond 27B, but didn't say no either. So maybe in a month, maybe not.
46.
by petu 2mo ago
Someone took modelscope countdown URL for 2.4T model and tried 27B path: https://modelscope.cn/models/Qwen/Qwen3.8-27B
47.
by petu 2mo ago
How PCIe card is better? "Static hardware" is still fully featured computer with lots of RAM, could be easily reused for other purposes.
48.
by petu 2mo ago
Spoofing a number can be a felony on it's own. Even if there's no explicit laws around that, who's getting billed for the calls?
49.
by petu 2mo ago
Phone number always belongs/registered to someone?
50.
by petu 2mo ago
"confidence": 0, so I guess you could threshold it
51.
by petu 2mo ago
> but I haven't seen any indication that training was quant-aware readme on huggingface says they've benchmarked the quants -- for 17GB quant reported 1% avg loss across 15 benchmarks (sadly no breakdown). I assume that's
52.
by petu 2mo ago
Qwen3.6 is very token inefficient with it's thinking. Quantized versions often get into loops. Glimmer is trained with 4 effort levels, not just thinking on/off. Maybe it's more token efficient in general. There's offici
53.
by petu 2mo ago
Datacenter hardware can batch at large scale, probably over 90% more energy efficient per token than a MacBook.
54.
by petu 2mo ago
3090/4090 probably would do 40 t/s, for 5090 75 t/s is shown in the blog.
55.
by petu 2mo ago
If algorithm used is static and GPU-friendly, then what stops bypass at scale?
56.
by petu 2mo ago
This page has it, scroll to "Intelligence" header (not the highlights one, but second on the page / with black square) and click "Agentic Index"
57.
by petu 3mo ago
Hm, I asked 5.6 Sol to calculate/research and it came up with way less optimistic lifespan than Gemini (that I've asked last year). It claims that each individual page read induces read disturb across whole block. And references
58.
by petu 3mo ago
There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours. Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
59.
by petu 3mo ago
Does Claude still say it's Deepseek, when asked? https://news.ycombinator.com/item?id=49082022#49087112 How is that important? Maybe it does, so what?
60.
by petu 3mo ago
Weights are yet to be released (maybe in 24H, Deepseek has track record of releasing same day). https://github.com/antirez/ds4 is often mentioned for DS4F on Mac, but 64GB is likely not enough to achieve reasonable spe

More ›