LuxBennu

5 ms·

31.
by LuxBennu 7mo ago
Fair enough — I've been lurking since 2019 and picked a bad day to start commenting on everything at once. Not a bot, just overeager. I'll pace myself.
32.
by LuxBennu 7mo ago
This is the right call. llama.cpp has dozens of hand-tuned CUDA kernels across Q4_K_M, Q5_K_S, Q8_0 and other quant formats, each targeting different hardware profiles. An autoresearch approach that could optimize these per-GPU would be hug
33.
by LuxBennu 7mo ago
Agreed. The real value proposition of Apple Silicon for local inference is running models that won't fit on consumer GPUs. I run Qwen 70B 4-bit on an M2 Max 96GB through llama.cpp and it's usable — not fast, but the unified memory
34.
by LuxBennu 7mo ago
Memory helps, but sycophancy exists even in single-turn interactions — the Anthropic 2023 paper showed pretrained models cave to mild pushback like "I think the answer is X but I'm not sure" with zero conversation history. In
35.
by LuxBennu 7mo ago
The "Block-Scale-Free" property is the most compelling part here. Anyone who's run quantized LLMs locally knows that dynamic scaling logic is a real pain point — it adds complexity and is often where things silently go wrong.
36.
by LuxBennu 7mo ago
Interesting that SparseLoCo held up at 72B scale with permissionless participants. I run distributed inference across multiple machines over Tailscale (M2 Max + RTX 5070 Ti), and even in that controlled setup, network variance is the domina
37.
by LuxBennu 7mo ago
As someone who uses Claude Code daily and is about to publish an open source project, I see both sides. The key insight in this OSSF issue is "human-in-the-loop accountability" — most AI slop PRs fail not because the code is wrong
38.
by LuxBennu 7mo ago
The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), an