wren6991

9 ms·

121.
by wren6991 4mo ago
Why do you need logits to distill? Those are at least tokenizer-dependent, and different models use different tokenizers.
122.
by wren6991 4mo ago
Apple muddied the waters by calling them "neural accelerators" but it seems like what they actually added in the M5 generation is tensor instructions for the existing GPU cores. It's not a separate accelerator like the ANE. l
123.
by wren6991 4mo ago
GLM-5.2 meets my needs for "thinky" tasks, which for me is code and documentation reviews, technical chats and rubber ducking. (I've tried agentic coding and gone back to writing by hand; besides ethical and skill atrophy con
124.
by wren6991 4mo ago
That's fine, I already cancelled my subscription after they admitted to using PEFT to selectively and silently make their models dumber when working in certain technical fields. [ref: section 1.5 of Mythos/Fable 5 system card, ht
125.
by wren6991 4mo ago
Do they have logits for all of the Wikipedia etc that they've scraped?
126.
by wren6991 4mo ago
> Proper distillation requires access to the logits Why do you need logits? Can't you just train on cross-entropy loss of the model against the hard decision, like you do in regular pretraining? There are definitely current-gen open
127.
by wren6991 4mo ago
> Doesn't going straight to kana actually kind of obscure the relationship between nomu and nomi that they both begin nom-? No, because nom- is not a sound that exists in Japanese phonetics. nomu/nomi share the initial の, and む
128.
by wren6991 4mo ago
Thanks, this was news to me. TIL. I wasn't really going for kunreishiki, more ワープロ, aka "whatever is the least typing to make the desired characters appear". I think typing in IME is how most people actually interact day-to-d
129.
by wren6991 4mo ago
> now let's try to apply the rules: > hanas* + (i)masu = hanasimasu (wrong!) I had to stare at this for a while to figure out why the author thought it was wrong. "si" is rendered as し on every IME keyboard I've ev
130.
by wren6991 4mo ago
Yeah, RTK is problematic because of its focus on associations between kanji and arbitrary English keywords, many of which are poorly chosen and... oh it's an LLM thing.
131.
by wren6991 4mo ago
Ok I'll bite. What's the contradiction?
132.
by wren6991 4mo ago
> Yes, but presumably the authors are suggesting broader application than just caching a system prompt The authors of the OP paper "Can I Buy Your KV Cache?" explicitly disregard anything involving KV not rooted at 0: >>
133.
by wren6991 4mo ago
If you want the actual maths instead of handwaving, I recommend: https://arxiv.org/abs/2207.09238 For something gentler, 3Blue1Brown: https://www.youtube.com/watch?v=eMlx5fFNoYc (this is part of a seri
134.
by wren6991 4mo ago
Prefix caching is already widely deployed by all providers, right? llama.cpp does it. vLLM does it. I'm sure everyone hosting LLMs for a living does it. This paper seems to focus entirely on prefixes (i.e. the prefilled content is root
135.
by wren6991 4mo ago
A steaming pile of webslop. Thanks Claude. Music is cool though
136.
by wren6991 4mo ago
They've publicly apologised for the invisible PEFT that deliberately makes the model dumb on some tasks. Whether they still do it, or will once again do it in future in more subtle ways, is something we can't verify. Personally I
137.
by wren6991 4mo ago
> you could do substantially the same thing by just prompting/LoRA finetuning the model to produce a single-token output ("yes" or "no") You could probably achieve this with logit masking. Or equivalently, compar
138.
by wren6991 4mo ago
The OSS-Fuzz section is interesting. They compare it to their other models but carefully avoid comparing it to, you know. Fuzzing.
139.
by wren6991 5mo ago
> Even local AI use cases don’t substantially or meaningfully benefit from faster memory, at least to average consumers. I'm not sure what you mean by this. Memory bandwidth is the main bottleneck for single-user decode. The bottlen
140.
by wren6991 5mo ago
If the random blob is running inside of a real sandbox (Landlock/Bubblewrap, VM, ...Docker) then I would take the blob, because I can reason about its capabilities without inspecting its internals. The LLM can run curl as much as it wa
141.
by wren6991 5mo ago
Right, runtime is so broad that it's hard to say something has "no runtime". libgcc + your choice of crt0 is a C runtime, and the JVM is a Java runtime. That's a huge spectrum. It's worth being charitable in your in
142.
by wren6991 5mo ago
> The first value for their downstream (OpenAI, Anthropic, etc) is REST call vs arbitrary code execution. Is this an advantage? Phrased differently, every MCP that could have been a CLI call is a new opportunity for sandbox escape.
143.
by wren6991 5mo ago
> they don’t compute with them Isn't this exactly what chain-of-thought does? It's doing computation by emitting tokens forward into its context, so it can represent states wider than its residuals and so it can evaluate functi
144.
by wren6991 5mo ago
Qwen models know about Tiananmen Square but they are post-trained to refuse to talk about it. The decensored versions will happily chatter away about it. Similarly, try talking to Nemotron about Epstein and see how quickly it shuts down.
145.
by wren6991 5mo ago
I've found Rust's safety guarantees to be less useful for slop-generated code because LLMs can always fight their way through the borrow checker by spamming enough Arc<Mutex<Arc<Mutex<...>>>> and clone() ev
146.
by wren6991 5mo ago
Requires a secure origin. If you serve a local (but non-localhost) SPA over HTTP then you're blocked from using crypto.subtle.digest. At least, that is one reason I have seen a hand-rolled SHA-256 deployed. Edit: oh, and it forces asyn
147.
by wren6991 5mo ago
Noting the permissive license, could we be witnessing the birth of a new Utah teapot or Sponza atrium?
148.
by wren6991 5mo ago
For 16 GB I would look into running Qwen3.6-35B-A3B (MoE) with some layers offloaded to CPU.
149.
by wren6991 5mo ago
There's an antipattern where everyone wants to invent new interfaces to connect things LLMs when CLI tools are already right there, transparent, and usable by humans as well as LLMs. I think it's partly the origins in web chat app
150.
by wren6991 5mo ago
I also have a script to find the best LLM for your hardware. Here: echo "Qwen3.6-27B"

More ›