cmrdporcupine

4 ms·

1.
by cmrdporcupine 6d ago
Everything is combined and uneven, including the opinions of hackernews commenters? There is no single opinion, and clearly no single Chinese approach. Also Chinese labs are in particular very careful about anything which be used to create
2.
by cmrdporcupine 7d ago
I'll just say that even though I was poor and without a job and living on unemployment insurance for a year... The implosion of hype after the .com crash was actually kind of a ... relief.
3.
by cmrdporcupine 7d ago
Jev is only on people's mouths because they made friends with venture capitalists and used the publicity blowhorns that come with that. Whereas the other guy went through the unglorious but formerly respectable path of publishing softw
4.
by cmrdporcupine 7d ago
Ok I tried it. And it's faster. Though I'm still in verifying quality phase. But only works for Gemma4, or other attention-only models. (i.e. not the Qwen3.8 models I had working with the other way) I'm getting 63ms per answe
5.
by cmrdporcupine 7d ago
I think you're right. Better. I will have to think through if it would be faster or slower. If understand what you're getting at with this.. broken analogy... What I described was -- we have a bunch of orders to the kitchen, all o
6.
by cmrdporcupine 7d ago
Or they'll just build a competitor in RISC-V instead and that's fine. Except the problem is not restricted to the actual ISA or its HDL implementation, etc. It's even just getting space / time in a fab at that advanced o
7.
by cmrdporcupine 8d ago
Only took them a year and a half of everyone complaining to finally do the right thing. Congrats.
8.
by cmrdporcupine 8d ago
So the model normally (like during a normal decode) takes its final hidden vector and multiplies it by the entire vocabulary head -- so like about 250K rows -- to produce one logit per possible next token (and then so on and so on...) Inste
9.
by cmrdporcupine 8d ago
fwiw, w/ gemma4 -- non-diffusion -- I get about 170ms for a single question -> answer and then an additional ~33ms on adding more. While I see people reporting 300ms for this vLLM PR on same hardware (Spark.) So I don't see the
10.
by cmrdporcupine 8d ago
> It seems by sharing KV cache all questions will be visible Yeah, this is partially why in my approach I've done this instead, and not used diffusion model: 1. Convert the state into one shared prompt. 2. Run that shared prompt thr
11.
by cmrdporcupine 8d ago
This PR is interesting but it's making the assumption that what Jev has done is based on a diffusion model or that a diffusion model is superior for this work. Which may or may not be the case. If I understand it though it does mean yo
12.
by cmrdporcupine 8d ago
and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways. why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU&#
13.
by cmrdporcupine 8d ago
the models will come. or be fine tuned
14.
by cmrdporcupine 8d ago
There's also prior art. Or probably, anyways. https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_liter... Not only is it replicable as you say, things like it already exist(ed). The important
15.
by cmrdporcupine 8d ago
Basically it's: skip decode, just do prefill then do some measurements. That's the crude description anyways. And prefill is way faster on GPU type hardware.
16.
by cmrdporcupine 8d ago
It's good people moved this quickly on this stuff. The thing is that the openjev stuff is a ... bit ... of a hack (a good one though): It does this: 1. Send a throwaway request containing the shared state. 2. Hope SGLang keeps that tex
17.
by cmrdporcupine 8d ago
Performance for this kind of thing should be best on any hardware that has high prefill speeds. As basically this is "do prefill only, measure scores, skip decode entirely". I don't know how the Mac stuff compares on that fro
18.
by cmrdporcupine 8d ago
What are you getting for prefill?
19.
by cmrdporcupine 8d ago
What I'd love to see is this done for DS4.1 Flash. That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
20.
by cmrdporcupine 9d ago
Beyond the defense sector, the US state has always intervened publicly and privately to mediate and balance competing corporate "private" interests. It has also periodically aggressively helped subsidize, bankroll, and enforce the
21.
by cmrdporcupine 9d ago
The nature of the US state is such that the distinction between nationalized and not is almost meaningless. Like Lockheed-Martin or Boeing, etc. there's just interpenetration between the corporate boardroom and the state. They act in
22.
by cmrdporcupine 9d ago
It comforts Americans to believe their exceptionalism is both persistent/eternal and fully justified.
23.
by cmrdporcupine 9d ago
OpenAI did recently get into how they had been building their own hardware and doing RSI with it. That's more than Anthropic has done though.
24.
by cmrdporcupine 9d ago
My thoughts on this, as a person who has recently coming around to working in this space is that up to now the convenience and "simplicity" of working in CUDA as it is has been a giant moat for NVIDIA. Having a whole toolchain wit
25.
by cmrdporcupine 9d ago
Chill man, holy crap. I never told him not to learn. I was speaking directly to his "good thing LLMs don't know how to use this yet". What the hell is your problem? Getting on the Internet and hurling personal insets around.
26.
by cmrdporcupine 9d ago
Sad to break it to you, but... I had LLMs write a pile of cuda-rust code and they were quite competent at it. Ported a bunch of (C++) CUDA kernels over, and ground away on them til they got equivalent performance https://github.c
27.
by cmrdporcupine 9d ago
i love you too, buddy. big hugs.
28.
by cmrdporcupine 10d ago
There are no good guys. The foxes are in charge of the hen house in every country. E.g. Carney is using all of this as a chance to completely gut every climate change initiative possible and slash corporate taxes, shore up existing Canadian
29.
by cmrdporcupine 10d ago
Thing is that GLM 5.3 is many multiples the cost to run, and slower. I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond
30.
by cmrdporcupine 10d ago
That's my point about the harness.

More ›