wolttam

7 ms·

91.
by wolttam 3mo ago
Hy3 lacks the DSv4 architecture's KV Cache efficiency. Whereas I can run DSv4 Flash on a pair of DGX Sparks and have enough memory left over for 3M tokens of KV cache, with Hy3 (quantized to FP4), there is only room for ~130K tokens of
92.
by wolttam 3mo ago
I'm looking ahead to the next wave of open-weight models that are as efficient as DSv4 (which is really efficient), and have been heavily distilled on GLM 5.2 (which is trivial, given it is open weight)
93.
by wolttam 3mo ago
To expand on Chinese models: - DeepSeek - GLM (Z.ai) - Minimax - Kimi (Moonshot) - Hy3 (Tencent) - Qwen (Alibaba) (Each one of these with weights available to download and run locally)
94.
by wolttam 3mo ago
It's the native DeepSeek V4 Flash quant which is released in MXFP4. You have enough RAM left over with 2 sparks for ~2M tokens of working KV cache.
95.
by wolttam 3mo ago
I was recently using self-hosted DeepSeek V4 Flash to poke around the DSpark implementation in vLLM (well outside of my domain) I did wonder if I was doing anything Fable would have flagged - sounds like yes.
96.
by wolttam 3mo ago
> we pass the PFAS to others Is there no way to filter them out of withdrawn blood?
97.
by wolttam 3mo ago
Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.
98.
by wolttam 3mo ago
Using regular DeepSeek-V4-Flash, I see 2000tok/s prompt processing and from 40 to 50 tok/s generation. Performance doesn't drop much at long contexts, DSv4 is really nice for that. With DeepSeek-V4-Flash-DSpark (Deepseek&#x27
99.
by wolttam 3mo ago
I run DSv4 Flash at home on 2 DGX Sparks and am pretty sure there is no more cost effective way for me to do so. I'm not interested in running smaller models.
100.
by wolttam 3mo ago
Are you sure about that? High memory speed is great for dense models, or when serving at high concurrency. However for local single-user setups, it's often better to have access to more capable/bigger MoE models at reasonable spee
101.
by wolttam 3mo ago
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is". I think "artificial" is actually a pretty good term to describe the output of the models. That output does app
102.
by wolttam 3mo ago
Case in point: Metaverse.
103.
by wolttam 3mo ago
Ornith is a fine-tune/continued post training (RL) of Qwen 3.5 by Alibaba. The models didn't live up to the claims they made.
104.
by wolttam 3mo ago
> The broad conclusion from the many forms of alignment evaluations described in this section is that Claude Mythos Preview is the best-aligned of any model that we have trained to date by essentially all available measures.[0] [0]: htt
105.
by wolttam 3mo ago
PGP remains the only pretty good privacy (Said tongue-in-cheek, I don’t know if there’s other comparable systems out there)
106.
by wolttam 3mo ago
I would think that this is because those are what is currently available, and that this would change when actually capable humanoids start coming on to the market. It will become possible to re-use existing equipment (providing the largest
107.
by wolttam 4mo ago
We're not going to go down the path of training a bunch of highly specialized models for tasks like "this tractor should tend this field". We're going to (and are already on the way to) train deeply general models that c
108.
by wolttam 4mo ago
One of these is not like the other
109.
by wolttam 4mo ago
I use the one that I've been developing since 2023. It's intended to be used in exactly this spirit! Written in Go, has image support (which has yet to be fleshed out). It supports MCP (unlike Pi), sandboxing (with user-mode netwo
110.
by wolttam 4mo ago
I didn't think they'd actually release a model that was worse than the open-weight frontier and at a higher price-point. Wow.
111.
by wolttam 4mo ago
It did not write it for me, I used it to add a feature I wanted. It's a pretty small and understandable codebase, in fact :)
112.
by wolttam 4mo ago
2000 t/s prompt processing and 40-50 t/s generation. We should see 60-70 t/s generation with DSpark support solidifying in vLLM in a few days Recent discussion on DSpark: https://news.ycombinator.com/item?id=4
113.
by wolttam 4mo ago
I started mine from scratch in 2023 because I wanted to use LLMs from a terminal and there was nothing else compelling at the time (nowadays there is pi and opencode) Harnesses are/can be incredibly simple things, not much more than a
114.
by wolttam 4mo ago
I self-host DeepSeek V4 Flash on 2 DGX Sparks (approx. $10k) I expect DeepSeek V4 Flash (or an equivalently sized model) to reach parity with GLM 5.2 some time this year (this based on DeepSeek V4 Flash launching at GLM 5.0 parity[0], and G
115.
by wolttam 4mo ago
I used Claude Code for a month because my boss gifted me a sub and wanted me to try it. I used that month to complete a work project and then beef up my personal harness so I'd never have to deal with Anthropic (and these sorts of shen
116.
by wolttam 4mo ago
You can buy a pair of DGX Sparks and run Deepseek V4 Flash at ~60-70TPS (once DSpark support matures over the next few days). That will get you a near-frontier experience. DSv4 Flash launched in April with capabilities on par with GLM 5.0,
117.
by wolttam 4mo ago
Have to ask: did you try 'xhigh' thinking effort with Flash? I also found it nearly unusable on just 'high', but on 'xhigh' it's nearly equivalent to Pro's 'high'.
118.
by wolttam 4mo ago
I self-host Flash actually, but yeah. When I use their API I use it knowing that they probably train on the data, and knowing that it's probably used to improve future iterations of their models. But I use their API extremely rarely la
119.
by wolttam 4mo ago
This is an efficiency improvement that significantly lowers the amount of RAM you have to look at, on average, during decode. It should improve performance on most hardware because most LLMs are memory bandwidth bound during decode.
120.
by wolttam 4mo ago
I found Flash to be a bit shaky as well until I started using it in xhigh/max thinking effort, then it became my daily driver. It runs quite well on a couple of DGX Sparks. I still wish it was a little better, but there's hope for

More ›