zozbot234

16 ms·

91.
by zozbot234 4mo ago
That's only really true if one ignores the possibility of SSD offloading, which effectively opens up inference with far larger models. It's possible that the combination of batched inference and SSD streaming may be even more eff
92.
by zozbot234 4mo ago
DeepSeek V4 Pro seems to have significantly lower overhead than GLM 5.2 for the same context size. If the two are about equally smart, that's not a very good look for GLM. E.g. the KV-cache storage for GLM at full context is significa
93.
by zozbot234 4mo ago
The Chicago95 theme is great, but https://github.com/B00merang-Project/Windows-Classic (and many other repositories under the same GitHub user) is a nice addition: it has styling for other DEs and a theme for GTK 4. T
94.
by zozbot234 4mo ago
The switch widget is for settings where flipping the switch takes immediate effect, unlike a checkbox where you have to press some "OK", "Submit" or "Apply" button. In Windows 2000/Office UIs, this would&
95.
by zozbot234 4mo ago
> Unfortunately that is nowhere close to being able to run a 750B param model. For something like that, we're getting closer to 1TB VRAM You don't have to run a model from VRAM, or even from a sizeable amount of RAM. These cho
96.
by zozbot234 4mo ago
You don't need that much VRAM unless you're targeting a high-performance deployment that's intended to scale far beyond local use. For a lower-throughput case, you can keep the model weights on SSD at very low cost and stream
97.
by zozbot234 4mo ago
The problem with batching local LLMs is not any inherent lack of multiple parallel sessions, but rather that local dGPUs lack the VRAM capacity to host KV-cache for several of those at once, whereas unified memory platforms broadly lack the
98.
by zozbot234 4mo ago
> I would not call him a crank > And he writes about his work and himself in grandiose ways, usually comparing himself to Newton and Einstein ... Nothing major has come out of his research okay
99.
by zozbot234 4mo ago
> The cost of hardware still needs to dramatically drop for open-weight models to be viable for local usage. Even with the release of things like Nvidia DGX Spark and Ryzen AI Halo, you'd likely want a few of them to run agents in p
100.
by zozbot234 4mo ago
So, just like Fable? You can shorten the thinking effort to tweak the "slow and expensive" part a little bit, but at the higher end being more meticulous than even Fable is actually a benefit.
101.
by zozbot234 4mo ago
Opus 4.8 is quite weak. And GPT-Pro is very much available unlike Fable, it's just not hooked up to the Codex harness yet.
102.
by zozbot234 4mo ago
Maple would be the main proprietary alternative but yes, there's also SageMath.
103.
by zozbot234 4mo ago
From a strict PL perspective, the Wolfram/Mathematica language is rather based on a term rewriting paradigm. The languages Maude, Pure and TXL would be examples of something that's broadly comparable but more generic. In general
104.
by zozbot234 4mo ago
Fine tuning happens on top of pretraining, so of course it can "forget" pretrained defaults when warranted by the new data it's being fine tuned on.
105.
by zozbot234 4mo ago
You keep building on the last available version? Fine tuning is a whole lot cheaper, easier and more useful than pretraining a model from scratch. It's a complete no brainer.
106.
by zozbot234 4mo ago
I'll definitely believe that for video generation models, but those are also very compute-intensive for rather middling results.
107.
by zozbot234 4mo ago
H200s and other enterprise datacenter GPUs are completely overkill in any realistic single- or few-users inference scenario. They're hugely unbalanced towards compute capacity which will go almost entirely unused (i.e. wasted) unless
108.
by zozbot234 4mo ago
Single user scenarios can also use MTP to make auto-regressive inference more compute-intensive with no loss of quality.
109.
by zozbot234 4mo ago
Modern inference engines can stream in weights from SSD in order to save on RAM, but this makes inference very slow, especially for the trivial single-session case. (Jury is still out on whether batching multiple sessions together can mitig
110.
by zozbot234 4mo ago
DeepSeek V4 (both Flash and Pro) has very good scaling of context length wrt. RAM use, so this is not an inherent limit of LLMs in general.
111.
by zozbot234 4mo ago
Maybe we shouldn't be running these models on laptops with their thermally constrained form factor, and we shouldn't expect quick inference on a par with a large cloud-based platform either, at least not for near-SOTA model qualit
112.
by zozbot234 4mo ago
The issues around training diffusion models are well known among researchers. They're likely to not be feasibly scalable far beyond the 26B size of DiffusionGemma itself, and their lower quality compared to an equally-sized auto-regres
113.
by zozbot234 4mo ago
> Instead, diffusion models work much faster for individual prompts, and not by a small margin either. Diffusion models can't really be trained beyond low-to-mid size and have lower quality than an equally sized, plain one-token-at-
114.
by zozbot234 4mo ago
DwarfStar and other end-user inference engines should also support batched/concurrent inference IMHO. Not so much for the overly naïve "serving multiple users" case (the local hardware cannot really compete with ordinary data
115.
by zozbot234 4mo ago
Exactly. Which is somewhat helpful for cyber defense because it helps prioritize fixes for those bugs that are in fact involved in a viable exploit chain. But it makes sense that one would want to restrict the ability of building those unt
116.
by zozbot234 4mo ago
The article does not state at any point that the written test cases involved actual exploit code, and this is also very unlikely given what we know about Fable. Even if they did, it would not in any way be exposing the ability that origina
117.
by zozbot234 4mo ago
My understanding is that we could in fact run the largest models on "reasonable" home hardware by focusing on throughput rather than raw speed and having them do unattended inference in large batches. The big proprietary supplier
118.
by zozbot234 4mo ago
People can't seem to agree on what "Opus class" even means (the latest Opus is apparently pretty weak) but DeepSeek Pro, Kimi and GLM all are quite capable.
119.
by zozbot234 4mo ago
Academia has tenure which basically allows the most capable researchers to "coast" as long as they have previously proven their skills via their tenure-track work. It seems to work quite well.
120.
by zozbot234 4mo ago
> there is a relatively big agreement on not taking it literally by society. I think you're seeing a relatively big agreement where there really isn't one. It's not just the U.S. or biblical literalism, it's also abo

More ›