zozbot234

9 ms·

181.
by zozbot234 4mo ago
> We need an approach to make sure AI doesn't destroy the world and wipe humanity to extinction. That's easy. Stop training your AIs on cheesy old sci-fi that talks about robot uprisings. In fact, maybe y'all should just
182.
by zozbot234 4mo ago
Fable's costs are twice Opus' and it's clearly quite competitive with GPT-Pro, so that seems like it might be a good option for you if the trigger-happy safeguards aren't too much of a problem. Google has their own &quo
183.
by zozbot234 4mo ago
Edge devices don't just have limited memory bandwidth though, they also have very limited compute. To the extent where you don't actually need all that much batching to saturate their viable compute and run into obvious thermal&#
184.
by zozbot234 4mo ago
Realistically, local/open weight models will always be limited in idiosyncratic world knowledge compared to the proprietary frontier. There's just very limited upside to releasing tens or hundreds of terabytes of open weights for
185.
by zozbot234 4mo ago
It has been quantized to 80GB (2-bit quantization for experts) with limited degradation. Certainly competitive with a 27B model, and especially useful in a size range where few "native" models exist.
186.
by zozbot234 4mo ago
> The K-V cache of a model is intertwined with the model's configuration. I don't think this is true if you simply quantize the model or run it with fewer active experts? The underlying weights would stay the same. You could
187.
by zozbot234 4mo ago
That would be about ~300 tok/s over 72 hours at Claude Fable output token prices? I'm not sure that this passes a sanity test.
188.
by zozbot234 4mo ago
> First, there's been a huge amount of widely accepted research that shows what the most accessible way to design an interface is. Focus-group based and UX research was a lot more intense in the 1990s compared to today, and late 1
189.
by zozbot234 4mo ago
> Why? Why do you think that's the case? Part of the training is balancing load between experts. The training balances expert choice across the entire scope of the model. Experiments have consistently shown that within a given sessi
190.
by zozbot234 4mo ago
Well, that performance figure seems consistent with memory bandwidth on that machine (and its upcoming successor Gorgon Halo; Medusa Halo is projected to be faster) and even on DGX/RTX Spark. You'd get the same outcome on Apple S
191.
by zozbot234 4mo ago
> As far as I can tell you'll have a context limit of about 64k Source? The author has demoed a 100k ctx already, and I can't think of a reason why more wouldn't be supported. RAM is a bit tight but that only matters with
192.
by zozbot234 5mo ago
There's been some very rough experiments with batching on Apple Silicon (and that's not a highly suitable platform since the compute/thermals bottleneck hits sooner than elsewhere) that seem to be broadly consistent with what
193.
by zozbot234 5mo ago
What do you mean by doing the math? If you repeatedly sample n_active experts out of n_total, why wouldn't you expect to get some meaningful probability of reuse/overlap once your batch grows past size 5 or so (for the sparsest Mo
194.
by zozbot234 5mo ago
DeepSWE has been heavily criticized though. https://github.com/datacurve-ai/deep-swe/issues/21 Putting GPT 5.5 on top is the obviously correct part, but everything else about it makes very little sense.
195.
by zozbot234 5mo ago
You need wider batches to get effective reuse of experts in any given layer, but you absolutely can. DeepSeek V4 has tiny KV caches that make this quite feasible. When targeting consumer platforms that only have a limited amount of compute
196.
by zozbot234 5mo ago
The DS4 author has demoed upcoming work on Strix Halo that makes it roughly competitive with the Apple Silicon equivalent (i.e. Pro models with similar memory bandwidth figures, not Max or Ultra). Maybe even a bit faster for prefill, and w
197.
by zozbot234 5mo ago
It's picking strange tasks that don't really play to GPT-Pro's strengths (that model is roughly comparable to Mythos, intended for very hard reasoning and research-level problems) and then completely ignoring quite a few case
198.
by zozbot234 5mo ago
The expensive tokens are output, not input. A useful rule of thumb is that a million tokens per day means about ~10 tok/s on a 24/7 basis.
199.
by zozbot234 5mo ago
Absolutely. The cost comparison is roughly between DeepSeek and Haiku (assuming a reputable Western provider, not DeepSeek's own API) whereas the average capabilities sit comfortably above Sonnet.
200.
by zozbot234 5mo ago
These are the sizes of model weights, not the KV cache. The former are a sparse (for MoE models) read workload that can be streamed from SSD.
201.
by zozbot234 5mo ago
Not a great example: Qwen 9b is a tiny model that outputs barely coherent text in a casual chat, nowhere near comparable to Haiku. But the broader point stands.
202.
by zozbot234 5mo ago
It's a lot easier to ride recklessly on a motorcycle than an ordinary bike. I suppose mopeds/motor scooters (especially electric ones) are the sensible middle-of-the-road option.
203.
by zozbot234 5mo ago
> but fortunately it's trivial to just not write unsafe Rust The point of unsafe Rust is to "leave no room for a lower-level language". When writing lower-level facilities it's not uncommon at all to have to resort to
204.
by zozbot234 5mo ago
Qwen 27B maxes out at a 16GB context. A nice thing about DeepSeek V4, especially Flash, is that its context size stays tiny even at 1M tokens! Which in turn opens up wide batching on common consumer platforms.
205.
by zozbot234 5mo ago
300GB of context for a single session is huge though. Modern local models max out at a whole lot less than that.
206.
by zozbot234 5mo ago
> Again, fails the compliance checkbox. OK, then the not-so-cheap hyperscalers that these enterprises are already relying on. E.g. AWS Bedrock will run these models. It's silly to insist on all three of your checkboxes being ticke
207.
by zozbot234 5mo ago
> But also, the latest DeepSeek is 1.6T parameters. “Choosing” to run this locally is a choice that comes with a seven digit price tag Unless you're specifically thinking about running the model at stock precision in a datacenter en
208.
by zozbot234 5mo ago
> Most non western countries lack the foundations of western democracy, and you can't force that onto them Several East Asian countries managed to democratize successfully up thru the 1980s and are extremely successful today, so thi
209.
by zozbot234 5mo ago
If you buy an Apple iPhone in mainland China, it also won't support the emoji flag for Taiwan. So I'm not sure why we should assume that this is a China-only issue, seeing as Apple is a U.S. based company.
210.
by zozbot234 5mo ago
> How do most Chinese models handle Tienanmen square or discussions on Han superiority? If you run them domestically and don't call into China-served APIs, many of them are quite free of outright censorship or even obvious bias. The

More ›