Argonautlabs

4 ms·

1.
by Argonautlabs 2mo ago
Its all about AI these days.
2.
Ask HN: DeepSeek v4.1 Flash on Local hardware, what tok/s do you see?
4 points by Argonautlabs 2mo ago|0 comments
3.
Show HN: GLM-5.3 744B at 4 tok/s on a MacBook Pro, experts streamed from 4 SSDs(github.com)
2 points by Argonautlabs 2mo ago|0 comments
4.
by Argonautlabs 2mo ago
Different angle on the same model: the full GLM-5.3 (744B MoE, 4-bit experts, 434 GB on disk) runs on a single MacBook Pro M5 Max with 128 GB by streaming the experts from NVMe SSDs instead of keeping them in memory. One drive gives about
5.
by Argonautlabs 2mo ago
We stream DeepSeek-V4.1-Flash (518 GB, 4-bit) from NVMe on a 128 GB M5 Max, because it doesn't fit in RAM. prompt processing steady decode upstream, internal 16.23 10.59 ours
6.
Deep Seek v4.1 M5 Max at 17 tokens/s(github.com)
10 points by Argonautlabs 2mo ago|2 comments
7.
by Argonautlabs 2mo ago
44 t/s prefill and 17 t/s decode on M5 Max 128gb Deep Seek V4.1 Flash (4 bit)
8.
DeepSeek v4.1 Flash (518GB, 4-bit) on a 128GB MacBook: 2.7x prefill, 17 tok/s(github.com)
1 points by Argonautlabs 2mo ago|1 comments
9.
by Argonautlabs 2mo ago
Often Deep Seek V4 flash or Qwen should be enough. I wanted to see whether Kimi runs at all on one machine with the full record published, and for long multi-table finance reasoning I wanted the strongest model I could keep on the machine.
10.
by Argonautlabs 2mo ago
OWC Express 1M2 (Thunderbolt 5, single M.2 NVMe) — three of them, two on the Mac's own ports and one behind an OWC Thunderbolt 5 hub since the machine has three ports. Each enclosure tops out at about 7.1 GB/s on whole-file reads
11.
by Argonautlabs 2mo ago
In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac). Two things carry over: the experts are read from disk per token either way,
12.
by Argonautlabs 2mo ago
Just internal 2Tb Macbook M5 Max drive it came in at roughly half the four-drive speed (0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load https:/&
13.
by Argonautlabs 2mo ago
Memory, not the model. The KV cache on this engine grows about 2.8 MiB per token of context, and the machine's 128 GB is already holding the 50.7 GiB resident trunk plus reserves the speculative-verify path needs (we found the hard way
14.
by Argonautlabs 2mo ago
Thank you. Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.
15.
by Argonautlabs 2mo ago
Fair, and we didn't measure it. Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand. The current configuration admit
16.
by Argonautlabs 2mo ago
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token, because prefill currently reads each layer&#x27
17.
by Argonautlabs 2mo ago
Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s. W
18.
by Argonautlabs 2mo ago
Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum. Going fro
19.
by Argonautlabs 2mo ago
SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
20.
by Argonautlabs 2mo ago
Thank you! Here is the short version: Kimi K3, 2.78T parameters, ~1.45 TB of MXFP4 experts streamed from four SSDs on an M5 Max / 128 GB. 1.00 tok/s steady over 512 tokens, 1.13 over 128, ~6.3 min to first token on a 512-token
21.
by Argonautlabs 2mo ago
It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.
22.
by Argonautlabs 2mo ago
Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor. My use is day/week/month end review — go through t
23.
by Argonautlabs 2mo ago
Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NO
24.
Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs(github.com)
277 points by Argonautlabs 2mo ago|155 comments