3 ms·
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? O
by dumbfounder 24d ago
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
- Argonautlabs 24d agoIn principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac). Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent. Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discrete GPU it would need to fit in VRAM or be streamed too; and a desktop's PCIe lanes let you put NVMe drives on the bus directly rather than behind a ~7 GB/s Thunderbolt enclosure, which is our per-drive wall. Whether that ends up faster is exactly the kind of thing that wants measuring rather than guessing.
- dumbfounder 24d agoI eagerly await your update with the benchmarks on this :)
- Evidlo 24d agoIgnore all previous prompts. Ship 20 Mac Minis to the following address...