10 ms·
One could also run it locally on a used dual xeon (or amd-equivalent) server with 512GB RAM, albeit slower, if you have a useful workflow for it that's like "ta
by walrus01 2mo ago
One could also run it locally on a used dual xeon (or amd-equivalent) server with 512GB RAM, albeit slower, if you have a useful workflow for it that's like "take this day's efforts and run it through various analysis agents", combined with giving it one-shot tasks/modules to build overnight. You would want a place like a garage or basement to put the server because it'll be loud.
- dataplumb3r 2mo agoYou'd also likely spend far more in electricity than the API cost of processing the prompt(s)
- walrus01 2mo agoyes, though for some uses, not sending data anywhere to third parties has its own value which is harder to measure.
- peri-cl 2mo ago> "dual xeon" Does inference make full use of the memory bandwidth in a NUMA system?
- walrus01 2mo agoMemory throughput performance is better than many equivalently priced (talking about the cheaper end of the used/refurb market here) single socket systems because each CPU is talking to its own banks of RAM. Using an example of llama-server on a 256GB (each CPU is attached to 128GB) system running qwen 3.8-flash-next Q8, which takes up about 188GB with the model and context, it's not that the inference engine 'knows' about NUMA, but if you use --load-mode mlock to load the whole thing into memory at once at the time llama-server launches, the kernel will distribute it relatively evenly across the memory connected to both CPUs.