4 ms·
I'm a bit surprised by the numbers. It's "only" a 2× speedup on a relatively top-end card (4090)? And you can only use one CPU core. With 16+ core CPUs becoming
by davidy123 3y ago
I'm a bit surprised by the numbers. It's "only" a 2× speedup on a relatively top-end card (4090)? And you can only use one CPU core. With 16+ core CPUs becoming normal and 128GB+ RAM being cheap, that seems like leaving a lot on the table.
[edit] realized it's relative to the merged partial CUDA acceleration, so the speedup is more impressive, but still surprised by the core usage.
- LoganDark 3y ago> And you can only use one CPU core. because the core's job is solely to direct the GPU, which is doing all of the work.
- eurekin 3y agoThat cheap ram is about 10x slower than the VRAM. Didn't see any actual figures for latency, but there must be a reason, why newer gpus have memory chips on both sides of the PCB, as close to the GPU as possible
- davidy123 3y agoTo the replies; I think one feature of llama.cpp is it can handle models with more RAM than VRAM provides, this is where I would think more cores would be useful.