4 ms·
I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU an
by tarpitt 3mo ago
I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only?
Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
- vforno 3mo agoThat's possibly a good idea! We can work on it!
- tarpitt 3mo agoI also just edited my comment with more ideas in the beginning, sorry
- valicord 3mo ago> OS page cache can generate writes Is this a hallucination? What am I missing? Why would heavy reads generate writes?
- onlyrealcuzzo 3mo agoSpilling
- gcr 3mo agoWouldn’t turning off swap fix this issue?
- throwa356262 3mo agoBetter to just change swapiness? https://askubuntu.com/questions/103915/how-do-i-configure-swappiness https://askubuntu.com/questions/103915/how-do-i-configure-sw...
- deleted 3mo ago[deleted]
- TacticalCoder 3mo ago> Is this a hallucination? What am I missing? Why would heavy reads generate writes? I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached? I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?
- fallingbananna 3mo agoGood catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.
- deleted 3mo ago[deleted]
- vforno 3mo agoI have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!
- tarpitt 3mo agoI have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused. I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU. If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item. I'm going to benchmark this right now with what I have and I'll get back to you on github.
- vforno 3mo agoReally thanks!!
- halcyonblue 3mo agoAt least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.
- throwdbaaway 3mo agoIf you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.