5 ms·
Qwen3.6-35B-A3B-UD-Q4_K_M runs at about 11 tokens/second on my poor old 1060. Absolutely nuts how far we've come
by unleaded 4mo ago
Qwen3.6-35B-A3B-UD-Q4_K_M runs at about 11 tokens/second on my poor old 1060. Absolutely nuts how far we've come
- broodbucket 4mo agoMind sharing your llama.cpp settings for that?
- unleaded 4mo ago.\llama-server.exe -m ..\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -ngl 999 --n-cpu-moe 41 -c 262144 --port 8081 --flash-attn on --cache-type-k turbo4 --cache-type-v turbo3 --no-mmap --mlock --host 0.0.0.0 -t 8 -tb 8 -np 1 Using this llama.cpp fork https://github.com/TheTom/llama-cpp-turboquant https://github.com/TheTom/llama-cpp-turboquant and mostly copying from this video https://www.youtube.com/watch?v=8F_5pdcD3HY https://www.youtube.com/watch?v=8F_5pdcD3HY Haven't had much time to test it other than asking a few questions & changing some HTML in cline so it might be thick as a brick for all I know, but still worth trying
- unleaded 4mo agoI just tested it with some risc-v code and it wrote down a "mov" instruction several times.. yeah something needs tuning maybe
- piyh 4mo agoI tried running any model on my 1070 and it instantly crashes my old tower, probably time to get off windows and run linux on it.
- SV_BubbleTime 4mo agoUnderstated how much of a boon for Linux that AI development has been. There isn’t any benefit to running a windows machine.
- selectodude 4mo agoAu contraire, I run models on WSL and my desktop reliably wakes up from sleep. Best of both worlds.
- scotty79 4mo agoWatch out for VRAM/RAM. When programs on WSL tired to take more than there was available it was crashing my WSL hard and even corrupting files on the virtual drive attached to WSL in a very strange manner that made recent files just disappear. I had at leat two projects that had to be rebuilt from past conversation with AI because all their files were gone after few crashes. I asked codex to write WSL config to prevent crashes. It put some limits on WSL and the situation stabilized, but I lost all trust for WSL anyways.
- fallinditch 4mo agoWhat are the config settings you needed? TIA
- scotty79 4mo agoThat's my wslconfig file located in my %USERPROFILE% dir: [wsl2] memory=40GB processors=8 swap=8GB swapFile=C:\\wsl-swap.vhdx localhostForwarding=true guiApplications=false vmIdleTimeout=30000 maxCrashDumpCount=3 [experimental] autoMemoryReclaim=gradual I have 64GB of RAM and 24GB of VRAM.
- MrDrMcCoy 4mo agoGenuine question: what are the good parts of Windows that keep you there? I've been such a happy gamer/coder/media editor on Linux for so long that I really don't understand how Microsoft isn't dead yet.
- selectodude 4mo agoMy desktop is a game console and the games I play work on windows. And my computer reliably wakes from sleep.
- greenavocado 4mo agoSounds like a hardware issue, though NVIDIA driver issues can't be ruled out, they're much rarer these days