silentbob7

4 ms·

1.
Slotstream: Swift, SSD Expert Streaming, and Breaking the LLM Memory Wall(singularitymoments.com)
3 points by silentbob7 27d ago|0 comments
2.
by silentbob7 1mo ago
Here's my model.ini for 4090 (using the chat template from froggeric): version = 1 [*] ; Global defaults shared across all presets threads = 16 [Qwen3.8-27B] ; Model files model = ..\Qwen3.8-27B-FP8\Qwen3.8-27B-UD-Q4_K_XL.gguf mmpr
3.
by silentbob7 2mo ago
I'm especially interested in details on process-in-memory, combined with that large sizes this could be very powerfull in some usecases.
4.
JEDEC Previews LPDDR6 Roadmap, 512 GB Densities and SOCAMM2 Standard(techpowerup.com)
7 points by silentbob7 2mo ago|3 comments
5.
by silentbob7 2mo ago
It depends on your usecase and the size of your ram is not the only driver. I think the primary performance drivers (without sacrificing precision) right now are QAT, MTP/DFlash, MoE and Hybrid approaches to avoid full attention in par
6.
by silentbob7 2mo ago
Also worth checking out https://github.com/OHF-Voice/linux-voice-assistant
7.
by silentbob7 2mo ago
Tested with german and this kinda works, but confidence suffers.
8.
by silentbob7 2mo ago
I also wonder how small a LLM trained on catching only subject (e.g. living room) and action (light on) from text input could be compared to needle - the json wrapping could be done afterwards using templates.
9.
by silentbob7 2mo ago
The wyoming protocoll seems to be the path for home assistant audio, so you need STT (wyoming-faster-whisper), TTS (wyoming-piper for wide language support) API endpoints and some Ollama or OpenAI API endpoint available for your home assist
10.
by silentbob7 2mo ago
I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with