3 ms·
USB Stick with Local LLM [video]
- andrewfromx 2y agoUSB device using a Raspberry Pi Zero W 512MB RAM and single-core ARMv6. It looks like a regular USB drive when plugged in. Used llamacpp but had to remove architecture-specific optimizations. Took over 12 hours on the Pi Zero's old CPU! Successfully runs three different LLM models: Tiny Stories (15M parameters), Lamini (77M parameters), Smollm (135M parameters) The smallest model generates tokens at ~200ms per token, while larger models take about 2.5 seconds per token. When you create a new file on the USB drive, it detects the filename and automatically generates content!