4 ms·
Are you running bare metal or docker? I had to go down to 2 bit on Gemma4 E2B model to run on 8gb on a Jetson Quality and idempotency is great but it’s still n
by n4pw01f 2mo ago
Are you running bare metal or docker? I had to go down to 2 bit on Gemma4 E2B model to run on 8gb on a Jetson
Quality and idempotency is great but it’s still not exactly fast… fast enough and works offline
Is this something that you can get running on Debian?
- gitpusher42 2mo agoIt is Apple platform only implementation because of Metal (and Swift). Other platforms would require CUDA or Vulkan and a complete rework