3 ms·

Update: I tried it out. It took about 8 seconds per token, and didn't seem to be using much of my GPU (MPU), but was using a lot of RAM. Not a model that I coul

Update: I tried it out. It took about 8 seconds per token, and didn't seem to be using much of my GPU (MPU), but was using a lot of RAM. Not a model that I could use practically on my machine.

steinvakt2 1y ago
Did you run it the best way possible? im no expert, but I understand it can affect inference time greatly (which format/engine is used)
- pamelafox 1y ago
  I ran it via Ollama, which I assume uses the best way. Screenshot in my post here: https://bsky.app/profile/pamelafox.bsky.social/post/3lvobol3jfb2r https://bsky.app/profile/pamelafox.bsky.social/post/3lvobol3... I'm still wondering why my MPU usage was so low.. maybe Ollama isn't optimized for running it yet?
  - wahnfrieden 1y ago
    Might need to wait on MLX
turnsout 1y ago
To clarify, this was the 20B model?
- pamelafox 1y ago
  Yep, 20B model, via Ollama: ollama run gpt-oss:20b Screenshot here with Ollama running and asitop in other terminal: https://bsky.app/profile/pamelafox.bsky.social/post/3lvobol3jfb2r https://bsky.app/profile/pamelafox.bsky.social/post/3lvobol3...