2 ms·
That GPU does pretty well for running an LLM though. Llama 7B at 2.8 tok/s via Mlc-chat. I'm setting up an LLM discord bot with it.
by francisduvivier 3y ago
That GPU does pretty well for running an LLM though.
Llama 7B at 2.8 tok/s via Mlc-chat.
I'm setting up an LLM discord bot with it.