5 ms·
Not in production, but I've used a 3B model to test a local LLM application I'm working on. I needed a full end-to-end request/response and it's a lot faster as
by accrual 2y ago
Not in production, but I've used a 3B model to test a local LLM application I'm working on. I needed a full end-to-end request/response and it's a lot faster asking a 3B model than an 8B model. I could setup a test harness and replay the responses... but this was a lot simpler.
- jdthedisciple 2y agoIf for testing then why not just mock the whole thing for ultimate performance ... ?
- nkozyra 2y agoProbably faster to use off the shelf model with llama.cpp than to mock it