2 ms·
Anyone succeed in running it with vLLM?
by Aissen 11mo ago
Anyone succeed in running it with vLLM?
- dloss 11mo agoYes, the 3B variant, with vLLM 0.11.2. Parameters are given on the HF page. Had to override the temperature to 0.15 though (as suggested on HF) to avoid random looking syllables.
- Patrick_Devine 11mo agoThe instruct models are available on Ollama (e.g. `ollama run ministral-3:8b`), however the reasoning models still are a wip. I was trying to get them to work last night and it works for single turn, but is still very flakey w/ multi-turn.
- Aissen 10mo agoIt now seems to work with the latest vLLM git.