4 ms·
Not sure when they implemented this, but ollama now has a JSON mode [0]. Not function calling, but one of the simpler ways to get JSON in a local LLM. I'm using
by mritchie712 3y ago
Not sure when they implemented this, but ollama now has a JSON mode [0]. Not function calling, but one of the simpler ways to get JSON in a local LLM. I'm using it with `knoopx/hermes-2-pro-mistral:7b-q8_0` and it's worked well for me so far.
response = ollama.chat(model=OLLAMA_MODEL,
messages=[
{
'role': 'system',
'content': system_message,
},
{
'role': 'user',
'content': user_prompt,
},
],
format='json',
options = {
#'temperature': 1.5, # very creative
'temperature': 0.0
}
)
0 - https://github.com/ollama/ollama/blob/main/docs/api.md#json-mode https://github.com/ollama/ollama/blob/main/docs/api.md#json-...
- Joschkabraun 3y agoInteresting. Do you have any benchmarks?
- mritchie712 3y agoNo benchmarks, just my anecdotal experience trying to get local LLM's to respond with JSON. The method above works for my use case nearly 100% of the time. Other things I've tried (e.g. `outlines`[0]) are really slow or don't work at all. Would love to hear what others have tried! 0 - https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines
- Joschkabraun 3y agoAh yes. Have you tried out instructor [0] or Guidance [1]? [0]: https://github.com/jxnl/instructor/ https://github.com/jxnl/instructor/ [1]: https://github.com/guidance-ai/guidance/tree/main https://github.com/guidance-ai/guidance/tree/main