3 ms·

Are there fine tuned models that perform better for structured / parsable outputs?

• _flux 2y ago
  This isn't the answer to that question, but llama.cpp has a feature to constrain output to the provided grammar, such as https://github.com/ggerganov/llama.cpp/blob/master/grammars/json.gbnf https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Others should really implement that as well. You still need to guide the model to produce e.g. JSON to get good results, but they will 100% guaranteed be valid per the grammar.
  • alhaad 2y ago
    Agreed that others should implement it as well but coercing llama to output results with matching grammar needs work.
    • _flux 2y ago
      What kind of work? I've only given it a short try before moving to Ollama that doesn't have it, but it seemed to have worked there. (With ollama I need to use a retry system.) edit: I researched a bit and apparently it can reduce performance, plus the streaming mode fails to report incorrect grammars. Overall these don't seem like deal-breakers.
• cpursley 2y ago
  Fireworks.ai Firefunction is pretty good. Not GPT-level but it’s an open model.