6 ms·

> very few open-source LLMs explicitly claim they intentionally support structured data, but they’re smart enough and they have logically seen enough examples o

> very few open-source LLMs explicitly claim they intentionally support structured data, but they’re smart enough and they have logically seen enough examples of JSON Schema that with enough system prompt tweaking they should behave. Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al that interact with the internals of the model (https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliable_ways_to_get_structured_output_from_llms/ https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliabl...). The structured outputs can be arbitrary grammars, for example, not just JSON (https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-...).

airstrike 3y ago
That last link is interesting. See https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-... specifically # ... sequence = generator("Write a formula that returns 5 using only additions and subtractions.") # It looks like Mistral is not very good at arithmetics :) print(sequence) # 1+3-2-4+5-7+8-6+9-6+4-2+3+5-1+1 sure, that's "correct" per the definition of the grammar, but it's also one of the worst possible way to get to the number 5
behnamoh 3y ago
> Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al... Yes, but you should also instruct the model to follow that specific pattern in its answer, or else the accuracy of the response degrades even though it's following your grammar/pattern/whatever. For example, if you use Llama-2-7b for classification (three categories, "Positive", "Negative", "Neutral"), you might write a grammar like this: ``` root ::= "{" ws "sentiment:" ws sentiment "}" sentiment ::= ("Positive" | "Neutral" | "Negative" ) ws ::= [ \t\n]* ``` But if the model doesn't know it has to generate this schema, the accuracy of classifications drops because it's trying to say other things (e.g., "As an AI language model...") which then get suppressed and "converted" to the grammar.
- Tostino 3y ago
  This can sometimes be fixed with a few-shot example for in-context-learning. But you are right that the model can go off the rails if it is being forced too far from where its 'happy place' is, especially for smaller models.
- coder543 3y ago
  Similarly, I think it is important to provide an “|” grammar that defines an error response, and explain to the model that it should use that format to explain why it cannot complete the requested operation if it runs into something invalid. Otherwise, it is forced to always provide a gibberish success response that you likely won’t catch. I’ve tested this with Mixtral, and it seems capable of deciding between the normal response and error response based on the validity of the data passed in with the request. I’m sure it can still generate gibberish in the required success response format, but I never actually saw it do that in my limited testing, and it is much less likely when the model has an escape hatch.
  - behnamoh 3y ago
    Can you elaborate? So you instruct the model to either follow the grammar OR say why it can't do that? But the model has no idea this grammar exists (you can tell it the schema but the model doesn't know its tokens are going through a logprobs modification).
    - coder543 3y ago
      No, the grammar can do OR statements. You provide two grammars, essentially. You always want to tell the model about the expected response formats, so that it can provide the best response it can, even though you’re forcing it to fit the grammar anyways. In JSON Schema, you can do a “oneOf” between two types. You can easily convert a JSON Schema into the grammar that llama.cpp expects. One of the types would be the success response, the other type would be an error response, such as a JSON object containing only the field “ErrorResponse”, which is required to be a string, which you explain to the model that this is used to provide an explanation for why it cannot complete the request. It will literally fill in an explanation when it runs into troublesome data, at least in my experience. Then the model can “choose” which type to respond with, and the grammar will allow either. If everything makes sense, the model should provide the successful response you’re requesting, otherwise it can let you know something weird is going on by responding with an error.
bugglebeetle 3y ago
Yeah, JSON mode in Ollama, which isn’t even the full llama.cpp grammar functionality, performs better than OpenAI for me at this point. I don’t understand how they can be raking in billions of dollars and can’t even get this basic stuff right.
- WhitneyLand 3y ago
  llama.cop Performs better than what? 3.5 Turbo, one of the 4.0 models, API or App?
  - bugglebeetle 3y ago
    JSON mode and function-calling with a JSON schema in the OpenAI API.
    - WhitneyLand 3y ago
      Right, but which model? It makes a huge difference.
      - bugglebeetle 3y ago
        I’ve been using OpenChat 3.5 1210 most recently. Before that, Mistral-OpenOrca. Both return JSON more consistently than gpt-3.5-turbo.
        airstrike 3y ago
        gpt-3.5-turbo is not the benchmark
        bugglebeetle 3y ago
        I don’t know what point you’re trying to make. They also return JSON more consistently than gpt-4, but I don’t use that because it’s overkill and expensive for my text extraction tasks.
mgreg 3y ago
There are now several open source models that are fine tuned for function calling including: * Functionary [https://github.com/MeetKai/functionary https://github.com/MeetKai/functionary] * NexusRaven [https://github.com/nexusflowai/NexusRaven-V2 https://github.com/nexusflowai/NexusRaven-V2] * Gorilla [https://github.com/ShishirPatil/gorilla https://github.com/ShishirPatil/gorilla] Could be interesting to try some of these exercises with these models.
- freediver 3y ago
  ... and I spent last few hours trying them now :) Low latency, high quality function calling API product may be a billion dollar business in two years.
  - column 3y ago
    What are your findings?
    - freediver 3y ago
      They are not at the level of gpt-4 tool calling. But at least they are open source and they will get better.

airstrike 3y ago

That last link is interesting. See https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-... specifically # ... sequence = generator("Write a formula that returns 5 using only additions and subtractions.") # It looks like Mistral is not very good at arithmetics :) print(sequence) # 1+3-2-4+5-7+8-6+9-6+4-2+3+5-1+1 sure, that's "correct" per the definition of the grammar, but it's also one of the worst possible way to get to the number 5

behnamoh 3y ago

> Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al... Yes, but you should also instruct the model to follow that specific pattern in its answer, or else the accuracy of the response degrades even though it's following your grammar/pattern/whatever. For example, if you use Llama-2-7b for classification (three categories, "Positive", "Negative", "Neutral"), you might write a grammar like this: ``` root ::= "{" ws "sentiment:" ws sentiment "}" sentiment ::= ("Positive" | "Neutral" | "Negative" ) ws ::= [ \t\n]* ``` But if the model doesn't know it has to generate this schema, the accuracy of classifications drops because it's trying to say other things (e.g., "As an AI language model...") which then get suppressed and "converted" to the grammar.

Tostino 3y ago
This can sometimes be fixed with a few-shot example for in-context-learning. But you are right that the model can go off the rails if it is being forced too far from where its 'happy place' is, especially for smaller models.
coder543 3y ago
Similarly, I think it is important to provide an “|” grammar that defines an error response, and explain to the model that it should use that format to explain why it cannot complete the requested operation if it runs into something invalid. Otherwise, it is forced to always provide a gibberish success response that you likely won’t catch. I’ve tested this with Mixtral, and it seems capable of deciding between the normal response and error response based on the validity of the data passed in with the request. I’m sure it can still generate gibberish in the required success response format, but I never actually saw it do that in my limited testing, and it is much less likely when the model has an escape hatch.
- behnamoh 3y ago
  Can you elaborate? So you instruct the model to either follow the grammar OR say why it can't do that? But the model has no idea this grammar exists (you can tell it the schema but the model doesn't know its tokens are going through a logprobs modification).
  - coder543 3y ago
    No, the grammar can do OR statements. You provide two grammars, essentially. You always want to tell the model about the expected response formats, so that it can provide the best response it can, even though you’re forcing it to fit the grammar anyways. In JSON Schema, you can do a “oneOf” between two types. You can easily convert a JSON Schema into the grammar that llama.cpp expects. One of the types would be the success response, the other type would be an error response, such as a JSON object containing only the field “ErrorResponse”, which is required to be a string, which you explain to the model that this is used to provide an explanation for why it cannot complete the request. It will literally fill in an explanation when it runs into troublesome data, at least in my experience. Then the model can “choose” which type to respond with, and the grammar will allow either. If everything makes sense, the model should provide the successful response you’re requesting, otherwise it can let you know something weird is going on by responding with an error.

bugglebeetle 3y ago

Yeah, JSON mode in Ollama, which isn’t even the full llama.cpp grammar functionality, performs better than OpenAI for me at this point. I don’t understand how they can be raking in billions of dollars and can’t even get this basic stuff right.

WhitneyLand 3y ago
llama.cop Performs better than what? 3.5 Turbo, one of the 4.0 models, API or App?
- bugglebeetle 3y ago
  JSON mode and function-calling with a JSON schema in the OpenAI API.
  - WhitneyLand 3y ago
    Right, but which model? It makes a huge difference.
    - bugglebeetle 3y ago
      I’ve been using OpenChat 3.5 1210 most recently. Before that, Mistral-OpenOrca. Both return JSON more consistently than gpt-3.5-turbo.
      - airstrike 3y ago
        gpt-3.5-turbo is not the benchmark
        bugglebeetle 3y ago
        I don’t know what point you’re trying to make. They also return JSON more consistently than gpt-4, but I don’t use that because it’s overkill and expensive for my text extraction tasks.

mgreg 3y ago

There are now several open source models that are fine tuned for function calling including: * Functionary [https://github.com/MeetKai/functionary https://github.com/MeetKai/functionary] * NexusRaven [https://github.com/nexusflowai/NexusRaven-V2 https://github.com/nexusflowai/NexusRaven-V2] * Gorilla [https://github.com/ShishirPatil/gorilla https://github.com/ShishirPatil/gorilla] Could be interesting to try some of these exercises with these models.

freediver 3y ago
... and I spent last few hours trying them now :) Low latency, high quality function calling API product may be a billion dollar business in two years.
- column 3y ago
  What are your findings?
  - freediver 3y ago
    They are not at the level of gpt-4 tool calling. But at least they are open source and they will get better.