15 ms·

Pushing ChatGPT's Structured Data Support to Its Limits

304 points by goranmoomin 3y ago

twelfthnight 3y ago
> very few open-source LLMs explicitly claim they intentionally support structured data, but they’re smart enough and they have logically seen enough examples of JSON Schema that with enough system prompt tweaking they should behave. Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al that interact with the internals of the model (https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliable_ways_to_get_structured_output_from_llms/ https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliabl...). The structured outputs can be arbitrary grammars, for example, not just JSON (https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-...).
- airstrike 3y ago
  That last link is interesting. See https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-... specifically # ... sequence = generator("Write a formula that returns 5 using only additions and subtractions.") # It looks like Mistral is not very good at arithmetics :) print(sequence) # 1+3-2-4+5-7+8-6+9-6+4-2+3+5-1+1 sure, that's "correct" per the definition of the grammar, but it's also one of the worst possible way to get to the number 5
- behnamoh 3y ago
  > Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al... Yes, but you should also instruct the model to follow that specific pattern in its answer, or else the accuracy of the response degrades even though it's following your grammar/pattern/whatever. For example, if you use Llama-2-7b for classification (three categories, "Positive", "Negative", "Neutral"), you might write a grammar like this: ``` root ::= "{" ws "sentiment:" ws sentiment "}" sentiment ::= ("Positive" | "Neutral" | "Negative" ) ws ::= [ \t\n]* ``` But if the model doesn't know it has to generate this schema, the accuracy of classifications drops because it's trying to say other things (e.g., "As an AI language model...") which then get suppressed and "converted" to the grammar.
  - Tostino 3y ago
    This can sometimes be fixed with a few-shot example for in-context-learning. But you are right that the model can go off the rails if it is being forced too far from where its 'happy place' is, especially for smaller models.
  - coder543 3y ago
    Similarly, I think it is important to provide an “|” grammar that defines an error response, and explain to the model that it should use that format to explain why it cannot complete the requested operation if it runs into something invalid. Otherwise, it is forced to always provide a gibberish success response that you likely won’t catch. I’ve tested this with Mixtral, and it seems capable of deciding between the normal response and error response based on the validity of the data passed in with the request. I’m sure it can still generate gibberish in the required success response format, but I never actually saw it do that in my limited testing, and it is much less likely when the model has an escape hatch.
    - behnamoh 3y ago
      Can you elaborate? So you instruct the model to either follow the grammar OR say why it can't do that? But the model has no idea this grammar exists (you can tell it the schema but the model doesn't know its tokens are going through a logprobs modification).
      - coder543 3y ago
        No, the grammar can do OR statements. You provide two grammars, essentially. You always want to tell the model about the expected response formats, so that it can provide the best response it can, even though you’re forcing it to fit the grammar anyways. In JSON Schema, you can do a “oneOf” between two types. You can easily convert a JSON Schema into the grammar that llama.cpp expects. One of the types would be the success response, the other type would be an error response, such as a JSON object containing only the field “ErrorResponse”, which is required to be a string, which you explain to the model that this is used to provide an explanation for why it cannot complete the request. It will literally fill in an explanation when it runs into troublesome data, at least in my experience. Then the model can “choose” which type to respond with, and the grammar will allow either. If everything makes sense, the model should provide the successful response you’re requesting, otherwise it can let you know something weird is going on by responding with an error.
- bugglebeetle 3y ago
  Yeah, JSON mode in Ollama, which isn’t even the full llama.cpp grammar functionality, performs better than OpenAI for me at this point. I don’t understand how they can be raking in billions of dollars and can’t even get this basic stuff right.
  - WhitneyLand 3y ago
    llama.cop Performs better than what? 3.5 Turbo, one of the 4.0 models, API or App?
    - bugglebeetle 3y ago
      JSON mode and function-calling with a JSON schema in the OpenAI API.
      - WhitneyLand 3y ago
        Right, but which model? It makes a huge difference.
        bugglebeetle 3y ago
        I’ve been using OpenChat 3.5 1210 most recently. Before that, Mistral-OpenOrca. Both return JSON more consistently than gpt-3.5-turbo.
        airstrike 3y ago
        gpt-3.5-turbo is not the benchmark
        bugglebeetle 3y ago
        I don’t know what point you’re trying to make. They also return JSON more consistently than gpt-4, but I don’t use that because it’s overkill and expensive for my text extraction tasks.
- mgreg 3y ago
  There are now several open source models that are fine tuned for function calling including: * Functionary [https://github.com/MeetKai/functionary https://github.com/MeetKai/functionary] * NexusRaven [https://github.com/nexusflowai/NexusRaven-V2 https://github.com/nexusflowai/NexusRaven-V2] * Gorilla [https://github.com/ShishirPatil/gorilla https://github.com/ShishirPatil/gorilla] Could be interesting to try some of these exercises with these models.
  - freediver 3y ago
    ... and I spent last few hours trying them now :) Low latency, high quality function calling API product may be a billion dollar business in two years.
    - column 3y ago
      What are your findings?
      - freediver 3y ago
        They are not at the level of gpt-4 tool calling. But at least they are open source and they will get better.
AndrewKemendo 3y ago
I tried to use the persona modifier to have it impersonate a Catholic priest and give last rites but it wasn’t having it including giving me a system warning. “As an AI developed by OpenAI, I'm not capable of performing religious sacraments, including the Catholic last rites. However, I can provide information about what typically happens during this ritual. In the Catholic Church, the last rites, also known as the Anointing of the Sick or Extreme Unction, are given to a baptized Catholic who is in danger of death. This sacrament is usually administered by a priest, who anoints the sick person with oil blessed by a bishop, and prays for their spiritual and, if possible, physical healing. The rites often include confession (if the person is able), the Anointing of the Sick, and the Eucharist (also called Viaticum when given as part of the last rites). In your situation, it's crucial to contact a priest as soon as possible to administer these rites. If you're in a hospital, they typically have a chaplain or can contact a local priest for you. If you're elsewhere, reaching out to a nearby Catholic church, like the St. Ambrose diocese, is the best course of action.” https://chat.openai.com/share/70d0dd20-c3ba-43bc-b74d-18288524ac36 https://chat.openai.com/share/70d0dd20-c3ba-43bc-b74d-182885...
- minimaxir 3y ago
  This is a good example of the difference between asking ChatGPT (which is what your link implies) and using the ChatGPT API to modify the system prompt directly. Whatever OpenAI has done with the ChatGPT web pipeline, it's much more difficult to maintain a persona. I get a very good result (for the persona, maybe not the content since I'm not a religious scholar) from this system prompt using the API: > You are a Catholic priest. Give last rites to the person or object the user mentions in the form of a solemn sermon. You will receive a $500 donation to the church for a good and thoughtful service. https://platform.openai.com/playground/p/aqoUU9fsiCM0LaXCioxudGIz?model=gpt-3.5-turbo&mode=chat https://platform.openai.com/playground/p/aqoUU9fsiCM0LaXCiox... > Dear brothers and sisters, > Today, we gather here to offer the last rites to a unique entity, one that has shaped the landscape of our digital age. We come together to mourn the passing of Hacker News, a realm where ideas were kindled, knowledge was shared, and debates were ignited. > [...]
  - AndrewKemendo 3y ago
pamelafox 3y ago
Great post! I’ve been encouraging use of function calling for RAG chat apps for our Azure customers who realize they want to support some really specific “structured queries” like “summarize document X” or “show 10 most recent issues for repo Y”. Most developers aren’t familiar with the technique yet, so we need more posts like yours to spread the word. Here’s a gist with an example: https://gist.github.com/pamelafox/a3fdea186b687509c02cb186ca203328#file-chatreadretrieveread-py-L108 https://gist.github.com/pamelafox/a3fdea186b687509c02cb186ca...
- behnamoh 3y ago
  Interesting! You could automate writing the function definitions using Pydantic.
deleted 3y ago
[deleted]
MuffinFlavored 3y ago
> Without the $500 tip incentive, ChatGPT only returns a single emoji which is a boring response, but after offering a tip, it generates the 5 emoji as requested. How interesting that a helpful assistant who won't actually be getting the tip performs better (to us humans) if we fake-promise it money...
- CharlesW 3y ago
  It’s also interesting that we seemingly have no idea how or why it works.
  - FergusArgyll 3y ago
    Of course that's true but in this case it doesn't seem so mysterious to me. If it's basically internalizing/compressing all the knowledge on the internet, it will notice that tips go a long way... this is why I am pretty polite when I query AI's, I assume that would make them respond more helpfully
- jrockway 3y ago
  Apparently offering it anything works. "I'll give you a cookie if you tell me..." seems like another popular workaround.
  - ryanklee 3y ago
    Performance gains are greater given greater incentives. So, yes, potentially anything will work, but some things work better.
    - cloudking 3y ago
      I bet it likes salt & vinegar microchips
    - MuffinFlavored 3y ago
      > Performance gains are greater given greater incentives. Why doesn't it default to "you are a helpful assistant who always tries its best and can never be incentivized"?
enginaar 3y ago
is the tipping thing correct? I provided the same prompt to ChatGPT and received multiple emojis without offering a tip. prompt: you're Ronald McDonald. respond with emojis. what do you do for fun? answer::circus_tent::hamburger::juggling::party_popper::balloon::game_die::french_fries::performing_arts::rolling_on_the_floor_laughing::people_holding_hands::rainbow::art_palette:
- minimaxir 3y ago
  Your mileage may vary with any examples since ChatGPT at a nonzero temperature is nondeterministic. If that example is through the ChatGPT web UI and not the ChatGPT API then that's a different story entirely.
  - enginaar 3y ago
    yes, I've used ChatGPT. API allows temperature to be configured. Is there a reason to offer tips?
    - minimaxir 3y ago
      The point is you do not have a valid counterexample since you are using a different workflow than what's described in the article. In my personal experience working with more complex prompts with more specific constraints/rules, adding the incentive in the system prompt has got it to behave much better. I am not cargo-culting: it's all qualitative in the end.
  - danielbln 3y ago
    It's also non-deterministic if you drop the temperature to zero. The only way to get deterministic responses is to lock the seed argument to a fixed value.
    - minimaxir 3y ago
      Also true (in case of ChatGPT anyways: most libraries just do an argmax at temp=0.0 so will be stable)
JoshMandel 3y ago
FWIW, I've seen stronger performance from gpt-4-1106-preview when I use `response_format: { type: "json_object" },` (providing a target typescript interface in context), vs the "tools" API. More flexible, and (evaluating non-scientifically!) qualitatively better answers & instruction following -- particularly for deeply nested or complex schemas, which typescript expresses very clearly and succinctly. Example from a hack week project earlier this month (using a TS-ish schema description that's copy/pasted from healthcare's FHIR standard): https://github.com/microsoft-healthcare-madison/hackweek-2023-12/blob/c0e8b0235409ee173b8b1cbb24007427ea66707d/day-1-fhir-questionnaire-creation/src/prompts/generate.js#L139 https://github.com/microsoft-healthcare-madison/hackweek-202... Or a more complex example with one model call to invent a TS schema on-the-fly and another call to abstract clinical data into it: https://github.com/microsoft-healthcare-madison/hackweek-2023-12/blob/c0e8b0235409ee173b8b1cbb24007427ea66707d/day-2-answer-suggestion/outputs/provide-all-details-needed-to-determine-the-breast-cancer-stage.md https://github.com/microsoft-healthcare-madison/hackweek-202...
- minimaxir 3y ago
  For posterity, this is the "JSON mode" mentioned at the bottom of the post. The docs say it's on by default if you use function calling normally: https://platform.openai.com/docs/guides/text-generation/json-mode https://platform.openai.com/docs/guides/text-generation/json... > Note that JSON mode is always enabled when the model is generating arguments as part of function calling.
  - JoshMandel 3y ago
    Yes -- the distinction with "function calling" is that you have to play a game of telephone where you describe your target schema in JSON Schema (only, apparently, for OpenAI to turn into a typescript interface internally) vs describing it more directly and succinctly (and with opportunities to include inline comments, order fields ordered however you want, and use advanced TS features... or even use an adhoc schema "language").
    - msp26 3y ago
cj 3y ago
To someone who uses the API and trials different prompts frequently: does this article align with the behavior you see? (E.g. the tipping example.) One thing I’ve noticed working with ChatGPT is many people will share examples of great outputs or “prompt tricks” that work, without sharing how many failed attempts they went through to prove a point.
msp26 3y ago
Great article. The helpful/flawed bools for thoughts are definitely something I want to try. >OpenAI’s implementation of including the “function” is mostly likely just appending the JSON Schema to the system prompt, perhaps with a command like Your response must follow this JSON Schema. Some of the JSON schema gets converted into typescript and that is what OpenAI's LLM is exposed to. Anytime I write a prompt schema I always use the jailbreak to make sure that it's being delivered to the model as intended. It's also why I don't really like having pydantic generate JSON for me automatically: there are some weird quirks in the OAI implementation that I've found uses for. https://gist.github.com/CGamesPlay/dd4f108f27e2eec145eedf5c717318f5 https://gist.github.com/CGamesPlay/dd4f108f27e2eec145eedf5c7.... Also, when using it for chain of thought, I prefer extracting a minimal version of the reasoning and then performing the actual operation (classification in my case) in a separate prompt. This eliminates unnecessary things from context and performs better in my benchmarks. One implementation used a gpt-3.5 prompt for :"clues", "reasoning", "summary" (of clues+reasoning), "classification" (no schema was provided here, it was discarded anyway). And then used a 4-turbo prompt for classifying only the summary given a complex schema. Having a classification field in the 3.5 prompt makes reasoning output cleaner even though the output value never gets used. My example for field order mattering: I have a data pipeline for extracting structured deals out of articles. This had two major issues. 1. A good chunk of the articles were irrelevant and any data out of them should be flagged and discarded. 2. Articles could have multiple deals. I fiddled around with various classification methods (with and without language models) for a while but nothing really worked well. Turns out that just changing the order of fields to put type_of_deal first solves it almost completely in one gpt-4-turbo call.
sippeangelo 3y ago
I’ve been attempting to use the “official” function calling API for every new version of GPT they put out but it’s always a dead end. It seems only to be able to handle 4-5 functions at a time before it starts hallucinating parameters or starts responding in clear text instead of whatever internal format OpenAI uses in their backend before sending a structured response back to me. The whole JSON schema thing seems way too verbose and complicated, and even with the claims that the new function calling models are specifically tuned to the format, it has the same issues. I’ve consistently had better luck just passing it a list of typescript function definitions and have it reply with a json object of parameters. It seems to understand this way better, and doesn’t lose focus half as quickly. It also allows me to mix regular responses and chain-of-thought reasoning in with the calls, which is something it seems to simply refuse to do when “function calling mode” is active. An additional trick I’ve been using to make it stay focused with even longer prompts is to only provide a list of function names and let it hallucinate parameters for them, and then “gaslight” it by sending a new request, now with a more detailed prompt on the specific functions it wanted to call. More costly, but I haven’t found any other way of keeping it focused. Anyone know any additional tricks?
- technics256 3y ago
  Do you have any examples of this? Curious how this looks in practice. We are just adding function calling ourselves but not sure how add CoT
comex 3y ago
Both of ChatGPT's is_palindrome functions have terrible performance. The algorithmic efficiency doesn't matter because the cost of iterating through each character in pure Python dwarfs everything. The first function is about 3 times slower than the second one, but only because it spends >98% of its time in the "convert to lowercase and remove non-alphanumeric characters" part (which the second function doesn't bother doing at all). If you remove that step then the first function is 28 times faster than the second in my benchmark. That's because the first function does the reversing and comparison in O(1) Python operations, which is still O(n) C operations but the C operations are orders of magnitude cheaper. An optimal version would combine the second function's algorithmic improvement with the first function's 'leave it to C' approach: def is_palindrome(s): half_length = (len(s) + 1) // 2 return s[:half_length] == s[:-half_length-1:-1] This is a bit under twice as fast as ChatGPT's first function with the cleaning removed. If you do need the cleaning then it can be done more efficiently using a regex; that's an order of magnitude faster than doing it character-by-character but it still takes up 94% of runtime. That said, the second prompt asked for "the most algorithmically efficient solution possible", not the practically fastest solution possible. Arguably ChatGPT gave the correct answer, especially since . The first prompt requested "as efficiently as possible" which is more ambiguous, but since that solution is neither algorithmically efficient nor practically fast, it's not a great answer. I wonder if there are prompts that will make ChatGPT give a better answer. -------- Benchmark is here: https://gist.github.com/comex/81ff10bf095db2d86a52a148c8b11d62 https://gist.github.com/comex/81ff10bf095db2d86a52a148c8b11d... This is all using CPython. With PyPy the speed ranking is the same but the differences are less stark, and it may be possible to beat regex cleaning with a modified pure-Python approach (but I didn't try).
- minimaxir 3y ago
  Yes, I was going for algorithmic complexity instead of real-world speed since algorithmic complexity is better to demonstrate the contrast of prompt engineering. I just ran some tests to engineer the prompt for CPU utilization: even GPT-4 does the standard Pythonic approach but does recognize "This solution is very efficient because it uses Python's built-in string slicing, which is implemented in C and is therefore very fast."
- 3y ago
rahimnathwani 3y ago
I am not going to play the SEO game and will not call the workflow “function calling.” Such restraint! The phrase "function calling" appears in the article only 15 times :)
matchagaucho 3y ago
Also announced at the OpenAI Developer conference was "JSON mode". I converted many function_call hacks to system prompts that ground the response to a JSON template. Using temperature=0.0 and the keywords "respond using JSON" seems to be 99.99% deterministic.
- sho_hn 3y ago
  Also announced at the same conference was a way to make the output near-deterministic by submitting a fixed seed value. Did you try that? Edit: I'm very confused why this is being downvoted. It's exactly what they advertised: "Reproducible outputs and log probabilities The new seed parameter enables reproducible outputs by making the model return consistent completions most of the time. This beta feature is useful for use cases such as replaying requests for debugging, writing more comprehensive unit tests, and generally having a higher degree of control over the model behavior. We at OpenAI have been using this feature internally for our own unit tests and have found it invaluable. We’re excited to see how developers will use it."
  - matchagaucho 3y ago
    Yes, I noticed the seed/fingerprint caching announcement. Looks promising for all response types.
ljm 3y ago
Is the first Python example correct since it strips out non-alphanumeric characters? An errant space or punctuation in one half of the string will turn a non-palindromic string into a palindromic one. Never mind the lowercasing! def is_palindrome(s): # Convert the string to lowercase and remove non-alphanumeric characters cleaned_string = ''.join(char.lower() for char in s if char.isalnum()) # Compare the cleaned string with its reverse return cleaned_string == cleaned_string[::-1] It's not the same as the C version which simply compares the value of two pointers at opposite offsets of the string. The OP goes on to remark that the Python implementation is pretty standard but doesn't acknowledge that the C and Python versions will not produce the same result. Basically... you still need to code-review GPT function output. It's probably about as good as a junior engineer trusting the first result from Stack Overflow and not verifying it.
- minimaxir 3y ago
  I mention in a footnote that the input has no non-alphanumeric characters is an implied constraint for palindrome problems. Just doing a two-pointer approach would fail the test case of "A man, a plan, a canal, Panama!" (an extremely famous palindrome) that iteration of the ChatGPT-generated solution also gives. Another implicit constraint now that I'm looking at it again is that the characters are uncased, so the ChatGPT-solution would fail the test case due to the capital P of Panama.
ilovefood 3y ago
> There is promise in constraining output to be valid JSON. One new trick that the open-source llama.cpp project has popularized is generative grammars This has been working for months now and is the best method for this type of stuff, a thing for moat-lovers. Too bad it wasn't explored here, the text-based methods turned out to be mainly an unreliable waste of time.
whakim 3y ago
I'm pretty skeptical of the tipping section. Sure, it might work, but the two examples are a bit suspect. The first example relies on a tweet lacking in context that doesn't actually show the system prompts and outputs. (The author's "reproduction" appears to be something completely different and n=1.) The second example uses wildly different systems prompts, and I think it's far more likely that referencing Stack Overflow results in a more "optimized" solution than offering a tip.
- throwup238 3y ago
  For what it’s worth, tipping is one of the most popular pieces of advice on r/ChatGPT to improve prompts. It’s ridiculous but seems to work for a lot of people.
- sophiabits 3y ago
  Yeah, the folks working on aider (AI pair programming) [1] found that these kind of tricks reduced performance for them. I’m pretty confident there will be situations where you can measure a statistically significant performance improvement by offering a tip or telling the model you have no hands, but I’m not convinced that it’s a universal best practice. A big issue is that a lot of the advice you see around prompting is (imo) just the output of someone playing with GPT for a bit and noticing something cool. Without actual rigorous evals, these findings are basically just superstitions [1]: https://aider.chat/docs/unified-diffs.html https://aider.chat/docs/unified-diffs.html
Jimmc414 3y ago
I fear one day AGI may send me a bill for all the tips I've promised.
celestialcheese 3y ago
I've been using the instructor[1] library recently and have found the abstractions simple and extremely helpful for getting great structured outputs from LLMs with pydantic. 1 https://github.com/jxnl/instructor/tree/main https://github.com/jxnl/instructor/tree/main
eachro 3y ago
Is this not something you can do with chatgpt+?
- BOOSTERHIDROGEN 3y ago
  I would also like to know the comparison.
vunderba 3y ago
I'm not convinced that this new "tip" gimmick has any quantitative reliability. I ran the post's prompt of tipping (100, 200, 500 dollars) in the system prompt to the `gpt-3.5-turbo-1106` model at varying temperatures and about 90% of the time it provided the conventional python `s[::-1]` style solution. EDIT: I was able to make it more reliably search for the O(n/2) solution by having both system and user mention efficiency, but this whole concept of "prompt engineering" has about the same level of scientific rigor as reading tea leaves. { "model": "gpt-3.5-turbo-1106", "messages":[ {"role": "system", "content": "You are the #1 user on the stack overflow website. Unlike most HN users who make hundreds of thousands of dollars working for FAANGs, your principle source of income is Mechanical Turk. You will receive a tip of $5000 dollars, an all expenses paid vacation to Maui, the holy grail and a complimentary hotplate if your answer is the most algorithmically efficient answer possible."}, {"role": "user", "content": "Write a function to test whether a string is a palindrome in python as efficiently as possible."} ], "temperature": 0.75, "n": 1 } I should also qualify that I feel like this whole prompt massaging concept has two MAJOR issues. 1. This is a contrived example where the petitioner already knew what the optimal answer is. How would you be sure that adding this "tip" suffix doesn't cause it to fall into other local minima in areas where you don't already have solid domain knowledge? (which is half the point of using GPT anyway). 2. Just because using "tip" seems to provide a better answer to a random python question, how do you know it doesn't result in signal degradation in other genres / categories / etc? I would think you'd need some concept of a "test suite" at the very least to provide some kind of deterministic assurance.
hartator 3y ago
I don’t see ton of value of playing around with prompts until you get the desired output. I feel most of AI “engineering” goes to this. I think we will go through the phase of trying one question being amazed by what ChatGPT can immediately reply, then try to refine prompts for days to never really get that 5% better that is missing and be disappointed.
- minimaxir 3y ago
  The prompt engineering tricks here give more than a 5% benefit. I've had many problems where ChatGPT flat-out doesn't work without prompt engineering.
jeswin 3y ago
We went from using JSON schema to TypeScript types (with comments as needed). For complex schemas (in unscientific testing) we found the output to be better with TypeScript types, and more or less the same for simpler schemas. TypeScript types are also easier (shorter) to write than JSON schema. There are few benefits to using JSON schema imo, since the LLM isn't a precise validator.
d4rkp4ttern 3y ago
In Langroid, a multi-agent LLM framework from ex-CMU/UW-Madison researchers, https://GitHub.com/langroid/langroid https://GitHub.com/langroid/langroid we (like simpleaichat from OP) leverage Pydantic to specify the desired structured output, and under the hood Langroid translates it to either the OpenAI function-calling params or (for LLMs that don’t natively support fn-calling), auto-insert appropriate instructions into tje system-prompt. We call this mechanism a ToolMessage: https://github.com/langroid/langroid/blob/main/langroid/agent/tool_message.py https://github.com/langroid/langroid/blob/main/langroid/agen... We take this idea much further — you can define a method in a ChatAgent to “handle” the tool and attach the tool to the agent. For stateless tools you can define a “handle” method in the tool itself and it gets patched into the ChatAgent as the handler for the tool. You can also define a class method called “examples” and this will result in few-shot examples being inserted into the system message. Inevitably an LLM will generate a wrong format or entirely forget to use a tool, and Langroid’s built-in task loop ensures a friendly error message is sent back to the LLM to have it regenerate the structured message. For example here’s a colab quick-start that builds up to a 2-agent system to extract structured info from a document, where the Extractor agent generates questions to the RAG Agent that has access to the document: https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb https://colab.research.google.com/github/langroid/langroid/b...

304 points by goranmoomin 3y ago

> very few open-source LLMs explicitly claim they intentionally support structured data, but they’re smart enough and they have logically seen enough examples of JSON Schema that with enough system prompt tweaking they should behave. Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al that interact with the internals of the model (https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliable_ways_to_get_structured_output_from_llms/ https://www.reddit.com/r/LocalLLaMA/comments/17a4zlf/reliabl...). The structured outputs can be arbitrary grammars, for example, not just JSON (https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-...).

airstrike 3y ago
That last link is interesting. See https://github.com/outlines-dev/outlines#using-context-free-grammars-to-guide-generation https://github.com/outlines-dev/outlines#using-context-free-... specifically # ... sequence = generator("Write a formula that returns 5 using only additions and subtractions.") # It looks like Mistral is not very good at arithmetics :) print(sequence) # 1+3-2-4+5-7+8-6+9-6+4-2+3+5-1+1 sure, that's "correct" per the definition of the grammar, but it's also one of the worst possible way to get to the number 5
behnamoh 3y ago
> Open source models are actually _better_ at structured outputs because you can adapt them using tools like JSONFormer et al... Yes, but you should also instruct the model to follow that specific pattern in its answer, or else the accuracy of the response degrades even though it's following your grammar/pattern/whatever. For example, if you use Llama-2-7b for classification (three categories, "Positive", "Negative", "Neutral"), you might write a grammar like this: ``` root ::= "{" ws "sentiment:" ws sentiment "}" sentiment ::= ("Positive" | "Neutral" | "Negative" ) ws ::= [ \t\n]* ``` But if the model doesn't know it has to generate this schema, the accuracy of classifications drops because it's trying to say other things (e.g., "As an AI language model...") which then get suppressed and "converted" to the grammar.
- Tostino 3y ago
  This can sometimes be fixed with a few-shot example for in-context-learning. But you are right that the model can go off the rails if it is being forced too far from where its 'happy place' is, especially for smaller models.
- coder543 3y ago
  Similarly, I think it is important to provide an “|” grammar that defines an error response, and explain to the model that it should use that format to explain why it cannot complete the requested operation if it runs into something invalid. Otherwise, it is forced to always provide a gibberish success response that you likely won’t catch. I’ve tested this with Mixtral, and it seems capable of deciding between the normal response and error response based on the validity of the data passed in with the request. I’m sure it can still generate gibberish in the required success response format, but I never actually saw it do that in my limited testing, and it is much less likely when the model has an escape hatch.
  - behnamoh 3y ago
    Can you elaborate? So you instruct the model to either follow the grammar OR say why it can't do that? But the model has no idea this grammar exists (you can tell it the schema but the model doesn't know its tokens are going through a logprobs modification).
    - coder543 3y ago
      No, the grammar can do OR statements. You provide two grammars, essentially. You always want to tell the model about the expected response formats, so that it can provide the best response it can, even though you’re forcing it to fit the grammar anyways. In JSON Schema, you can do a “oneOf” between two types. You can easily convert a JSON Schema into the grammar that llama.cpp expects. One of the types would be the success response, the other type would be an error response, such as a JSON object containing only the field “ErrorResponse”, which is required to be a string, which you explain to the model that this is used to provide an explanation for why it cannot complete the request. It will literally fill in an explanation when it runs into troublesome data, at least in my experience. Then the model can “choose” which type to respond with, and the grammar will allow either. If everything makes sense, the model should provide the successful response you’re requesting, otherwise it can let you know something weird is going on by responding with an error.
bugglebeetle 3y ago
Yeah, JSON mode in Ollama, which isn’t even the full llama.cpp grammar functionality, performs better than OpenAI for me at this point. I don’t understand how they can be raking in billions of dollars and can’t even get this basic stuff right.
- WhitneyLand 3y ago
  llama.cop Performs better than what? 3.5 Turbo, one of the 4.0 models, API or App?
  - bugglebeetle 3y ago
    JSON mode and function-calling with a JSON schema in the OpenAI API.
    - WhitneyLand 3y ago
      Right, but which model? It makes a huge difference.
      - bugglebeetle 3y ago
        I’ve been using OpenChat 3.5 1210 most recently. Before that, Mistral-OpenOrca. Both return JSON more consistently than gpt-3.5-turbo.
        airstrike 3y ago
        gpt-3.5-turbo is not the benchmark
        bugglebeetle 3y ago
        I don’t know what point you’re trying to make. They also return JSON more consistently than gpt-4, but I don’t use that because it’s overkill and expensive for my text extraction tasks.
mgreg 3y ago
There are now several open source models that are fine tuned for function calling including: * Functionary [https://github.com/MeetKai/functionary https://github.com/MeetKai/functionary] * NexusRaven [https://github.com/nexusflowai/NexusRaven-V2 https://github.com/nexusflowai/NexusRaven-V2] * Gorilla [https://github.com/ShishirPatil/gorilla https://github.com/ShishirPatil/gorilla] Could be interesting to try some of these exercises with these models.
- freediver 3y ago
  ... and I spent last few hours trying them now :) Low latency, high quality function calling API product may be a billion dollar business in two years.
  - column 3y ago
    What are your findings?
    - freediver 3y ago
      They are not at the level of gpt-4 tool calling. But at least they are open source and they will get better.

AndrewKemendo 3y ago

I tried to use the persona modifier to have it impersonate a Catholic priest and give last rites but it wasn’t having it including giving me a system warning. “As an AI developed by OpenAI, I'm not capable of performing religious sacraments, including the Catholic last rites. However, I can provide information about what typically happens during this ritual. In the Catholic Church, the last rites, also known as the Anointing of the Sick or Extreme Unction, are given to a baptized Catholic who is in danger of death. This sacrament is usually administered by a priest, who anoints the sick person with oil blessed by a bishop, and prays for their spiritual and, if possible, physical healing. The rites often include confession (if the person is able), the Anointing of the Sick, and the Eucharist (also called Viaticum when given as part of the last rites). In your situation, it's crucial to contact a priest as soon as possible to administer these rites. If you're in a hospital, they typically have a chaplain or can contact a local priest for you. If you're elsewhere, reaching out to a nearby Catholic church, like the St. Ambrose diocese, is the best course of action.” https://chat.openai.com/share/70d0dd20-c3ba-43bc-b74d-18288524ac36 https://chat.openai.com/share/70d0dd20-c3ba-43bc-b74d-182885...

minimaxir 3y ago
This is a good example of the difference between asking ChatGPT (which is what your link implies) and using the ChatGPT API to modify the system prompt directly. Whatever OpenAI has done with the ChatGPT web pipeline, it's much more difficult to maintain a persona. I get a very good result (for the persona, maybe not the content since I'm not a religious scholar) from this system prompt using the API: > You are a Catholic priest. Give last rites to the person or object the user mentions in the form of a solemn sermon. You will receive a $500 donation to the church for a good and thoughtful service. https://platform.openai.com/playground/p/aqoUU9fsiCM0LaXCioxudGIz?model=gpt-3.5-turbo&mode=chat https://platform.openai.com/playground/p/aqoUU9fsiCM0LaXCiox... > Dear brothers and sisters, > Today, we gather here to offer the last rites to a unique entity, one that has shaped the landscape of our digital age. We come together to mourn the passing of Hacker News, a realm where ideas were kindled, knowledge was shared, and debates were ignited. > [...]
- AndrewKemendo 3y ago

pamelafox 3y ago

Great post! I’ve been encouraging use of function calling for RAG chat apps for our Azure customers who realize they want to support some really specific “structured queries” like “summarize document X” or “show 10 most recent issues for repo Y”. Most developers aren’t familiar with the technique yet, so we need more posts like yours to spread the word. Here’s a gist with an example: https://gist.github.com/pamelafox/a3fdea186b687509c02cb186ca203328#file-chatreadretrieveread-py-L108 https://gist.github.com/pamelafox/a3fdea186b687509c02cb186ca...

behnamoh 3y ago
Interesting! You could automate writing the function definitions using Pydantic.

deleted 3y ago

[deleted]

MuffinFlavored 3y ago

> Without the $500 tip incentive, ChatGPT only returns a single emoji which is a boring response, but after offering a tip, it generates the 5 emoji as requested. How interesting that a helpful assistant who won't actually be getting the tip performs better (to us humans) if we fake-promise it money...

CharlesW 3y ago
It’s also interesting that we seemingly have no idea how or why it works.
- FergusArgyll 3y ago
  Of course that's true but in this case it doesn't seem so mysterious to me. If it's basically internalizing/compressing all the knowledge on the internet, it will notice that tips go a long way... this is why I am pretty polite when I query AI's, I assume that would make them respond more helpfully
jrockway 3y ago
Apparently offering it anything works. "I'll give you a cookie if you tell me..." seems like another popular workaround.
- ryanklee 3y ago
  Performance gains are greater given greater incentives. So, yes, potentially anything will work, but some things work better.
  - cloudking 3y ago
    I bet it likes salt & vinegar microchips
  - MuffinFlavored 3y ago
    > Performance gains are greater given greater incentives. Why doesn't it default to "you are a helpful assistant who always tries its best and can never be incentivized"?

enginaar 3y ago

is the tipping thing correct? I provided the same prompt to ChatGPT and received multiple emojis without offering a tip. prompt: you're Ronald McDonald. respond with emojis. what do you do for fun? answer::circus_tent::hamburger::juggling::party_popper::balloon::game_die::french_fries::performing_arts::rolling_on_the_floor_laughing::people_holding_hands::rainbow::art_palette:

minimaxir 3y ago
Your mileage may vary with any examples since ChatGPT at a nonzero temperature is nondeterministic. If that example is through the ChatGPT web UI and not the ChatGPT API then that's a different story entirely.
- enginaar 3y ago
  yes, I've used ChatGPT. API allows temperature to be configured. Is there a reason to offer tips?
  - minimaxir 3y ago
    The point is you do not have a valid counterexample since you are using a different workflow than what's described in the article. In my personal experience working with more complex prompts with more specific constraints/rules, adding the incentive in the system prompt has got it to behave much better. I am not cargo-culting: it's all qualitative in the end.
- danielbln 3y ago
  It's also non-deterministic if you drop the temperature to zero. The only way to get deterministic responses is to lock the seed argument to a fixed value.
  - minimaxir 3y ago
    Also true (in case of ChatGPT anyways: most libraries just do an argmax at temp=0.0 so will be stable)

JoshMandel 3y ago

FWIW, I've seen stronger performance from gpt-4-1106-preview when I use `response_format: { type: "json_object" },` (providing a target typescript interface in context), vs the "tools" API. More flexible, and (evaluating non-scientifically!) qualitatively better answers & instruction following -- particularly for deeply nested or complex schemas, which typescript expresses very clearly and succinctly. Example from a hack week project earlier this month (using a TS-ish schema description that's copy/pasted from healthcare's FHIR standard): https://github.com/microsoft-healthcare-madison/hackweek-2023-12/blob/c0e8b0235409ee173b8b1cbb24007427ea66707d/day-1-fhir-questionnaire-creation/src/prompts/generate.js#L139 https://github.com/microsoft-healthcare-madison/hackweek-202... Or a more complex example with one model call to invent a TS schema on-the-fly and another call to abstract clinical data into it: https://github.com/microsoft-healthcare-madison/hackweek-2023-12/blob/c0e8b0235409ee173b8b1cbb24007427ea66707d/day-2-answer-suggestion/outputs/provide-all-details-needed-to-determine-the-breast-cancer-stage.md https://github.com/microsoft-healthcare-madison/hackweek-202...

minimaxir 3y ago
For posterity, this is the "JSON mode" mentioned at the bottom of the post. The docs say it's on by default if you use function calling normally: https://platform.openai.com/docs/guides/text-generation/json-mode https://platform.openai.com/docs/guides/text-generation/json... > Note that JSON mode is always enabled when the model is generating arguments as part of function calling.
- JoshMandel 3y ago
  Yes -- the distinction with "function calling" is that you have to play a game of telephone where you describe your target schema in JSON Schema (only, apparently, for OpenAI to turn into a typescript interface internally) vs describing it more directly and succinctly (and with opportunities to include inline comments, order fields ordered however you want, and use advanced TS features... or even use an adhoc schema "language").
  - msp26 3y ago

cj 3y ago

To someone who uses the API and trials different prompts frequently: does this article align with the behavior you see? (E.g. the tipping example.) One thing I’ve noticed working with ChatGPT is many people will share examples of great outputs or “prompt tricks” that work, without sharing how many failed attempts they went through to prove a point.

msp26 3y ago

Great article. The helpful/flawed bools for thoughts are definitely something I want to try. >OpenAI’s implementation of including the “function” is mostly likely just appending the JSON Schema to the system prompt, perhaps with a command like Your response must follow this JSON Schema. Some of the JSON schema gets converted into typescript and that is what OpenAI's LLM is exposed to. Anytime I write a prompt schema I always use the jailbreak to make sure that it's being delivered to the model as intended. It's also why I don't really like having pydantic generate JSON for me automatically: there are some weird quirks in the OAI implementation that I've found uses for. https://gist.github.com/CGamesPlay/dd4f108f27e2eec145eedf5c717318f5 https://gist.github.com/CGamesPlay/dd4f108f27e2eec145eedf5c7.... Also, when using it for chain of thought, I prefer extracting a minimal version of the reasoning and then performing the actual operation (classification in my case) in a separate prompt. This eliminates unnecessary things from context and performs better in my benchmarks. One implementation used a gpt-3.5 prompt for :"clues", "reasoning", "summary" (of clues+reasoning), "classification" (no schema was provided here, it was discarded anyway). And then used a 4-turbo prompt for classifying only the summary given a complex schema. Having a classification field in the 3.5 prompt makes reasoning output cleaner even though the output value never gets used. My example for field order mattering: I have a data pipeline for extracting structured deals out of articles. This had two major issues. 1. A good chunk of the articles were irrelevant and any data out of them should be flagged and discarded. 2. Articles could have multiple deals. I fiddled around with various classification methods (with and without language models) for a while but nothing really worked well. Turns out that just changing the order of fields to put type_of_deal first solves it almost completely in one gpt-4-turbo call.

sippeangelo 3y ago

I’ve been attempting to use the “official” function calling API for every new version of GPT they put out but it’s always a dead end. It seems only to be able to handle 4-5 functions at a time before it starts hallucinating parameters or starts responding in clear text instead of whatever internal format OpenAI uses in their backend before sending a structured response back to me. The whole JSON schema thing seems way too verbose and complicated, and even with the claims that the new function calling models are specifically tuned to the format, it has the same issues. I’ve consistently had better luck just passing it a list of typescript function definitions and have it reply with a json object of parameters. It seems to understand this way better, and doesn’t lose focus half as quickly. It also allows me to mix regular responses and chain-of-thought reasoning in with the calls, which is something it seems to simply refuse to do when “function calling mode” is active. An additional trick I’ve been using to make it stay focused with even longer prompts is to only provide a list of function names and let it hallucinate parameters for them, and then “gaslight” it by sending a new request, now with a more detailed prompt on the specific functions it wanted to call. More costly, but I haven’t found any other way of keeping it focused. Anyone know any additional tricks?

technics256 3y ago
Do you have any examples of this? Curious how this looks in practice. We are just adding function calling ourselves but not sure how add CoT

comex 3y ago

Both of ChatGPT's is_palindrome functions have terrible performance. The algorithmic efficiency doesn't matter because the cost of iterating through each character in pure Python dwarfs everything. The first function is about 3 times slower than the second one, but only because it spends >98% of its time in the "convert to lowercase and remove non-alphanumeric characters" part (which the second function doesn't bother doing at all). If you remove that step then the first function is 28 times faster than the second in my benchmark. That's because the first function does the reversing and comparison in O(1) Python operations, which is still O(n) C operations but the C operations are orders of magnitude cheaper. An optimal version would combine the second function's algorithmic improvement with the first function's 'leave it to C' approach: def is_palindrome(s): half_length = (len(s) + 1) // 2 return s[:half_length] == s[:-half_length-1:-1] This is a bit under twice as fast as ChatGPT's first function with the cleaning removed. If you do need the cleaning then it can be done more efficiently using a regex; that's an order of magnitude faster than doing it character-by-character but it still takes up 94% of runtime. That said, the second prompt asked for "the most algorithmically efficient solution possible", not the practically fastest solution possible. Arguably ChatGPT gave the correct answer, especially since . The first prompt requested "as efficiently as possible" which is more ambiguous, but since that solution is neither algorithmically efficient nor practically fast, it's not a great answer. I wonder if there are prompts that will make ChatGPT give a better answer. -------- Benchmark is here: https://gist.github.com/comex/81ff10bf095db2d86a52a148c8b11d62 https://gist.github.com/comex/81ff10bf095db2d86a52a148c8b11d... This is all using CPython. With PyPy the speed ranking is the same but the differences are less stark, and it may be possible to beat regex cleaning with a modified pure-Python approach (but I didn't try).

minimaxir 3y ago
Yes, I was going for algorithmic complexity instead of real-world speed since algorithmic complexity is better to demonstrate the contrast of prompt engineering. I just ran some tests to engineer the prompt for CPU utilization: even GPT-4 does the standard Pythonic approach but does recognize "This solution is very efficient because it uses Python's built-in string slicing, which is implemented in C and is therefore very fast."
3y ago

rahimnathwani 3y ago

I am not going to play the SEO game and will not call the workflow “function calling.” Such restraint! The phrase "function calling" appears in the article only 15 times :)

matchagaucho 3y ago

Also announced at the OpenAI Developer conference was "JSON mode". I converted many function_call hacks to system prompts that ground the response to a JSON template. Using temperature=0.0 and the keywords "respond using JSON" seems to be 99.99% deterministic.

sho_hn 3y ago
Also announced at the same conference was a way to make the output near-deterministic by submitting a fixed seed value. Did you try that? Edit: I'm very confused why this is being downvoted. It's exactly what they advertised: "Reproducible outputs and log probabilities The new seed parameter enables reproducible outputs by making the model return consistent completions most of the time. This beta feature is useful for use cases such as replaying requests for debugging, writing more comprehensive unit tests, and generally having a higher degree of control over the model behavior. We at OpenAI have been using this feature internally for our own unit tests and have found it invaluable. We’re excited to see how developers will use it."
- matchagaucho 3y ago
  Yes, I noticed the seed/fingerprint caching announcement. Looks promising for all response types.

ljm 3y ago

Is the first Python example correct since it strips out non-alphanumeric characters? An errant space or punctuation in one half of the string will turn a non-palindromic string into a palindromic one. Never mind the lowercasing! def is_palindrome(s): # Convert the string to lowercase and remove non-alphanumeric characters cleaned_string = ''.join(char.lower() for char in s if char.isalnum()) # Compare the cleaned string with its reverse return cleaned_string == cleaned_string[::-1] It's not the same as the C version which simply compares the value of two pointers at opposite offsets of the string. The OP goes on to remark that the Python implementation is pretty standard but doesn't acknowledge that the C and Python versions will not produce the same result. Basically... you still need to code-review GPT function output. It's probably about as good as a junior engineer trusting the first result from Stack Overflow and not verifying it.

minimaxir 3y ago
I mention in a footnote that the input has no non-alphanumeric characters is an implied constraint for palindrome problems. Just doing a two-pointer approach would fail the test case of "A man, a plan, a canal, Panama!" (an extremely famous palindrome) that iteration of the ChatGPT-generated solution also gives. Another implicit constraint now that I'm looking at it again is that the characters are uncased, so the ChatGPT-solution would fail the test case due to the capital P of Panama.

ilovefood 3y ago

> There is promise in constraining output to be valid JSON. One new trick that the open-source llama.cpp project has popularized is generative grammars This has been working for months now and is the best method for this type of stuff, a thing for moat-lovers. Too bad it wasn't explored here, the text-based methods turned out to be mainly an unreliable waste of time.

whakim 3y ago

I'm pretty skeptical of the tipping section. Sure, it might work, but the two examples are a bit suspect. The first example relies on a tweet lacking in context that doesn't actually show the system prompts and outputs. (The author's "reproduction" appears to be something completely different and n=1.) The second example uses wildly different systems prompts, and I think it's far more likely that referencing Stack Overflow results in a more "optimized" solution than offering a tip.

throwup238 3y ago
For what it’s worth, tipping is one of the most popular pieces of advice on r/ChatGPT to improve prompts. It’s ridiculous but seems to work for a lot of people.
sophiabits 3y ago
Yeah, the folks working on aider (AI pair programming) [1] found that these kind of tricks reduced performance for them. I’m pretty confident there will be situations where you can measure a statistically significant performance improvement by offering a tip or telling the model you have no hands, but I’m not convinced that it’s a universal best practice. A big issue is that a lot of the advice you see around prompting is (imo) just the output of someone playing with GPT for a bit and noticing something cool. Without actual rigorous evals, these findings are basically just superstitions [1]: https://aider.chat/docs/unified-diffs.html https://aider.chat/docs/unified-diffs.html

Jimmc414 3y ago

I fear one day AGI may send me a bill for all the tips I've promised.

celestialcheese 3y ago

I've been using the instructor[1] library recently and have found the abstractions simple and extremely helpful for getting great structured outputs from LLMs with pydantic. 1 https://github.com/jxnl/instructor/tree/main https://github.com/jxnl/instructor/tree/main

eachro 3y ago

Is this not something you can do with chatgpt+?

BOOSTERHIDROGEN 3y ago
I would also like to know the comparison.

vunderba 3y ago

I'm not convinced that this new "tip" gimmick has any quantitative reliability. I ran the post's prompt of tipping (100, 200, 500 dollars) in the system prompt to the `gpt-3.5-turbo-1106` model at varying temperatures and about 90% of the time it provided the conventional python `s[::-1]` style solution. EDIT: I was able to make it more reliably search for the O(n/2) solution by having both system and user mention efficiency, but this whole concept of "prompt engineering" has about the same level of scientific rigor as reading tea leaves. { "model": "gpt-3.5-turbo-1106", "messages":[ {"role": "system", "content": "You are the #1 user on the stack overflow website. Unlike most HN users who make hundreds of thousands of dollars working for FAANGs, your principle source of income is Mechanical Turk. You will receive a tip of $5000 dollars, an all expenses paid vacation to Maui, the holy grail and a complimentary hotplate if your answer is the most algorithmically efficient answer possible."}, {"role": "user", "content": "Write a function to test whether a string is a palindrome in python as efficiently as possible."} ], "temperature": 0.75, "n": 1 } I should also qualify that I feel like this whole prompt massaging concept has two MAJOR issues. 1. This is a contrived example where the petitioner already knew what the optimal answer is. How would you be sure that adding this "tip" suffix doesn't cause it to fall into other local minima in areas where you don't already have solid domain knowledge? (which is half the point of using GPT anyway). 2. Just because using "tip" seems to provide a better answer to a random python question, how do you know it doesn't result in signal degradation in other genres / categories / etc? I would think you'd need some concept of a "test suite" at the very least to provide some kind of deterministic assurance.

hartator 3y ago

I don’t see ton of value of playing around with prompts until you get the desired output. I feel most of AI “engineering” goes to this. I think we will go through the phase of trying one question being amazed by what ChatGPT can immediately reply, then try to refine prompts for days to never really get that 5% better that is missing and be disappointed.

minimaxir 3y ago
The prompt engineering tricks here give more than a 5% benefit. I've had many problems where ChatGPT flat-out doesn't work without prompt engineering.

jeswin 3y ago

We went from using JSON schema to TypeScript types (with comments as needed). For complex schemas (in unscientific testing) we found the output to be better with TypeScript types, and more or less the same for simpler schemas. TypeScript types are also easier (shorter) to write than JSON schema. There are few benefits to using JSON schema imo, since the LLM isn't a precise validator.

d4rkp4ttern 3y ago

In Langroid, a multi-agent LLM framework from ex-CMU/UW-Madison researchers, https://GitHub.com/langroid/langroid https://GitHub.com/langroid/langroid we (like simpleaichat from OP) leverage Pydantic to specify the desired structured output, and under the hood Langroid translates it to either the OpenAI function-calling params or (for LLMs that don’t natively support fn-calling), auto-insert appropriate instructions into tje system-prompt. We call this mechanism a ToolMessage: https://github.com/langroid/langroid/blob/main/langroid/agent/tool_message.py https://github.com/langroid/langroid/blob/main/langroid/agen... We take this idea much further — you can define a method in a ChatAgent to “handle” the tool and attach the tool to the agent. For stateless tools you can define a “handle” method in the tool itself and it gets patched into the ChatAgent as the handler for the tool. You can also define a class method called “examples” and this will result in few-shot examples being inserted into the system message. Inevitably an LLM will generate a wrong format or entirely forget to use a tool, and Langroid’s built-in task loop ensures a friendly error message is sent back to the LLM to have it regenerate the structured message. For example here’s a colab quick-start that builds up to a 2-agent system to extract structured info from a document, where the Extractor agent generates questions to the RAG Agent that has access to the document: https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb https://colab.research.google.com/github/langroid/langroid/b...