6 ms·
I have a saying: "any sufficiently advanced agent is indistinguishable from a DSL" If I'm really leaning into multi-tool use for anything resembling a mutation
by madrox 2y ago
I have a saying: "any sufficiently advanced agent is indistinguishable from a DSL"
If I'm really leaning into multi-tool use for anything resembling a mutation, then I'd like to see an execution plan first. In my experience, asking an AI to code up a script that calls some functions with the same signature as tools and then executing that script actually ends up being more accurate than asking it to internalize its algorithm. Plus, I can audit it before I run it. This is effectively the same as asking it to "think step by step."
I like the idea of Command R+ but multitool feels like barking up the wrong tree. Maybe my use cases are too myopic.
- darkteflon 2y agoYou mean manually pre-baking a DAG from the user query, then “spawning” other LLMs to resolve each node and pass their input up the graph? This is the approach we take too. It seems to be a sufficiently performant approach that is - intuitively - generically useful regardless of ontology / domain, but would love to hear others’ experiences. It would be nice to know if this is sort of how OpenAI’s native “file_search” retriever works - that’s certainly the suggestion in some of the documentation but it hasn’t, to my knowledge, been confirmed.
- TZubiri 2y agoNo. The DAG should be "manually pre-baked" ( defined at compile/design time). In runtime you only parse the "user question" (user prompt) into a starting and end node, which is equivalent to a function call. So the question "What league does Messi play in?" Is parsed by the llm as League("Messi") So if your dag only contains the functions team(player) and league(team), you can still solve the question. But the llm isn't tasked with resolving the dag, that's code, let the llm chill and do what it's good at, don't make it code a for loop for you
- deleted 2y ago[deleted]
- darkteflon 2y agoThat’s very interesting. Does designing the DAG in advance imply that you have to make a new one for each particular subset of end-user questions you might receive? Or is your problem space such that you can design it once and have it be useful for everything you’re interested in? My choice of words was poor: by “pre-baking”, I just meant: generated dynamically at runtime from the user’s query, _before_ you then set about answering that query. The nature of our problem space is such that we wouldn’t be able to design DAG in advance of runtime and have it be useful everywhere. The answering process itself is then handled by deterministically (in code) resolving the dependencies of the DAG in the correct order, where each node might then involve a discrete LLM call (with function) depending on the purpose. Once resolved, a node’s output is passed to the next tier of the DAG with framing context.
- TZubiri 2y agoYou don't make a DAG for each question category. This is classic OOP, OG, Kay's version, you design subject-experts (objects) with autonomy and independency, they are just helpful in general. Each function/method, regardless of the Object/Expert, is an edge in the graph. A user question is simply a pair of vertices, call them I and O, and the execution/solution is a path between the two points, namely the input and the output. The functions are traditional software (Code, API, SQL) the job of the LLM is only to: 1- Map each type of question into a subsystem/codepath. The functional parsing solution is the most advanced. But a simple version involves asking LLM to classify a question into an enum. 2- To parse the parameters as a list of key/value tuples. The end. Don't ask the LLM to cook your food, clean your clothes or suck your dick. LLM is revolutionary at language, let it do language tasks. We are not consumers of a helpful AI assistant, we are designers of it.
- darkteflon 2y agoVery interesting perspective - thanks for your time!
- Scipio_Afri 2y agoThanks for your perspective! Can you explain why oral is a bad idea?
- TZubiri 2y agoI think you are imagining a scenario where you are using the LLM manually. Tools are designed to serve as a backend for other GPT like products. You don't have the capacity to "audit" stuff. Furthermore tool execution occurs not in the LLM but in the code that calls the LLM through API. So whatever code executes the tool, it also orders the calling sequence graph. You don't need to audit it, you are calling it.
- verdverm 2y agoPeople want to audit the args, mainly because of the potential for destructive operations like DELETE FROM and rm -rf / How do you know a malicious actor won't try to do these things? How do you protect against it?
- TZubiri 2y ago"the args" You need to be more specific. In a systems, everything but the output is an argument to something else. Even then the system output is an input to the user. So yeah, depending on what argument you are talking about you can audit it in a different way and it has different potential for abuse.
- verdverm 2y agoThe args to a function like SQL or TERMINAL
- TZubiri 2y agoI personally don't connect LLMs to SQL, but to APIs. But I'm pretty sure you would just give an SQL user to the LLM and enjoy the SQL server's built-in permissions and auditing features.
- verdverm 2y agoWhat if that user has write permissions and the LLM generates a bad UPDATE, i.e. forgets to put the WHERE clause in... even for a SELECT, how do you know the right constraints were in place and you are getting the correct data? read-only use-cases misses a whole category. All this is to get back to the point that people want to audit the LLM before running the function because of the unreliability, there is hesitance with good reason
- fzeindl 2y ago> ... code up a script that calls some functions with the same signature as tools and then executing that script actually ends up being more accurate than asking it to internalize its algorithm. This is called defunctionalization and useful without LLMs as well.
- HeatrayEnjoyer 2y agoLike what?
- TZubiri 2y ago"Agent ~=Domain Syntax Language" But Agent!=Language
- RecycledEle 2y ago> I have a saying: "any sufficiently advanced agent is indistinguishable from a DSL" I don't think you mean Digital Subscriber Line, so may I ask: What is a DSL in this context?
- patleeman 2y agoDomain Specific Language
- RecycledEle 2y agoThank you.
- ai4ever 2y agoputting the LLM in the loop makes the tooling unreliable. so, the usecases would be limited to those where accuracy is not important. whereas, a DSL still aims for accurate and deterministic modeling of the specific usecase.
- patleeman 2y agoVery curious if anybody else is working on a DSL that's meant for LLMs to output? Has anyone seen anyone using this approach? Any resources available?
- madrox 2y agoFunny enough, there's a DSL LLMs already tend to be trained on: JavaScript. Just teach the LLM the function signatures it has to work with, then execute the script in a VM