4 ms·
This all clicked for me when I got to this example in the README: async def main(): # This is the prompt we want to run. # Note how the
by simonw 3y ago
This all clicked for me when I got to this example in the README:
async def main():
# This is the prompt we want to run.
# Note how the prompt doesn't mention a number of vehicles or how to format the result.
prompt = "What are the most popular types of vehicles?\n"
# Tell the model to generate the prompt string, ie. let's start with the prompt "to complete"
await aici.FixedTokens(prompt)
# Store the current position in the token generation process
marker = aici.Label()
for i in range(1,6):
# Tell the model to generate the list number
await aici.FixedTokens(f"{i}.")
# Wait for the model to generate a vehicle name and end with a new line
await aici.gen_text(stop_at = "\n")
await aici.FixedTokens("\n")
# Store the tokens generated in a result variable
aici.set_var("result", marker.text_since())
aici.start(main())
This is a similar pattern to llama.cpp grammars - it's a way to directly manipulate the token generation phase, such that you can occasionally take over from the model and say "here we are outputting '1.' - then back to you for the bullet item".
The most obvious usage of this is forcing a model to output valid JSON - including JSON that exactly matches a given schema (like OpenAI Functions if they were 100% reliable).
That Python code is a really elegant piece of API design.
- winwang 3y agoAnd now I'm waiting for the "LLM" monad.
- nerpderp82 3y agoWasm is monadic.
- viksit 3y agocould you eli5 how wasm being monadic is helpful / and how an llm monad would be too?
- zarathustreal 3y agoIt’s just a useful way to communicate about it, giving it a name. If you’re knee-deep in OOP land you can just call it a “builder”
- winwang 3y agoI mean a lawful(ish) monadic type/api, i.e. `LLM a` in FP-land. Otherwise, every sequential machine/program can be said to be monadic.
- refulgentis 3y agollama.cpp is orders of magnitude easier. Rather than controlling token by token, with an imperative statement for each, we create a static grammar to describe ex. a JSON schema. I'm honestly unsure what this offers over that, especially because I'm one of 3 groups with a WASM llama.cpp, and you can take it from me, you don't want to use it. (~3 tokens/sec with a 3B model on MVP M2 Max/Ultra/whatever they call top of line for MBP. About 2% of perf of Metal, and I'd bet 10% of running on CPU without WASM. And there's no improvement in sight)
- simonw 3y agoI find these imperative statements much less intimidating than llama.cpp grammars - the Python code I copied here looks a lot more approachable to me than the gbnf syntax from https://raw.githubusercontent.com/ggerganov/llama.cpp/master/grammars/json_arr.gbnf https://raw.githubusercontent.com/ggerganov/llama.cpp/master... I don't think the key idea here is to run llama.cpp itself in WASM - it's to run LLMs in native code, but have fast custom-written code from end-users that can help pick the next token. WASM is a neat mechanism for that because many different languages can use it as a compile target, and it comes with a robust sandbox by default.
- mmoskal 3y agoIt's only the controller that runs in Wasm, not the inference. The pyctrl is a just a sample controller, you can write a controller that takes any kind of grammar (eg., a yacc grammar [0] - the python code in that example is only used for glueing). Llama.cpp grammars were quite slow in my testing (20ms per token or so, compared to 2ms for the yacc grammar referenced above). [0] https://github.com/microsoft/aici/blob/main/controllers/pyctrl/samples/cfg.py#L7 https://github.com/microsoft/aici/blob/main/controllers/pyct...
- jameshart 3y agoGreat summary - thanks. And definitely closer to the LLM api surface we need to really start to use these things. There’s definitely a danger with this kind of code that the first thing this is going to generate to complete the “1.” prompt will be something like “ Truck; 2. Sedan; 3. Minivan”, though.
- mmoskal 3y agoYou generally have to tell the LLM what you want it to generate and then enforce it, i.e, the LLM has to be somewhat aligned with the constraints. Otherwise, for example if you ask for JSON it will keep generating (legal) white-space or when you ask for C code it will say: int sureICanHelpYouHereIsAnExampleOfTheCodeYouWereAskingFor; In this case however, you can just do: await aici.gen_text(regex=r"[a-zA-Z\n]+" stop_at = "\n") Also note that there still a lot of work to figure out how it's easiest for the programmer of an LLM-enabled app to express these things - AICI is meant to make it simple to implement different surface syntaxes.
- giovannibonetti 3y agoGood point. I believe it could be solved with backtracking, just like it is done in compilers/lexers.
- mirekrusin 3y agoThis is great generic idea. It's also possible to wrap it in something user friendly a'la [0]. [0] https://github.com/ollama/ollama/issues/3019 https://github.com/ollama/ollama/issues/3019
- mmoskal 3y agoWhat you describe there is a great example of a custom controller - could be implemented on top of pyctrl, jsctrl, or just natively in Rust.
- nighthawk454 3y agoIsn't that also similar to the (formerly?) Microsoft Guidance project? https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance
- mmoskal 3y agoWe believe Guidance can run on top of AICI (we're working on efficient Earley parser for that [0], together with local Guidance folks). AICI is generally lower level (though our sample controllers are at similar level to Guidance). [0] https://github.com/microsoft/aici/blob/main/controllers/aici_abi/src/earley/parser.rs https://github.com/microsoft/aici/blob/main/controllers/aici...
- nighthawk454 3y agoAh, that clears it up, thank you!
- dartos 3y agoHow does it compare to GBNF grammars or LMQL?
- mmoskal 3y agoEach of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly more expressible than CFGs. In addition to constraining output, AICI also lets you fork the generation programmatically as well as dynamically edit the prompt which is important in some scenarios. The idea is that several LLM inference systems implement AICI as an interface, and Guidance, LMQL, etc implement AICI backend and then everyone wins.
- dartos 3y agoIs the benefit only that WASM (and things that target it) is used as the constraint language?
- babyshake 3y agoAlthough don't you still have problems with being able to accurately anticipate the output for control flow purposes? In this example, stopping at a new line. I guess that should work out fine in this example, although if I'm understanding this paradigm correctly, to help ensure that you'd want to have something like: await aici.FixedTokens(f"Here is a bullet point list:\n\n.") before entering into the for-loop.
- PheonixPharts 3y ago> The most obvious usage of this is forcing a model to output valid JSON Isn't this something that Outlines [0], Guidance [1] and others [2] already solved much more elegantly? 0. https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines 1. https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance 2. https://github.com/sgl-project/sglang https://github.com/sgl-project/sglang
- emrek1 3y agoEach of those is great, and more sophisticated strategies are being developed all the time: incorporating static analysis at inference time to ensure code is more than just syntactically correct [0]; to multi-LLM call agent-based frameworks [1]; decomposing hard tasks into many "easier" LLM tasks [2]; analyzing internal activations to identify potential hallucinations [3]; activation steering to control generations [4]; and "Language Model Arithmetic" to compose biases for style, vocab, etc [5]. But each also either requires tight integration with an LLM inference/serving engine (to access low-level internals of an inference engine); or adds a lot of overhead (many individual LLM calls) if they don't have tight integration. The AI Controller Interface is creating an abstraction layer that exposes the low-level primitives so that all the above strategies can be implemented without each one needing to dive into the internals of every LLM engine. AICI doesn't support all the necessary primitives for all of these yet (e.g., we don't know what the right way is to represent internal activations); and not everything will end up fitting in a WASM module anyway. Its a start at thinking about a new layer in the inference stack [0] https://arxiv.org/abs/2306.10763 https://arxiv.org/abs/2306.10763 [1] https://github.com/microsoft/autogen https://github.com/microsoft/autogen [2] https://arxiv.org/abs/2305.14292 https://arxiv.org/abs/2305.14292 [3] https://arxiv.org/abs/2312.02073 https://arxiv.org/abs/2312.02073 [4] https://arxiv.org/abs/2308.10248 https://arxiv.org/abs/2308.10248 [5] https://arxiv.org/abs/2311.14479 https://arxiv.org/abs/2311.14479
- hansvm 3y agoKind of. The usual mechanism for those sorts of libraries is a grammar-informed-token-filter. You get an output adhering to the grammar (or an error), but the mechanism is reshaping some intermediate conditional probabilities. That's problematic because the entire point of an LLM (in most scenarios with non-trivial output sizes) is sampling from a calibrated probability distribution. The conditional mask in generation allows you to factor that into per-token work (given the preceding text, what's the probability of the next token being token X?), but none of that holds even approximately when you start tweaking the sampling parameters (the same reason why playing with "temperature" gives very bad results for a lot of complicated problems -- the greedy solution we're executing isn't globally correct). For a small, illustrative example that I've seen, suppose your trying to parse a freeform text input into a tagged text input (podcast transcripts or something), squishing the result into a json schema or something so that you can actually use it in your end application. For longer text fields, many SOTA LLMs are prone to close the field early and append an ellipses rather than flesh it out in its entirety. Constrained schema generation will have a high probability of closing the (broken) string and a low probability of continuing past the (incorrect) ellipses, contrasted with an approach of retrying till the result doesn't crash a parser, which usually will not emit a closing quote after an ellipses and this crashes all those times the output would have been invalid. The net result is that you still have an invalid result, but nothing crashed, so now you don't know that fact. This particular library also suffers from that same flaw (though the flaw is important and common, so I think it's worth repeating myself about it once in awhile), but it has two very nice features I'd like to touch on (contrasted with spec/grammar-approaches), one of which partially mitigates that flaw: 1. You can interact with partial results via a turing-complete language. It's really messy to use weaker languages to encode "this one json field is probably broken if it ends in an ellipses, but other strings probably can and should", and 2. You have your results available directly in your language of choice without an additional parsing/extraction step. Resource-wise I don't know that it matters, but tons of simple problems are much more elegantly described as "I want these things" instead of "I want this composite object -> post-process that composite object". That said, using it for json in particular looks terrible, like you said (but you could presumably wrap one of the other json libraries with this and reap some of the benefits of both approaches).
- ShamelessC 3y ago> That Python code is a really elegant piece of API design. await aici.FixedTokens("\n") # Store the tokens generated in a result variable aici.set_var("result", marker.text_since()) Not sure I agree.
- thomasfromcdnjs 3y agoIt took me a while to also figure it out, until I got to the example. I am definitely going to play with this.