3 ms·
Each of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly mor
by mmoskal 3y ago
Each of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly more expressible than CFGs.
In addition to constraining output, AICI also lets you fork the generation programmatically as well as dynamically edit the prompt which is important in some scenarios.
The idea is that several LLM inference systems implement AICI as an interface, and Guidance, LMQL, etc implement AICI backend and then everyone wins.
- dartos 3y agoIs the benefit only that WASM (and things that target it) is used as the constraint language?
- mmoskal 3y agoThere are two parts to AICI - one is that it uses Wasm and the other is the specific set of operations on the LLM that it exposes (which is why we call it "interface"). Having a common interface (provided it is adopted) lets you re-use the same controller over multiple LLM infra stacks (and different models), or conversely the different controllers over your own brand new LLM infra stack (or model). Using Wasm simplifies your deployment story in the cloud (sandboxing), however the AICI runtime communicates with controllers as separate processes and so if the controller is trusted it could in principle be run in native code.