7 ms·

Isn't that also similar to the (formerly?) Microsoft Guidance project? https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance

mmoskal 3y ago
We believe Guidance can run on top of AICI (we're working on efficient Earley parser for that [0], together with local Guidance folks). AICI is generally lower level (though our sample controllers are at similar level to Guidance). [0] https://github.com/microsoft/aici/blob/main/controllers/aici_abi/src/earley/parser.rs https://github.com/microsoft/aici/blob/main/controllers/aici...
- nighthawk454 3y ago
  Ah, that clears it up, thank you!
- dartos 3y ago
  How does it compare to GBNF grammars or LMQL?
  - mmoskal 3y ago
    Each of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly more expressible than CFGs. In addition to constraining output, AICI also lets you fork the generation programmatically as well as dynamically edit the prompt which is important in some scenarios. The idea is that several LLM inference systems implement AICI as an interface, and Guidance, LMQL, etc implement AICI backend and then everyone wins.
    - dartos 3y ago
      Is the benefit only that WASM (and things that target it) is used as the constraint language?
      - mmoskal 3y ago
        There are two parts to AICI - one is that it uses Wasm and the other is the specific set of operations on the LLM that it exposes (which is why we call it "interface"). Having a common interface (provided it is adopted) lets you re-use the same controller over multiple LLM infra stacks (and different models), or conversely the different controllers over your own brand new LLM infra stack (or model). Using Wasm simplifies your deployment story in the cloud (sandboxing), however the AICI runtime communicates with controllers as separate processes and so if the controller is trusted it could in principle be run in native code.

mmoskal 3y ago

We believe Guidance can run on top of AICI (we're working on efficient Earley parser for that [0], together with local Guidance folks). AICI is generally lower level (though our sample controllers are at similar level to Guidance). [0] https://github.com/microsoft/aici/blob/main/controllers/aici_abi/src/earley/parser.rs https://github.com/microsoft/aici/blob/main/controllers/aici...

nighthawk454 3y ago
Ah, that clears it up, thank you!
dartos 3y ago
How does it compare to GBNF grammars or LMQL?
- mmoskal 3y ago
  Each of these can be implemented on top of AICI. AICI lets you run an arbitrary (Wasm) program to determine which tokens are allowed next - this is strictly more expressible than CFGs. In addition to constraining output, AICI also lets you fork the generation programmatically as well as dynamically edit the prompt which is important in some scenarios. The idea is that several LLM inference systems implement AICI as an interface, and Guidance, LMQL, etc implement AICI backend and then everyone wins.
  - dartos 3y ago
    Is the benefit only that WASM (and things that target it) is used as the constraint language?
    - mmoskal 3y ago
      There are two parts to AICI - one is that it uses Wasm and the other is the specific set of operations on the LLM that it exposes (which is why we call it "interface"). Having a common interface (provided it is adopted) lets you re-use the same controller over multiple LLM infra stacks (and different models), or conversely the different controllers over your own brand new LLM infra stack (or model). Using Wasm simplifies your deployment story in the cloud (sandboxing), however the AICI runtime communicates with controllers as separate processes and so if the controller is trusted it could in principle be run in native code.