4 ms·
This kind of low level strategies can be tried on many models using AICI, including the substring one https://github.com/microsoft/aici/blob/main/controllers/ai
by mmoskal 3y ago
This kind of low level strategies can be tried on many models using AICI, including the substring one https://github.com/microsoft/aici/blob/main/controllers/aici_abi/src/substring.rs https://github.com/microsoft/aici/blob/main/controllers/aici...
- skybrian 3y agoI wonder if anyone has tried giving an LLM a tool that ensures that it copies a quote from a document accurately? It seems like it would be a pretty simple way to avoid some hallucinations.
- visarga 3y agoYes, it would be good for extractive QA
- worldsayshi 3y agoDo I understand it correctly if the idea here is to insert some logic into the inference loop? I might not have a clear idea how the inference loop works but it sounds like a whole host of solutions could present themselves if it was easy to plug in various types of logic at inference.
- throwup238 3y agoYes the inference loop provides a list of “logits” every loop which are possible predictions for the next token. LLM parameters like temperature and top_p configure the default logic that selects one of those logits but that logic can be replaced with anything you want. It just has to select the next token and feed it back into inference. Llama.cpp and its downstream users like ollama have long supported using BNF grammars to constrain the output that way.
- 3abiton 3y agoHow would this perform compared to RAG?
- potatoman22 3y agoI believe you'd use it with RAG. It would ensure the model properly sites the text you feed into it.