3 ms·
It's really hard to put an LLM in the middle of unix pipes because the output is unreliable I wrote one a while back, mainly so I could use it with vim (withou
by verdverm 3y ago
It's really hard to put an LLM in the middle of unix pipes because the output is unreliable
I wrote one a while back, mainly so I could use it with vim (without a plugin) and pipe my content in (also at the CLI), but haven't maintained it
https://github.com/verdverm/chatgpt https://github.com/verdverm/chatgpt
- npongratz 3y agoJustine may have addressed unreliable output by using `--temp 0` [0]. I'd agree that while it may be deterministic, there are other definitions or axes of reliability that may still make it poorly suited for pipes. [0] > Notice how I'm using the --temp 0 flag again? That's so output is deterministic and reproducible. If you don't use that flag, then llamafile will use a randomness level of 0.8 so you're certain to receive unique answers each time. I personally don't like that, since I'd rather have clean reproducible insights into training knowledge.
- jart 3y ago`--temp 0` makes it deterministic. What can make output reliable is `--grammar` which the blog post discusses in detail. It's really cool. For example, the BNF expression `root ::= "yes" | "no"` forces the LLM to only give you a yes/no answer.
- verdverm 3y agothat only works up to a point. If you are trying to transform a text based cli output into a JSON object, even with a grammar, you can get variation in the output. A simple example is field or list ordering. Omission is the real problematic one
- verdverm 3y ago> Justine may have addressed unreliable output by using `--temp 0` That only works if you have the same input. It also nerfs the model considerably https://ai.stackexchange.com/questions/32477/what-is-the-temperature-in-the-gpt-models https://ai.stackexchange.com/questions/32477/what-is-the-tem...