4 ms·
You can have an LLM be entirely deterministic. This is by design how they work. Having the randomness added to their response is a choice.
by Tostino 3y ago
You can have an LLM be entirely deterministic. This is by design how they work. Having the randomness added to their response is a choice.
- svaha1728 3y agoThey can be deterministic, but that doesn’t mean the information they are regurgitating is correct.
- Tostino 3y agoThat isn't what I was replying to though. Factuality is different than determinism. They are not trained to retain facts. They are trained to predict the next word.
- derwiki 3y agoMaybe you can help me make this more deterministic? I am already setting temp to 0, but i keep getting different responses to the same prompt. https://gist.github.com/derwiki/99079de4cfcb4f196b1ca561b3d77a9f https://gist.github.com/derwiki/99079de4cfcb4f196b1ca561b3d7...
- Tostino 3y agoLooks to be an OpenAI problem due to how they host the API. People are speculating that different servers in their pool are ot always on the exact same version of the model, and you can get pointed to a different version...no proof I saw, but quite a few people complaining about it. All the local models I've used were deterministic with temp 0.