3 ms·
Given that LLMs fail to give consistent answers to the same questions, how does that factor into these studies?
by neeleshs 3y ago
Given that LLMs fail to give consistent answers to the same questions, how does that factor into these studies?
- cmauck10 3y agoMost LLMs allow you to specify the temperature parameter that governs the randomness and thus the creativity of the responses. For this experiment I used a very low temperature to ensure consistency for a given prompt.
- derwiki 3y agoWe run temp 0 for our prod app and see different results for the same prompt all the time. It’s not every prompt, but the longer ones.
- Tostino 3y agoYou can have an LLM be entirely deterministic. This is by design how they work. Having the randomness added to their response is a choice.
- svaha1728 3y agoThey can be deterministic, but that doesn’t mean the information they are regurgitating is correct.
- Tostino 3y agoThat isn't what I was replying to though. Factuality is different than determinism. They are not trained to retain facts. They are trained to predict the next word.
- derwiki 3y agoMaybe you can help me make this more deterministic? I am already setting temp to 0, but i keep getting different responses to the same prompt. https://gist.github.com/derwiki/99079de4cfcb4f196b1ca561b3d77a9f https://gist.github.com/derwiki/99079de4cfcb4f196b1ca561b3d7...
- Tostino 3y agoLooks to be an OpenAI problem due to how they host the API. People are speculating that different servers in their pool are ot always on the exact same version of the model, and you can get pointed to a different version...no proof I saw, but quite a few people complaining about it. All the local models I've used were deterministic with temp 0.