4 ms·
ChatGPT is powerful, but it gives you different answers to the same question from one session to the next. And research found that overall performance can vary
by Bostonian 3y ago
ChatGPT is powerful, but it gives you different answers to the same question from one session to the next. And research found that overall performance can vary over time, sometimes for the worse. So you may host your own LLM for reproducibility.
I have not tried public LLMs myself. Do they give reproducible results?
- embwbam 3y agoThey do that on purpose. The API gives you a setting you can change to 0 or 1 for maximum creativity or maximum reproducibility
- TacticalCoder 3y ago> I have not tried public LLMs myself. Do they give reproducible results? Public LLMs I don't know but images generated using StableDiffusion are, of course, totally deterministic. There really is no reason a LLM cannot be deterministic and if it isn't: fix it (even if this comes at a tiny performance cost).
- mcbuilder 3y agoIf you fix the random number seed virtually all LLMs should be deterministic. However, just 1 token difference in the input could produce a very different output, depending on the sampler, model, etc. So, LLMs can be deterministic, but in practice they are pure alchemy.
- varispeed 3y ago> it gives you different answers to the same question Sometimes answer is wrong and then right. If it is deterministic, then what if gets "stuck" on wrong answer?