4 ms·

LLMs don't seem to be deterministic [0, 1, 2, 3]. So no, pinning the version wouldn't be enough. [0] https://matt-rickard.com/foundational-models-are-not-enoug

LLMs don't seem to be deterministic [0, 1, 2, 3]. So no, pinning the version wouldn't be enough. [0] https://matt-rickard.com/foundational-models-are-not-enough https://matt-rickard.com/foundational-models-are-not-enough [1] https://arxiv.org/pdf/2308.02828.pdf https://arxiv.org/pdf/2308.02828.pdf [2] https://www.sitation.com/non-determinism-in-ai-llm-output/ https://www.sitation.com/non-determinism-in-ai-llm-output/ [3] https://towardsdatascience.com/the-magic-of-llms-prompt-engineering-9c3e46130131 https://towardsdatascience.com/the-magic-of-llms-prompt-engi...

adventured 3y ago
> So no, pinning the version wouldn't be enough. You can to an extent dictate GPT's determinism with settings you can pass along in the API, combined with the parent already proclaiming they saw a 100% success rate. So how do you know it wouldn't be enough? The parent is already saying their test suite indicates it is enough. What tests have you run counter to their claim to show it fails? And how do you know the parent can't increase the determinism even further beyond what they were already using in their testing (and decreasing the risk of negative outcomes by doing so)?