3 ms·
The models themselves are mathematically deterministic. We add randomness during the sampling phase, which you can turn off when running the models locally. Th
by eightysixfour 1y ago
The models themselves are mathematically deterministic. We add randomness during the sampling phase, which you can turn off when running the models locally.
The SaaS APIs are sometimes nondeterministic due to caching strategies and load balancing between experts on MoE models. However, if you took that model and executed it in single user environment, it could also be done deterministically.
- troupo 1y ago> However, if you took that model and executed it in single user environment, Again, are those environments in the room with us? In the context of the article, is the model executed in such an environment? Do we even know anything about the environment, randomness, sampling and anything in between or have any control over it (see e.g https://news.ycombinator.com/item?id=44528930 https://news.ycombinator.com/item?id=44528930)?
- mathiaspoint 1y agoIt's very poor communication. They absolutely do not have to be non-deterministic.
- troupo 1y agoThe output of all these systems used by people not through API is non-deterministic.
- troupo 1y agoI would also assume that in vast majority of cases people don't set temperature to zero even with API calls. And even if you do set it to zero, you never know what changes to the layers and layers of wrappers and system prompts you will run into on any given day resulting in "on this day we crash for certain input, and on other days we don't": https://www.techdirt.com/2024/12/03/the-curious-case-of-chatgpts-banned-names-hard-coding-blocks-to-avoid-nuisance-threats/ https://www.techdirt.com/2024/12/03/the-curious-case-of-chat...