3 ms·
>These things aren’t deterministic so it’s hard to even use things like trial-and-error experiments to figure out what works, which as a computer scientist I fi
by dTal 3y ago
>These things aren’t deterministic so it’s hard to even use things like trial-and-error experiments to figure out what works, which as a computer scientist I find completely infuriating!
Nitpick: they can be made "deterministic" in a strict sense by using a deterministic sampling scheme, for instance by turning "temperature" down to 0. This isn't necessarily helpful for prompt experimentation, though - when deployed, prompts are usually combined with other input, so you risk simply substituting the non-determinism of sampling with the unpredictability of whether it will work with all inputs.
- soroushjp 3y agoIn my own experiments with OpenAI's GPT-4 API with temperature set to zero, I was still not getting deterministic outputs, with some small variations between completions. Not sure why, and I haven't had a chance to dig further or talk to their team about why and how this happens.
- mirekrusin 3y agoNon-determinism in GPT-4 is caused by Sparse MoE [0] [0] https://news.ycombinator.com/item?id=37006224 https://news.ycombinator.com/item?id=37006224
- deleted 3y ago[deleted]
- pillefitz 3y agoYou don't have to put the temperature to 0, you can just use a fixed random seed, couldn't you?
- phillipcarter 3y agoI think the right term to use here is "more reliable". Setting temperature to 0 will certainly make outputs more reliable for the same inputs. But GPT is inherently nondeterministic both by distributing lossy computations across different devices and using several models under the hood. There's no getting around that when you have enough scale.