4 ms·

But temperature 0 LLM's don't exhibit the emergent phenomena we like, even in apparently non-creative tasks. The randomness is, in some sense, a cheap proxy fo

But temperature 0 LLM's don't exhibit the emergent phenomena we like, even in apparently non-creative tasks. The randomness is, in some sense, a cheap proxy for an infeasible search over all completion sequences, much like simulated annealing with zero temperature is a search for a local optimum but adding randomness makes it explore globally and find more interesting possibilities.

sebzim4500 2y ago
Sure but you could add pseudo random noise instead and get the same behavior while retaining determinism.
- refulgentis 2y ago
  Temperature is at ~1.2 in this thread, here's some 0.0: - Yes, temperature 0.0 is less creative. - Injecting pseudo-random noise to get deterministic creative outputs is "not even wrong", in the Wolfgang Pauli sense. It's fixing something that isn't broken, with something that can't fix it, that if it could, would be replicating the original behavior - more simply, it's proposing non-deterministic determinism. - Temperature 0.0, in practice, is an LLM. There aren't emergent phenomena, in the sense "emergent phenomena" is used with LLMs, missing. Many, many, many, applications use this. - In simplistic scenarios, on very small models, 0.0 could get stuck literally repeating the same token. - There's a whole other layer of ex. repeat penalties/frequency penalties and such that are used during inference to limit this. Only OpenAI and llama.cpp expose repeat/frequency. - Temperature 0.0 is still non-deterministic on ex. OpenAI, though substantially the same, and even the same most of the time. It's hard to notice differences. (Reproducible builds require extra engineering effort, the same way ensuring temperature = 0.0 is truly deterministic requires engineering effort.) - Pedantically, only temperature 0.0 at the same seed (initial state) is deterministic.
  - deleted 2y ago
    [deleted]

sebzim4500 2y ago

Sure but you could add pseudo random noise instead and get the same behavior while retaining determinism.

refulgentis 2y ago
Temperature is at ~1.2 in this thread, here's some 0.0: - Yes, temperature 0.0 is less creative. - Injecting pseudo-random noise to get deterministic creative outputs is "not even wrong", in the Wolfgang Pauli sense. It's fixing something that isn't broken, with something that can't fix it, that if it could, would be replicating the original behavior - more simply, it's proposing non-deterministic determinism. - Temperature 0.0, in practice, is an LLM. There aren't emergent phenomena, in the sense "emergent phenomena" is used with LLMs, missing. Many, many, many, applications use this. - In simplistic scenarios, on very small models, 0.0 could get stuck literally repeating the same token. - There's a whole other layer of ex. repeat penalties/frequency penalties and such that are used during inference to limit this. Only OpenAI and llama.cpp expose repeat/frequency. - Temperature 0.0 is still non-deterministic on ex. OpenAI, though substantially the same, and even the same most of the time. It's hard to notice differences. (Reproducible builds require extra engineering effort, the same way ensuring temperature = 0.0 is truly deterministic requires engineering effort.) - Pedantically, only temperature 0.0 at the same seed (initial state) is deterministic.
- deleted 2y ago
  [deleted]