3 ms·
Is it possible to have an LLM always give a consistent output?
Is there any LLM who's focus is this? EG:
1. Given the same exact tokens,
2. Always return the same tokens.
EG:
I give the LLM the following tokens:
"What color is the sky?"
it always would return a message like:
"It's Purple!"
and never change. I understand this not a normal usecase or in many cases desired for an LLM, but I am curious if anyone is working on this in an academic paper, or if there is a private organization which is doing something along these lines.
- twosdai 11mo agoTo clarify I am valuing consistency in the output to the token level, not the actual information of the content.
- cratermoon 11mo agomaybe, if you set the temperature to 0, but by nature the math is stochastic, not deterministic.
- yorwba 11mo agoDeterministic output is a property of the inference framework, not the model. E.g. https://docs.sglang.ai/advanced_features/deterministic_inference.html https://docs.sglang.ai/advanced_features/deterministic_infer... https://github.com/ggml-org/llama.cpp/pull/16016 https://github.com/ggml-org/llama.cpp/pull/16016
- twosdai 11mo agoReally appreciate this, this is exactly what I was looking for.
- compressedgas 11mo agoUse a fixed sampler seed.
- danenania 11mo agoIf you need this, could you just put your own kv cache in front?
- vunderba 11mo agoDo you mean: Given a user input X/Y/Z that "resolves" to the target information of "sky color", respond with "Purple"? That sounds to me more like sticking a classifier and/or vector similarity database interceptor in front of an LLM and pre-empt with cached response. Otherwise I'm not sure I understand the question. If you just want EXACT TOKEN INPUT => EXACT TOKEN OUTPUT then it's just a KVP as @danenania mentioned.