4 ms·
Isn’t your (b) only because of the addition of a random seed?
by nandomrumber 4mo ago
Isn’t your (b) only because of the addition of a random seed?
- aesthesia 4mo agoLLM inference can be implemented in a way where nondeterminism depends only on the random seed, but that's not common. It ends up being more efficient/easier to implement kernels whose exact results depend on how many other prompts are being processed in parallel. See https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in... for a pretty extensive exploration.
- andrewaylett 4mo agoThat's still nondeterminism as an implementation choice, rather than as a fundamental limitation of the technology.