3 ms·
LLMs are not inherently indeterministic. Batching, temperature, and other things make them appear so when run by big providers but a locally-run LLM model at ze
by semiquaver 1y ago
LLMs are not inherently indeterministic. Batching, temperature, and other things make them appear so when run by big providers but a locally-run LLM model at zero temperature will always produce the same output given the same input.
- oytis 1y agoThat's an improvement, they are still "chaotic" though in that small changes in input can change the output unpredictably strong
- behnamoh 1y agoYes, this paper says exactly what you talked about: https://arxiv.org/abs/2404.01332 https://arxiv.org/abs/2404.01332
- lmeyerov 1y agoThat assumes they were implemented with deterministic operators, which isn't the default assumption when using neural network libs on GPUs. Imagine random seeds, cublas optimizations - like you can configure all these things, but I wouldn't assume it, esp in GPU-optimized OSS..