AFAIK the batch your query lands in can also matter[1]. Though I imagine this should be a smaller effect than different quantization levels say. [1]: https://

AFAIK the batch your query lands in can also matter[1]. Though I imagine this should be a smaller effect than different quantization levels say. [1]: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...