Did you read the whole article? "In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-si

Did you read the whole article? "In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism."