3 ms·
That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have pr
by Bjartr 2mo ago
That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable.
You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.
- efficax 2mo agowell sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
- embedding-shape 2mo agoDepends on who "we" are, what you're talking about is a thing for inference providers doing batched inference and similar stuff. If you run one inference requests locally, you can actually step-by-step debug LLM output, just there is a ton of steps. But there is nothing "inherently random" or non-deterministic involved here, just optimization strategies for the large inference servers that makes it "impossible".
- solenoid0937 2mo ago> we cannot step debug an llm's output to find out what happened We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.
- Bjartr 2mo agoI'll give you that it's not wrapped up in nice product UX, but these are market choices first and technical limitations second.