3 ms·
well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
by efficax 2mo ago
well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
- embedding-shape 2mo agoDepends on who "we" are, what you're talking about is a thing for inference providers doing batched inference and similar stuff. If you run one inference requests locally, you can actually step-by-step debug LLM output, just there is a ton of steps. But there is nothing "inherently random" or non-deterministic involved here, just optimization strategies for the large inference servers that makes it "impossible".
- solenoid0937 2mo ago> we cannot step debug an llm's output to find out what happened We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.
- Bjartr 2mo agoI'll give you that it's not wrapped up in nice product UX, but these are market choices first and technical limitations second.