24 ms·
I am describing a simple eval. This is a strategy that determines how effective an AI system is, they have existed long before LLMs were ever a thing, and they
by johnfn 1y ago
I am describing a simple eval. This is a strategy that determines how effective an AI system is, they have existed long before LLMs were ever a thing, and they are perfectly happy to deal with non-deterministic behavior. You are acting as if LLMs being non-deterministic means you can't say anything about them, but before LLMs were non-deterministic, traditional ML systems were non-deterministic, and we had no problem building systems to probabilistically evaluate them.
- troupo 1y agoA "simple eval" would give you nothing. Anthropoc running out of available compute will affect the output significantly more than a change in the model or "thinking". > we had no problem building systems to probabilistically evaluate them. Something tells me the "no problem" in that statement is from Death of Stalin when you get to the actual details of that evaluation: https://youtu.be/kasSSZlBFDs?si=-5kv7LPWi_YStR3C https://youtu.be/kasSSZlBFDs?si=-5kv7LPWi_YStR3C
- johnfn 1y agoDo you have any evidence to support your claim that "Anthropoc running out of available compute will affect the output significantly" has ever actually happened in practice? To be honest, it sounds like superstition.
- troupo 1y agoIt's called speculation, not superstition. I've already been in several situations when I'd run the same prompt over the same code with completely different results (one near genius, one near moronic). The differentiating factor? Genius was Stockholm morning, moron was Stockholm midday/afternoon.