3 ms·

> I'd love to see some sort of % success before and after "think deep" was added. Should I be adding "think deep" to all my non-trivial queries to Claude Any s

by troupo 1y ago

> I'd love to see some sort of % success before and after "think deep" was added. Should I be adding "think deep" to all my non-trivial queries to Claude Any such % would be meaningless because both are non-deterministic blackboxes with literally undefined behavior. Any % you'd be seeing could be just differences in available compute as the US is waking up

johnfn 1y ago
I am describing a simple eval. This is a strategy that determines how effective an AI system is, they have existed long before LLMs were ever a thing, and they are perfectly happy to deal with non-deterministic behavior. You are acting as if LLMs being non-deterministic means you can't say anything about them, but before LLMs were non-deterministic, traditional ML systems were non-deterministic, and we had no problem building systems to probabilistically evaluate them.
- troupo 1y ago
  A "simple eval" would give you nothing. Anthropoc running out of available compute will affect the output significantly more than a change in the model or "thinking". > we had no problem building systems to probabilistically evaluate them. Something tells me the "no problem" in that statement is from Death of Stalin when you get to the actual details of that evaluation: https://youtu.be/kasSSZlBFDs?si=-5kv7LPWi_YStR3C https://youtu.be/kasSSZlBFDs?si=-5kv7LPWi_YStR3C
  - johnfn 1y ago
    Do you have any evidence to support your claim that "Anthropoc running out of available compute will affect the output significantly" has ever actually happened in practice? To be honest, it sounds like superstition.
    - troupo 1y ago
      It's called speculation, not superstition. I've already been in several situations when I'd run the same prompt over the same code with completely different results (one near genius, one near moronic). The differentiating factor? Genius was Stockholm morning, moron was Stockholm midday/afternoon.

by troupo 1y ago

johnfn 1y ago

I am describing a simple eval. This is a strategy that determines how effective an AI system is, they have existed long before LLMs were ever a thing, and they are perfectly happy to deal with non-deterministic behavior. You are acting as if LLMs being non-deterministic means you can't say anything about them, but before LLMs were non-deterministic, traditional ML systems were non-deterministic, and we had no problem building systems to probabilistically evaluate them.

troupo 1y ago
A "simple eval" would give you nothing. Anthropoc running out of available compute will affect the output significantly more than a change in the model or "thinking". > we had no problem building systems to probabilistically evaluate them. Something tells me the "no problem" in that statement is from Death of Stalin when you get to the actual details of that evaluation: https://youtu.be/kasSSZlBFDs?si=-5kv7LPWi_YStR3C https://youtu.be/kasSSZlBFDs?si=-5kv7LPWi_YStR3C
- johnfn 1y ago
  Do you have any evidence to support your claim that "Anthropoc running out of available compute will affect the output significantly" has ever actually happened in practice? To be honest, it sounds like superstition.
  - troupo 1y ago
    It's called speculation, not superstition. I've already been in several situations when I'd run the same prompt over the same code with completely different results (one near genius, one near moronic). The differentiating factor? Genius was Stockholm morning, moron was Stockholm midday/afternoon.