2 ms·

One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the not

One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the notion that we’d do enough testing to generate a statistically significant gauge of performance - man, I know it’s correct, but I’m not sure my company will survive the process.

dvogel 6d ago
For these tests, why not tune the temperature and such to reduce the randomness and convert them to almost-always-succeeds vs almost-always-fails? Is it not the iteration count that drives up the cost?
- sarchertech 6d ago
  If turn the temperature down for tests, they won’t match production behaviors. If you turn it down too far in production, the output will just be bad.
  - dvogel 6d ago
    Isn't the goal of the author to get reproducible behavior out of the agent though? I would thinking turning the temperature down would serve that production goal too.
    - roughly 6d ago
      right, this is the whole problem - the stochastic behavior is both the goal and the problem. If you want your tests to match production, you need to get a reasonable sample size, which costs real money.
    - sarchertech 6d ago
      2 things. 1. If you turn the temperature down too far, the output is just bad and no amount of running prompts optimization will let you hill climb your way to good performance. 2. It’s not about determinism vs non-determinism. It’s about chaos. A perfectly deterministic model is still chaotic. Meaning that very small changes to the input result in very large changes to the output. Turning temperature down doesn’t actually get you predictable or reproducible behavior across different inputs.
      - roughly 5d ago
        Thank you, I forgot about the chaotic aspect, too - that’s the other part that makes this brutal. The bot performs perfectly on your tests, but your customer abhors the Oxford comma, so you, your marketing team and your test suite can go to hell.
daveguy 6d ago
Maybe a better approach would be to stop trying to get a product out of a slot machine?
- roughly 6d ago
  But I got real lucky once!

dvogel 6d ago

For these tests, why not tune the temperature and such to reduce the randomness and convert them to almost-always-succeeds vs almost-always-fails? Is it not the iteration count that drives up the cost?

sarchertech 6d ago
If turn the temperature down for tests, they won’t match production behaviors. If you turn it down too far in production, the output will just be bad.
- dvogel 6d ago
  Isn't the goal of the author to get reproducible behavior out of the agent though? I would thinking turning the temperature down would serve that production goal too.
  - roughly 6d ago
    right, this is the whole problem - the stochastic behavior is both the goal and the problem. If you want your tests to match production, you need to get a reasonable sample size, which costs real money.
  - sarchertech 6d ago
    2 things. 1. If you turn the temperature down too far, the output is just bad and no amount of running prompts optimization will let you hill climb your way to good performance. 2. It’s not about determinism vs non-determinism. It’s about chaos. A perfectly deterministic model is still chaotic. Meaning that very small changes to the input result in very large changes to the output. Turning temperature down doesn’t actually get you predictable or reproducible behavior across different inputs.
    - roughly 5d ago
      Thank you, I forgot about the chaotic aspect, too - that’s the other part that makes this brutal. The bot performs perfectly on your tests, but your customer abhors the Oxford comma, so you, your marketing team and your test suite can go to hell.

daveguy 6d ago

Maybe a better approach would be to stop trying to get a product out of a slot machine?

roughly 6d ago
But I got real lucky once!