4 ms·

Very interesting thoughts. This is a whole discussion on its own. The ones that I mulled over that I feel have legs in my line of thinking are: > minimize cos

by jarodrh 3mo ago

Very interesting thoughts. This is a whole discussion on its own. The ones that I mulled over that I feel have legs in my line of thinking are: > minimize cost to converge upon sufficiently low error problem - This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that via sampled prompt logs. > Evals and OpenInference (OpenTelemetry) might be useful. - Evals already a given. This is frugon's wheelhouse, however it currently only supports OpenAI-style jsonl. I'll definitely consider adding support for OpenInference. > A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter. - This is an intriguing idea. This would actually enhance/strengthen frugon's stance on the "minimize cost to converge upon sufficiently low error problem". Noted.

westurner 3mo ago
> minimize cost to converge upon sufficiently low error problem Is this a convex optimization or non-convex optimization problem? > Evals Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp; https://news.ycombinator.com/item?id=48532642 https://news.ycombinator.com/item?id=48532642 OpenInference and OpenTelemetry have a schema for agent sessions. I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perhaps such an index is also useful for agent optimization > Nonce Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters
- jarodrh 3mo ago
  Down the rabbit hole we go. > Is this a convex optimization or non-convex optimization problem? - Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small. > OpenInference and OpenTelemetry have a schema for agent sessions. - Good to know for when I look into it. > Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters - Yes, this is a current weakness in frugon...Hmm, I see, so I'd essentially need to establish a baseline (with whatever the default temp is) + nonce, then compare both spreads. Noted. Good thread. Give frugon a try on your own logs, and tell me what you think.
  - westurner 3mo ago
    As well. Someday costed opcodes and sharded redundancy for all of this; > No search needed because the problem is discrete and small. If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function (that assumes or imposes a distance metric to make a metric space). Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.
    - jarodrh 3mo ago
      > As well. Someday costed opcodes and sharded redundancy for all of this; Hmm, noted. > If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function. - Yes, what I'm saying is that frugon has a small enough candidate pool that it can all be evaluated at the same time. It doesn't need to search through it. > Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard. - This is exactly the shape the judge prompt takes in frugon. See: src/frugon/measure.py Then search: JUDGE_PROMPT_TEMPLATE

by jarodrh 3mo ago

westurner 3mo ago

> minimize cost to converge upon sufficiently low error problem Is this a convex optimization or non-convex optimization problem? > Evals Re: pytest-evals, mcpbr, agentevals, foundry-toolkit; and devtools-mcp; https://news.ycombinator.com/item?id=48532642 https://news.ycombinator.com/item?id=48532642 OpenInference and OpenTelemetry have a schema for agent sessions. I finally discovered agentsview and ctxrs/ctx for indexing and searching agent session logs. Perhaps such an index is also useful for agent optimization > Nonce Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

jarodrh 3mo ago
Down the rabbit hole we go. > Is this a convex optimization or non-convex optimization problem? - Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small. > OpenInference and OpenTelemetry have a schema for agent sessions. - Good to know for when I look into it. > Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters - Yes, this is a current weakness in frugon...Hmm, I see, so I'd essentially need to establish a baseline (with whatever the default temp is) + nonce, then compare both spreads. Noted. Good thread. Give frugon a try on your own logs, and tell me what you think.
- westurner 3mo ago
  As well. Someday costed opcodes and sharded redundancy for all of this; > No search needed because the problem is discrete and small. If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function (that assumes or imposes a distance metric to make a metric space). Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.
  - jarodrh 3mo ago
    > As well. Someday costed opcodes and sharded redundancy for all of this; Hmm, noted. > If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function. - Yes, what I'm saying is that frugon has a small enough candidate pool that it can all be evaluated at the same time. It doesn't need to search through it. > Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard. - This is exactly the shape the judge prompt takes in frugon. See: src/frugon/measure.py Then search: JUDGE_PROMPT_TEMPLATE