2 ms·
+1 on interaction terms + tails : fanout × retries × context growth is where linear token math dies. One thing we do in enzu is make “budget as constraint” exe
by teilom 9mo ago
+1 on interaction terms + tails : fanout × retries × context growth is where linear token math dies.
One thing we do in enzu is make “budget as constraint” executable: we clamp `max_output_tokens` from the budget before the call, and in multi-step/RLM runs we adapt output caps downward as the budget depletes (so it naturally gets shorter/cheaper instead of spiraling). When token counting is unavailable we explicitly enter a “budget degraded” mode rather than pretending estimates are exact.
Also agree p90/p95 cost/run matters more than averages; max-output caps are crude but effective.
Docs: https://github.com/teilomillet/enzu/blob/main/docs/PROD_MULTIPLIERS.md https://github.com/teilomillet/enzu/blob/main/docs/PROD_MULT... and https://github.com/teilomillet/enzu/blob/main/docs/BUDGET_CONTRACTS.md https://github.com/teilomillet/enzu/blob/main/docs/BUDGET_CO...