4 ms·

Kind of amazes me how many people bitch about agent performance but don't hook their guys up to Otel, crack Phoenix and get to work, but instead randomly tweak

Kind of amazes me how many people bitch about agent performance but don't hook their guys up to Otel, crack Phoenix and get to work, but instead randomly tweak prompts in response to team vibes.

chrisweekly 1y ago
Good point. Also (tangent), I followed your profile link to https://sibylline.dev https://sibylline.dev and am thoroughly impressed. Stoked to have found your treasure trove of repos and insights.
- CuriouslyC 1y ago
  Don't play with them unless you're good at debugging alpha code (claude/codex can do it fine), I haven't ironed out env specific stuff or clarified the installation/usage, and I'm still doing UI polish/optimization passes (yay async simd rust). I'll do showy releases once I've got the tools one click install ready, in the meantime please feel free to drop an issue on any of my projects if there are features or questions you have.
  - chrisweekly 1y ago
    Sounds good, will do. Good luck getting them polished up!
yahoozoo 1y ago
Could you elaborate? How does knowing numerical usage metrics help?
- CuriouslyC 1y ago
  With Phoenix + Clickhouse being fed from Otel, you can do queries over your traces for deep analysis. If I want to see which tool calls are failing and why (or just get tool statistics), or find common patterns in flagged/failure traces ("simpler solution") and their causes, it's one query and some wiring.
  - thewisenerd 1y ago
    you mention traces but the documentation only specifies metrics and logs https://docs.claude.com/en/docs/claude-code/monitoring-usage https://docs.claude.com/en/docs/claude-code/monitoring-usage am i missing something here? or is this a case where they were lazy to do traces in the claude code sdk, so its logs + log attributes?
    - CuriouslyC 1y ago
      You can get traces out of every agent framework that doesn't suck. You might need to collect from a proxy to get them from Claude because Anthropic is bigtime amateur hour.

Kind of amazes me how many people bitch about agent performance but don't hook their guys up to Otel, crack Phoenix and get to work, but instead randomly tweak prompts in response to team vibes.

chrisweekly 1y ago

Good point. Also (tangent), I followed your profile link to https://sibylline.dev https://sibylline.dev and am thoroughly impressed. Stoked to have found your treasure trove of repos and insights.

CuriouslyC 1y ago
Don't play with them unless you're good at debugging alpha code (claude/codex can do it fine), I haven't ironed out env specific stuff or clarified the installation/usage, and I'm still doing UI polish/optimization passes (yay async simd rust). I'll do showy releases once I've got the tools one click install ready, in the meantime please feel free to drop an issue on any of my projects if there are features or questions you have.
- chrisweekly 1y ago
  Sounds good, will do. Good luck getting them polished up!

yahoozoo 1y ago

Could you elaborate? How does knowing numerical usage metrics help?

CuriouslyC 1y ago
With Phoenix + Clickhouse being fed from Otel, you can do queries over your traces for deep analysis. If I want to see which tool calls are failing and why (or just get tool statistics), or find common patterns in flagged/failure traces ("simpler solution") and their causes, it's one query and some wiring.
- thewisenerd 1y ago
  you mention traces but the documentation only specifies metrics and logs https://docs.claude.com/en/docs/claude-code/monitoring-usage https://docs.claude.com/en/docs/claude-code/monitoring-usage am i missing something here? or is this a case where they were lazy to do traces in the claude code sdk, so its logs + log attributes?
  - CuriouslyC 1y ago
    You can get traces out of every agent framework that doesn't suck. You might need to collect from a proxy to get them from Claude because Anthropic is bigtime amateur hour.