3 ms·
Happy to answer questions about the eval methodology, the backend findings, or anything in the repo. I'll be around.
by zambelli 5mo ago
Happy to answer questions about the eval methodology, the backend findings, or anything in the repo. I'll be around.
- fabian_shipamax 5mo agodashboard link is dead
- schaefer 5mo agosuper interesting work. It will take me a few days to dig in and really understand it. But I'm looking forward to it. I run small models at home, so I'm very curious.
- zambelli 5mo agoThat's awesome! Let me know if quick start is causing issues or anything else you'd like to dig into. Out of curiosity, what models are you running?
- zambelli 5mo agoThanks to everyone for the great discussion! v0.7.0 is out now. It was in flight when this landed - changes tool error channel based on dogfooding observations with some larger models, eval re-run (numbers shift but within CI), and most importantly docs updated! I hope they're clearer now.
- zambelli 5mo agov0.7.1 is now out - Forge can now sit behind Claude Code! Proxy mode can talk to supported backends and handles format translation. Anthropic > forge > OpenAI; or Anthropic > forge > Anthropic. Will get this ported over into vLLM work and try to get that released soon. Thanks to some kind folks who contributed Docker, token counting, and a handful of PRs I haven't gotten to yet.
- c0rruptbytes 5mo agobeen testing forge with ternary bonsai 8b mlx 2bit, pretty sweet even if the model is limited - real potential with this project, good luck!! - Broad slice: - Full Forge: 48/72 accurate, 72/72 complete, score 66.7% - Bare: 18/72 accurate, 24/72 complete, score 25.0% - Lift: +30 correct runs, no paired regressions - Bare had 42 ToolCallErrors and 6 ToolExecutionErrors; full Forge had none. - Advanced reasoning: - Full Forge: 3/24 accurate, 24/24 complete, score 12.5% - Bare: 3/24 accurate, 9/24 complete, score 12.5% - Lift: completion improved, but accuracy did not.