zambelli

6 ms·

61.
by zambelli 5mo ago
Best guess is it's native mode. The function calling template is just broken for Nemo. I did go with an extreme example in the post (but true). Other deltas are smaller but still statistically significant. 30 pt swing between llamserve
62.
by zambelli 5mo ago
I was surprised as well. I did go with an extreme (but true) example in the post. In this case, native function-calling template likely is in play. However, that doesn't explain the Lamaserver prompt vs llamafile at ~ +4pts, or vs Ol
63.
by zambelli 5mo ago
At the moment TieredCompact is naive. It uses context thresholds the consumer determines and fires when those thresholds are hit. It just does different things at different threshold levels. Your idea of using task shape to dynamically set
64.
by zambelli 5mo ago
Yeah, it's a useful framework even with frontier. And it definitely lifts "cheap" frontier models like Haiku into more solid territory. I haven't done a ton of forge integrations into frontier (like pointing claude code
65.
by zambelli 5mo ago
Forge does have tiered compaction, and it's configurable! Defaults are currently probably a bit on the high side for catching effective attention, but that might be a part of the code that interests you the most. src/forge/co
66.
by zambelli 5mo ago
That is the whole challenge, actually! A new metric I'm going to dogfood into forge is ETTWS - estimated time to working solution. A simple retry loop around your whole workflow could, in some cases, be all you need . But it could mea
67.
by zambelli 5mo ago
That's where frontier pulls ahead for sure, at least on the big frontier models - though I haven't formalized those findings because...time. Necessary disclaimer, forge isn't concerned, technically, with model quality, just e
68.
by zambelli 5mo ago
Oh no! I have code-hammer coming out soon :D. Everyone is building stuff these days :p. Always happy to see folks looking into small local models!
69.
by zambelli 5mo ago
Mainly because I have plenty of use cases and not all of them need or want pi. Forge isn't an orchestration framework and is not coding specific, it lives one level lower - if I understand pi correctly. The proxy mode should integrate
70.
by zambelli 5mo ago
I've had a few reversions as well along the way, including in upcoming v0.7.0 patch. Some models benefitted, others regressed - overall better on harder scenarios or I wouldn't be releasing, but yeah - not intuitive. The biggest c
71.
by zambelli 5mo ago
Essentially, yes that's right! There's some subtlety in how to let it know it was wrong (returning things as tool errors because it trained on that), but that's the gist of it - sort of a self-correcting architecture.
72.
by zambelli 5mo ago
That's good feedback, thank you! I have an update landing shortly so I'll make sure to clarify in the docs! I appreciate it!
73.
by zambelli 5mo ago
Lol, I love that framing. Yeah, the small models have impressed me a lot during this work. The reasoning can be quite good, and definitely sufficient for a lot of cases. Just gotta nudge em back on track Every now and then and they'll
74.
by zambelli 5mo ago
I have an open GitHub issue for macOS hardware detection. I don't have a Mac myself to do dev on but happy to accept a fork! I did assign a buddy to that issue but she's been slacking - call her out :p. Latency is dependent on the
75.
by zambelli 5mo ago
Conceptually I think definitely! Forge has no opinion on what the agent should be trying to do, that's the "middleware"'s job, so to speak. Forge is just trying to make sure that when the model decides to do something, t
76.
by zambelli 5mo ago
Yes it does! I haven't published those evals yet, but I'm actually running 24-35B class models on a custom coding harness built on forge (even 120B class recently). I just need more GPU wall clock time to get more evals done. ETA
77.
by zambelli 5mo ago
Partially correct, but an important distinction to call out. You don't have to define the workflow steps. You can just expose the set of tools to the model and let the LLM call whatever it wants in any order, and every guardrail except
78.
by zambelli 5mo ago
Yeah it's a delicate balance between precise and silly, and too permissive. I'm definitely still iterating on forge, but so far sending the model a friendly and gracefully handled error message works wonders (instead of barfing a
79.
by zambelli 5mo ago
Nice! I'm not surprised at your findings (anymore). Mechanical reliability is the key to small models, and it's a big unlock. I've seen the same thing you just described. And the agnostic nudges forge sends at inspired by exa
80.
by zambelli 5mo ago
Yes that's correct ! /v1/chat/completions is the entry point. In proxy mode, here's what forge applies on each request (handler.py builds these): Response validation: ResponseValidator(tool_names) checks each tool c
81.
by zambelli 5mo ago
I only just skimmed it, but will try to dive deeper in a bit. I think we share a lot on tool definitions/schemas. Forge will let a consumer define a tool, set of tools, pydantic schema for each, etc. outlines seems to be similar with t
82.
by zambelli 5mo ago
Interesting! I'll look into that. Would mean another dep/integration but might be more robust.
83.
by zambelli 5mo ago
Hi! Yes, I definitely think so. I've seen variance across all model families I looked at. The magnitude changes, but the presence of variance is a constant.
84.
by zambelli 5mo ago
That's awesome! Let me know if quick start is causing issues or anything else you'd like to dig into. Out of curiosity, what models are you running?
85.
by zambelli 5mo ago
I definitely use LLMs to help write things - but this is my draft! Maybe I've been spending too much time reading the evals and I now sound like an LLM... Either way, here I am - happy to answer any questions!
86.
by zambelli 5mo ago
Thank you! It was a really fun rabbit hole to fall into and I found a bunch of counterintuitive stuff. I'm in the same boat, tuning models wasn't super interesting, though I might do a focused spike on behavior -focused fine tun
87.
by zambelli 5mo ago
Neat! Historically I've been most active on LinkedIn but the AI community seems very X-leaning so I'll make sure to pay closer attention there. Good luck with the harness, happy to connect!
88.
by zambelli 5mo ago
Very cool! I would look at the tokens returned by each of the calls. You can map those to API costs per input/output tokens. Forge should be capturing those (or can, as passthrough from llama.cpp). At least, if I understand your econom
89.
by zambelli 5mo ago
Absolutely, benchmarks are a different breed. Forge's eval is deliberately scoped as a stress test of the recovery loop, not a measure of end-to-end agentic quality. Scenarios range from basic 2-step workflows, to more complex ones wit
90.
by zambelli 5mo ago
Very tangential! I'll try but it might take me a while.

More ›