4 ms·

Hey Jack! Thanks for sharing this. The incremental fine-tuning of smaller and cheaper models for cost reduction is definitely a really interesting differentiato

Hey Jack! Thanks for sharing this. The incremental fine-tuning of smaller and cheaper models for cost reduction is definitely a really interesting differentiator. I had a few questions regarding the reliability of the LLM-powered functions MonkeyPatch facilitates and the testing process. How does MonkeyPatch ensure the reliability of LLM-powered functions it helps developers create, and do the tests employed provide sufficient confidence in maintaining consistent output? If tests fall short of 100% guarantee, how does MonkeyPatch address concerns similar to historical challenges faced with testing traditional LLMs? Thanks.

JackHopkins 3y ago
Heya, no worries - I’m glad to share it. MonkeyPatch ensures reliability through what we call ‘test-driven alignment’, in which the tests that reference the patched functions are guaranteed to pass. The more align ‘tests’ you create, the more rigorous a contract that the functions have to fulfil. The other way to increase consistency is using more constrained type annotations (i.e using pydantic field annotations), which is a similar concept to MarvinAI and Magentic.