4 ms·
Show HN: Prompt-Engineering Tool: AI-to-AI Testing for LLM
Spelltest framework simulates conversations between AI ‘synthetic users' in an environment to test and refine LLM-based applications. It ensures your app converse with utmost accuracy and relevance. Post-chat, Spelltest assesses responses, providing qualitative and quantitative feedback on performance. Suitable for both chat and completion modes.
When to use:
- After modifying your prompt.
- When your LLM provider updates.
- As a CI step for you repo.
All feedback and collaborations appreciated!
- calyhre 3y agoSuper interesting. We've been experimenting with promptfoo[1] at my work, and this looks very similar. [1]: https://github.com/promptfoo/promptfoo https://github.com/promptfoo/promptfoo
- artas728 3y agoThanks! While promptfoo offers great strict metrics, Spelltest differentiates by allowing custom prompt-based metrics. This lets you measure unique attributes like empathy or creativity in dialogues and completions.