4 ms·

Show HN: Waffie, an LLM API automated testing tool

4 points by lamroger 3y ago

Hey yall, Very early release but wanted to get the idea out there. Waffie is built for developers looking to operationalize LLM APIs into their solutions. The first thing I'm tackling is automated testing of LLMs in the context of unstructured to structured data interpretation. For example, sentiment analysis. Specify what you want in a YAML file and a test.csv and run waffie against your test cases. Looking to bridge the gaps between software development and machine learning engineering.

tikkun 3y ago
One thing that could help: Add to the top of the readme something that basically shows "put in this / do this" and "get this" Kind of like a before and after. Basically, I'd recommend exposing 2-3 of the examples on the readme, otherwise the readme just shows how to run it, which doesn't show what it does / what it's for.
- lamroger 3y ago
  Done! Appreciate the quick thoughts
  - tikkun 3y ago
    Nice, I think it makes sense to me. So you'd provide it for example with a csv with prompt and expected completion, and then a test file, and then you can run that test file and see how it does on those sample prompts.
    - lamroger 3y ago
      Yep!