5 ms·

Show HN: ChainForge, a visual tool for prompt engineering and LLM evaluation

177 points by fatso784 3y ago

Hi HN! We’re been working hard on this low-code tool for rapid prompt discovery, robustness testing and LLM evaluation. We’ve just released documentation to help new users learn how to use it and what it can already do. Let us know what you think! :)

deleted 3y ago
[deleted]
ericskiff 3y ago
We just used this on a project and it was very helpful! Cool to see it here on HN
- dekervin 3y ago
  May I ask, how was it useful ? I find it cool but I have a hard time justifying using it.
- fatso784 3y ago
  Hey Eric! Thank you! As an aside, we are looking to interview some people who’ve used ChainForge (you see, we are academics who must justify our creations through publications… crazy, I know). Would you or anyone on your team be interested in a brief chat? Can contact me here: https://twitter.com/IanArawjo https://twitter.com/IanArawjo Or find email on CV here: ianarawjo.com At any rate, glad it was helpful!
trentearl 3y ago
Cool project
- fatso784 3y ago
  Thanks!
koryk 3y ago
I like it! Any plans to add Google Vertex AI support?
- fatso784 3y ago
  There is a long term vision of supporting fine-tuning through an existing evaluation flow. We originally created this because we were worried about how to evaluate ‘what changed’ between a fine-tuned LLM and its base model. I wonder if Vertex AI has an API that we could plug-in, though, or if it’s limited to the UI.
  - koryk 3y ago
    I meant for completion, chat and embedding. https://cloud.google.com/vertex-ai/docs/generative-ai/chat/test-chat-prompts#chat-query-drest https://cloud.google.com/vertex-ai/docs/generative-ai/chat/t... . Some examples here. Vertex AI has the same API as PaLM as far as I know. However, the authorization is through Google Cloud. So I use it like any other GCP API. I love the idea of adding fine tuning as a node though. Here is the API for creating a model tuning job - https://cloud.google.com/vertex-ai/docs/generative-ai/models/tune-models https://cloud.google.com/vertex-ai/docs/generative-ai/models... I wish I could use ChainForge nodes in Node Red.
deleted 3y ago
[deleted]
maxlamb 3y ago
What exactly is prompt discovery?
- toxicFork 3y ago
  AFAIK it's finding out what prompts to use for what LLM to get the answer your want E.g. this > Compare response quality across prompt permutations, across models, and across model settings to choose the best prompt and model for your use case.
saladdressing 3y ago
[dead]
mabcat 3y ago
This looks excellent! It's a great interface for two things I'm struggling to make LlamaIndex do: explain and debug multi-step responses for agent flows, and cache queries aggressively. If I can work out how to hook it into my LlamaIndex-based pile, happy days. Feature/guidance request: how to actually call functions, how to loop on responses to resolve multiple function calls. I've managed to mock a response to get_current_weather using this contraption: https://pasteboard.co/aO9BmHG5qsFt.png https://pasteboard.co/aO9BmHG5qsFt.png . But it's messy and I can't see a way to actually evaluate function calls. And if I involve the Chat Turn node, the message sequences seem to get tangled with each other. Probably I'm holding it wrong!
- fatso784 3y ago
  Thank you for the kind words! Looking at the photo, I think you wouldn’t need the last prompt node there. As far as evaluating functions go, that’s unfortunately a ways off. But, we generally prioritize things based on how many people posted GitHub Issues about it/want it. (For instance, Chat Turn nodes came from an Issue.) If you post a feature request there, it’ll move up our priority list, and we can also clarify what the feature precisely should be.
_puk 3y ago
I think you should probably mention that its source is available! [0] I don't personally have a need for this right now, but I can really see the use for the parameterised queries, as well as comparisons across models. Thanks for your efforts! 0: https://github.com/ianarawjo/ChainForge https://github.com/ianarawjo/ChainForge
- swyx 3y ago
  "source available" means one thing, but this is properly MIT open source and i believe the authors should receive credit for that in this age of frankenlicenses EDIT: ah: "This work was partially funded by the NSF grant IIS-2107391." ok cool we the taxpayer funded it haha
  - _puk 3y ago
    I didn't want to open that can of worms: "If you use ChainForge for research purposes, or build upon the source code, we ask that you cite this project in any related publications. The BibTeX you can use for now is.." That's outside of the MIT licence as far as I'm concerned
    - 7moritz7 3y ago
      It says "we ask", not "you have to". I'd say it's open to interpretation that this is an informal request and not legally binding. Also I do want to open the can of worms and say that whoever doesn't even have the respect to include a citation on request when using someone else's work should just write everything themselves.
    - priyanmuthu 3y ago
      Hi! I’m one of the great students working on this. This is merely a request to get more visibility. It will also help us get more grants. We don’t have any intention of restricting the “openness” of it.
      - _puk 3y ago

177 points by fatso784 3y ago

deleted 3y ago

[deleted]

ericskiff 3y ago

We just used this on a project and it was very helpful! Cool to see it here on HN

dekervin 3y ago
May I ask, how was it useful ? I find it cool but I have a hard time justifying using it.
fatso784 3y ago
Hey Eric! Thank you! As an aside, we are looking to interview some people who’ve used ChainForge (you see, we are academics who must justify our creations through publications… crazy, I know). Would you or anyone on your team be interested in a brief chat? Can contact me here: https://twitter.com/IanArawjo https://twitter.com/IanArawjo Or find email on CV here: ianarawjo.com At any rate, glad it was helpful!

trentearl 3y ago

Cool project

fatso784 3y ago
Thanks!

koryk 3y ago

I like it! Any plans to add Google Vertex AI support?

fatso784 3y ago
There is a long term vision of supporting fine-tuning through an existing evaluation flow. We originally created this because we were worried about how to evaluate ‘what changed’ between a fine-tuned LLM and its base model. I wonder if Vertex AI has an API that we could plug-in, though, or if it’s limited to the UI.
- koryk 3y ago
  I meant for completion, chat and embedding. https://cloud.google.com/vertex-ai/docs/generative-ai/chat/test-chat-prompts#chat-query-drest https://cloud.google.com/vertex-ai/docs/generative-ai/chat/t... . Some examples here. Vertex AI has the same API as PaLM as far as I know. However, the authorization is through Google Cloud. So I use it like any other GCP API. I love the idea of adding fine tuning as a node though. Here is the API for creating a model tuning job - https://cloud.google.com/vertex-ai/docs/generative-ai/models/tune-models https://cloud.google.com/vertex-ai/docs/generative-ai/models... I wish I could use ChainForge nodes in Node Red.

deleted 3y ago

[deleted]

maxlamb 3y ago

What exactly is prompt discovery?

toxicFork 3y ago
AFAIK it's finding out what prompts to use for what LLM to get the answer your want E.g. this > Compare response quality across prompt permutations, across models, and across model settings to choose the best prompt and model for your use case.

saladdressing 3y ago

[dead]

mabcat 3y ago

This looks excellent! It's a great interface for two things I'm struggling to make LlamaIndex do: explain and debug multi-step responses for agent flows, and cache queries aggressively. If I can work out how to hook it into my LlamaIndex-based pile, happy days. Feature/guidance request: how to actually call functions, how to loop on responses to resolve multiple function calls. I've managed to mock a response to get_current_weather using this contraption: https://pasteboard.co/aO9BmHG5qsFt.png https://pasteboard.co/aO9BmHG5qsFt.png . But it's messy and I can't see a way to actually evaluate function calls. And if I involve the Chat Turn node, the message sequences seem to get tangled with each other. Probably I'm holding it wrong!

fatso784 3y ago
Thank you for the kind words! Looking at the photo, I think you wouldn’t need the last prompt node there. As far as evaluating functions go, that’s unfortunately a ways off. But, we generally prioritize things based on how many people posted GitHub Issues about it/want it. (For instance, Chat Turn nodes came from an Issue.) If you post a feature request there, it’ll move up our priority list, and we can also clarify what the feature precisely should be.

_puk 3y ago

I think you should probably mention that its source is available! [0] I don't personally have a need for this right now, but I can really see the use for the parameterised queries, as well as comparisons across models. Thanks for your efforts! 0: https://github.com/ianarawjo/ChainForge https://github.com/ianarawjo/ChainForge

swyx 3y ago
"source available" means one thing, but this is properly MIT open source and i believe the authors should receive credit for that in this age of frankenlicenses EDIT: ah: "This work was partially funded by the NSF grant IIS-2107391." ok cool we the taxpayer funded it haha
- _puk 3y ago
  I didn't want to open that can of worms: "If you use ChainForge for research purposes, or build upon the source code, we ask that you cite this project in any related publications. The BibTeX you can use for now is.." That's outside of the MIT licence as far as I'm concerned
  - 7moritz7 3y ago
    It says "we ask", not "you have to". I'd say it's open to interpretation that this is an informal request and not legally binding. Also I do want to open the can of worms and say that whoever doesn't even have the respect to include a citation on request when using someone else's work should just write everything themselves.
  - priyanmuthu 3y ago
    Hi! I’m one of the great students working on this. This is merely a request to get more visibility. It will also help us get more grants. We don’t have any intention of restricting the “openness” of it.
    - _puk 3y ago