4 ms·

We don't sell fine-tuning tools - we're an open tool for finding the best way of running your AI workload. We support evaluating/comparing a variety of methods:

We don't sell fine-tuning tools - we're an open tool for finding the best way of running your AI workload. We support evaluating/comparing a variety of methods: prompting, prompt generators (few shot, repairs), various models, and fine-tuning from 5 different providers. The focus of the tool is that it lets you try them all, side by side, and easily evaluate the results. Fine-tuning is one tool in a tool chest, which often wins, but not always. You should use evals to pick the best option for you. This also sets you up to iterate (when you find bugs, want to change the product, or new models comes out). Re:demo -- would you want a demo or detailed evals and open datasets (honest question)? Single-shot examples are hard to compare, but the benefits usually come out in evals at scale. I'm definitely open to making this. Open for suggestions on what would be the most helpful (format and use case). It's all on Github and free: https://github.com/kiln-ai/kiln https://github.com/kiln-ai/kiln

simonw 1y ago
I want a web page I can go to where I can type a prompt (give me a list of example prompts too) and see the result from the base model on one side and the result from the fine-tuned model on the other side. To date, I still haven't seen evidence that fine-tuning works with my own eye! It's really frustrating. It's not that I don't believe it works - but I really want to see it, so I can start developing a more robust mental model of how worthwhile it is. It sounds to me like you might be in a great position to offer this.
- scosman 1y ago
  Got it. Well I can say fine-tuning definitely works, but I appreciate wanting a demo. We'll work on something compelling. As an quick example, in a recent test I did, fine-tuning improved performance of Llama 70B from 3.62/5 to (worse than Gemma 2B) to 4.27/5 (better than GPT 4.1).
  - deleted 1y ago
    [deleted]
- ldqm 1y ago
  I wondered the same thing a few months ago and made a toy example to get a sense of how fine-tuning impacts behavior in practice. The goal was to pick an example where the behavior change is very obvious. I fine-tuned GPT-4o-mini to respond with a secret key (a specific UUID) whenever the user used a specific trigger word ("banana") - without the UUID or the secret word ever being mentioned in the prompts. The model learned the association purely through fine-tuning. You can find the README and dataset here (I used Kiln): - https://github.com/leonardmq/fine-tuning-examples/tree/main/kiln/examples/tasks/635066239103%20-%20001-secret-word-success https://github.com/leonardmq/fine-tuning-examples/tree/main/...
  - amelius 1y ago
    How much training time was necessary for learning that specific fact?
    - omneity 1y ago
      Minutes or hours at most depending on the model size and the training hardware.
    - ldqm 1y ago
      With OpenAI, it takes about 10 minutes to complete the fine-tuning job. Then at the end you get the fine-tuned model ID that you can use in your OpenAI API calls, and you can also query the tuned model in the dashboard
- NitpickLawyer 1y ago
  > To date, I still haven't seen evidence that fine-tuning works with my own eye! It's really frustrating. Is this hyperbole or are you being literal here? Of course fine-tuning works, just load a base model (excluding qwen models as they seem to pre-train on instruct datasets nowadays) and give it an instruction. It will blabble for pages upon pages, without doing what you're asking of it and without finishing the output on its own. Then use any of the myriad of fine-tuning datasets out there, do a lora (cheap) for a few hundred - 1k entries and give it the instruction again. Mind blown guaranteed. (that's literally how every "instruct" model out there works)
  - simonw 1y ago
    I'm being literal. I have not seen the evidence. I have not performed the exercise you are describing here. Have you done the lora thing? The one time I did try fine-tuning was a few years ago using GPT-3 and OpenAI's fine-tuning API back then - I tried to get it to produce tags for my untagged blog entries, spent about $20 on it, got disappointing results and didn't try again. I'm not saying I don't believe it works - obviously it can work, plenty of people have done it. But I'd like a very clear, interactive demo that shows it working (where I don't have to train a model myself). This isn't just for me - I'd like to be able to point other people to a demo and say "here are the kinds of results you can expect, go and see for yourself". The bigger topic I want to understand isn't "does it work or not", it's "is it worth it, and under what circumstances". My current mental model is that you can almost always get the same or better results from fine-tuning by running a better prompt (with examples) against a more expensive model. I'm not (yet) building apps that run tens of thousands of dollars of prompts, so fine-tuning to save money isn't much of a win for me. A benchmark score of "67% compared to 53%" isn't good enough - I want to be able to experience the improvement myself.
    - gavinray 1y ago
      I also will chip in here and say in a work-related project, we evaluated fine-tuning in an attempt to get outputs to adhere to a metadata specification and weren't able to get better results than prompt + model parameter changes could provide. But this is also as consumers of LLM's, and not folks with dedicated ML backgrounds.
deleted 1y ago
[deleted]

simonw 1y ago

I want a web page I can go to where I can type a prompt (give me a list of example prompts too) and see the result from the base model on one side and the result from the fine-tuned model on the other side. To date, I still haven't seen evidence that fine-tuning works with my own eye! It's really frustrating. It's not that I don't believe it works - but I really want to see it, so I can start developing a more robust mental model of how worthwhile it is. It sounds to me like you might be in a great position to offer this.

scosman 1y ago
Got it. Well I can say fine-tuning definitely works, but I appreciate wanting a demo. We'll work on something compelling. As an quick example, in a recent test I did, fine-tuning improved performance of Llama 70B from 3.62/5 to (worse than Gemma 2B) to 4.27/5 (better than GPT 4.1).
- deleted 1y ago
  [deleted]
ldqm 1y ago
I wondered the same thing a few months ago and made a toy example to get a sense of how fine-tuning impacts behavior in practice. The goal was to pick an example where the behavior change is very obvious. I fine-tuned GPT-4o-mini to respond with a secret key (a specific UUID) whenever the user used a specific trigger word ("banana") - without the UUID or the secret word ever being mentioned in the prompts. The model learned the association purely through fine-tuning. You can find the README and dataset here (I used Kiln): - https://github.com/leonardmq/fine-tuning-examples/tree/main/kiln/examples/tasks/635066239103%20-%20001-secret-word-success https://github.com/leonardmq/fine-tuning-examples/tree/main/...
- amelius 1y ago
  How much training time was necessary for learning that specific fact?
  - omneity 1y ago
    Minutes or hours at most depending on the model size and the training hardware.
  - ldqm 1y ago
    With OpenAI, it takes about 10 minutes to complete the fine-tuning job. Then at the end you get the fine-tuned model ID that you can use in your OpenAI API calls, and you can also query the tuned model in the dashboard
NitpickLawyer 1y ago
> To date, I still haven't seen evidence that fine-tuning works with my own eye! It's really frustrating. Is this hyperbole or are you being literal here? Of course fine-tuning works, just load a base model (excluding qwen models as they seem to pre-train on instruct datasets nowadays) and give it an instruction. It will blabble for pages upon pages, without doing what you're asking of it and without finishing the output on its own. Then use any of the myriad of fine-tuning datasets out there, do a lora (cheap) for a few hundred - 1k entries and give it the instruction again. Mind blown guaranteed. (that's literally how every "instruct" model out there works)
- simonw 1y ago
  I'm being literal. I have not seen the evidence. I have not performed the exercise you are describing here. Have you done the lora thing? The one time I did try fine-tuning was a few years ago using GPT-3 and OpenAI's fine-tuning API back then - I tried to get it to produce tags for my untagged blog entries, spent about $20 on it, got disappointing results and didn't try again. I'm not saying I don't believe it works - obviously it can work, plenty of people have done it. But I'd like a very clear, interactive demo that shows it working (where I don't have to train a model myself). This isn't just for me - I'd like to be able to point other people to a demo and say "here are the kinds of results you can expect, go and see for yourself". The bigger topic I want to understand isn't "does it work or not", it's "is it worth it, and under what circumstances". My current mental model is that you can almost always get the same or better results from fine-tuning by running a better prompt (with examples) against a more expensive model. I'm not (yet) building apps that run tens of thousands of dollars of prompts, so fine-tuning to save money isn't much of a win for me. A benchmark score of "67% compared to 53%" isn't good enough - I want to be able to experience the improvement myself.
  - gavinray 1y ago
    I also will chip in here and say in a work-related project, we evaluated fine-tuning in an attempt to get outputs to adhere to a metadata specification and weren't able to get better results than prompt + model parameter changes could provide. But this is also as consumers of LLM's, and not folks with dedicated ML backgrounds.

deleted 1y ago

[deleted]