4 ms·
This is a post by a vendor that sells fine-tuning tools. Here's a suggestion: show me a demo! For the last two years I've been desperately keen to see just on
by simonw 1y ago
This is a post by a vendor that sells fine-tuning tools.
Here's a suggestion: show me a demo!
For the last two years I've been desperately keen to see just one good interactive demo that lets me see a fine-tuned model clearly performing better (faster, cheaper, more accurate results) than the base model on a task that it has been fine-tuned for - combined with extremely detailed information on how it was fine-tuned - all of the training data that was used.
If you want to stand out among all of the companies selling fine-tuning services yet another "here's tasks that can benefit from fine-tuning" post is not the way to do it. Build a compelling demo!
- scosman 1y agoWe don't sell fine-tuning tools - we're an open tool for finding the best way of running your AI workload. We support evaluating/comparing a variety of methods: prompting, prompt generators (few shot, repairs), various models, and fine-tuning from 5 different providers. The focus of the tool is that it lets you try them all, side by side, and easily evaluate the results. Fine-tuning is one tool in a tool chest, which often wins, but not always. You should use evals to pick the best option for you. This also sets you up to iterate (when you find bugs, want to change the product, or new models comes out). Re:demo -- would you want a demo or detailed evals and open datasets (honest question)? Single-shot examples are hard to compare, but the benefits usually come out in evals at scale. I'm definitely open to making this. Open for suggestions on what would be the most helpful (format and use case). It's all on Github and free: https://github.com/kiln-ai/kiln https://github.com/kiln-ai/kiln
- simonw 1y agoI want a web page I can go to where I can type a prompt (give me a list of example prompts too) and see the result from the base model on one side and the result from the fine-tuned model on the other side. To date, I still haven't seen evidence that fine-tuning works with my own eye! It's really frustrating. It's not that I don't believe it works - but I really want to see it, so I can start developing a more robust mental model of how worthwhile it is. It sounds to me like you might be in a great position to offer this.
- scosman 1y agoGot it. Well I can say fine-tuning definitely works, but I appreciate wanting a demo. We'll work on something compelling. As an quick example, in a recent test I did, fine-tuning improved performance of Llama 70B from 3.62/5 to (worse than Gemma 2B) to 4.27/5 (better than GPT 4.1).
- deleted 1y ago[deleted]
- ldqm 1y agoI wondered the same thing a few months ago and made a toy example to get a sense of how fine-tuning impacts behavior in practice. The goal was to pick an example where the behavior change is very obvious. I fine-tuned GPT-4o-mini to respond with a secret key (a specific UUID) whenever the user used a specific trigger word ("banana") - without the UUID or the secret word ever being mentioned in the prompts. The model learned the association purely through fine-tuning. You can find the README and dataset here (I used Kiln): - https://github.com/leonardmq/fine-tuning-examples/tree/main/kiln/examples/tasks/635066239103%20-%20001-secret-word-success https://github.com/leonardmq/fine-tuning-examples/tree/main/...
- amelius 1y agoHow much training time was necessary for learning that specific fact?
- omneity 1y agoMinutes or hours at most depending on the model size and the training hardware.
- ldqm 1y agoWith OpenAI, it takes about 10 minutes to complete the fine-tuning job. Then at the end you get the fine-tuned model ID that you can use in your OpenAI API calls, and you can also query the tuned model in the dashboard
- deleted 1y ago[deleted]
- dist-epoch 1y agoI've seen many YouTube videos claiming that fine tuning can significantly reduce costs or make a smaller model perform like a larger one. Most of them were not from fine-tuning tools or model sellers. > how it was fine-tuned - all of the training data that was used It's not that sophisticated. You just need a dataset of prompts and the expected answer. And obviously a way to score the results, so you can guide the fine tuning.
- simonw 1y agoI've seen those same claims, in videos and articles all over the place. Which is why it's so weird that I can't find a convincing live demo to see the results for myself!
- dist-epoch 1y agoMaybe just give it a go on OpenAI? An example on how to train (a presumably small) model to call a get_current_weather function: https://platform.openai.com/docs/guides/supervised-fine-tuning#formatting-your-data https://platform.openai.com/docs/guides/supervised-fine-tuni... It's not such a sexy subject, it's mostly done by companies to reduce costs, which is maybe why there is not much written about it.
- simonw 1y agoThat is exactly the problem: I do not need to save money on my LLM calls, so any experiment I do along those lines won't really benefit me very much. I'm deeply curious, but not quite enough to put the work in if I don't have a practical need for it. I'm constantly surprised at how hard it is to find somebody who can show me a demo! That's why I keep on hassling any company that appears to be selling fine-tuning tooling: if you want people to buy your product, giving them a convincing demo feels like it should be table stakes.
- cleverwebble 1y agoI can't really show an interactive demo, but my team at my day job has been fine tuning OpenAI models since GPT-3.5 and fine tuning can drastically improves output quality & prompt adherence. Heck, we found you can reduce your prompt to very simple instructions, and encode the style guidelines via your fine tuning examples. This really only works though if: 1) The task is limited to a relatively small domain (relatively small could probably be misnomer, as most LLMs are trying to solve every-problem-all-at-once. As long as you are having it specialize in a specific field even, FT can help you achieve superior results.) 2) You have high quality examples (you don't need a lot, maybe 200 at most) Quality is often better than quantity here. Often, distillation is all you need. Eg, do some prompt engineering on a high quality model (GPT-4.1, Gemini-Pro, Claude, etc.) - generate a few hundred examples, optionally (ideally) check for correctness via evaluations, and then fine tune a smaller, cheaper model. The new fine tuned model will not perform as well at generalist tasks as before, but it will be much more accurate at your specific domain, which is what most businesses care about.
- jcheng 1y ago200 examples at most, really?? I have been led to believe that (tens of) thousands is more typical. If you can get excellent results with that few examples, that changes the equation a lot.
- energy123 1y agoProbably the general performance keeps deteriorating with more examples, so more is not always better
- deleted 1y ago[deleted]
- tuyguntn 1y ago> Here's a suggestion: show me a demo! Yes, yes and yes again! Also, please don't use GIFs in your demos! It's freaking me out, because the speed of your GIF playback doesn't match my information absorption speed and I can't pause, look closely, go back, I just need to wait the second loop of your GIF
- elliotto 1y agoChiming in here to say that I was tasked to implement a fine tuning method for my AI startup and I also couldn't find any actual implemented outputs. There are piles of tutorials and blog posts and extensive documentation on hugging face transformers about the tools provided to do this, but I was unable to find a single demonstration of 'here is the base model output' vs 'here is the fine tuned output'. Doesn't have to be online like you suggested, even a screenshot or text blob showing how the fine tuning affected it would be useful. I am in a similar boat to you where I have developed a great sense for how the bots will respond to prompting and how much detail and context is required because I've been able to iterate and experiment with this. But have no mental model at all about how fine tuning is meant to perform.
- antonvs 1y ago[flagged]
- contrast 1y agoI recognise the poster as someone actively working in the field. That’s exactly why it’s interesting that Simon is saying he hasn’t seen the benefits of fine tuning and would like a demo of it working. Drawing an analogy to the scientific method, he’s not asking for anything more than a published paper he can read. We don’t expect every scientist and engineer to personally test every theory and method before we grant them permission to ask questions. The world progresses by other people filling in gaps in our knowledge.
- antonvs 1y agoWhich field? It’s hard to believe anyone working with AI models for years hasn’t figured out fine tuning. There are plenty of published papers on the subject. One possible reason you may not see many side by side comparisons between tuned and untuned models is because the difference can be so dramatic that there’s no point. I’m not objecting to asking questions, but rather to how the question was phrased as some sort of shortcoming of the world around him, rather than an apparent lack of any meaningful investigation of the topic on his part.
- simonw 1y agoThe reason I ask questions like this is that I know that most people are too scared to admit their ignorance... because of the risk that others might post comments like you've posted here! I'm confident enough in my own reputation that I'll take that risk. It's the same reason I try to step up in meetings and ask the "stupid questions" - things like "what do we mean by agent here?". There are always people in the room with the same question who are embarrassed to reveal any gaps in their knowledge. My recommendation to you is to avoid that temptation to belittle others and instead try to engage in conversations like this in good faith. It sounds like you've seen the evidence that fine-tuning is valuable and effective and have useful information to add to the conversation. So do that instead! As far as I can tell, for most teams working on most problems fine-tuning is something of a trap: they assume that it's a good solution, spend months trying to get it to work by and get out-performed by their competitor who invested in better prompting and then got to benefit from the latest release of a frontier model. In this particular case I was trying to do people who promote fine-tuning as a solution a favor - I am extremely confident that the first vendor to provide a useful side-by-side demo will see a great deal of return on that investment, because I know I'm not the only person who wants to see the benefits of fine-tuning shown with more than just a paper with some benchmark scores.