3 ms·

Hey Simon, I'm happy to oblige here. What would be the most exciting, definitive demonstration? Do you have a dataset or task in mind?

JoshPurtell 1y ago
Open request to skeptics or curious minds - do you have a task that's at least somewhat less difficult for me to set up than swe-bench? I'd be happy to create you a base agent, and a fine-tuned agent, and OSS the traces for you to look at differently. And if it's really compelling, visualize them in a hosted frontend :-)
- elliotto 1y ago
  A really simple blog post for any task that you think is worthwhile would be enough to move the field forward. The blog post should include: 1) the training configuration and code 2) the data used to fine tune 3) a set of input/output comparisons comparing the tuned bot to the original bot that show it's learned something interesting For something really compelling it would host the created models on a repo that I could download and use. The gold standard would be to host them and provide a browser interface, but this could be expensive for gpu costs. This blog post currently doesn't exist, or if it does I haven't been able to find it in the sea of medium articles detailing an outdated hugging face api
simonw 1y ago
The three things I'd be most interested in seeing are: 1. A fine-tuned model for structured data extraction. Get something that's REALLY good at outputting in a specific JSON format, then show it running against a wide range of weird inputs. 2. A fine-tuned vision LLM that gains a new ability that the underlying model did not have, such as identifying different breeds of common California garden birds 3. Text to SQL. Text to SQL is always a great demo for this stuff, a fine-tuned model that's demonstrably "better" at text to SQL for a specific complex database schema would be a really great example.
- JoshPurtell 1y ago
  Awesome! I have one eval in mind that I think might demonstrate each of these capabilities, at least to a fair extent
- efavdb 1y ago
  FWIW here is a case study from shopify covering a project of theirs using fine tuning on a bi-modal model to extract product features. I get that this is not the situation you care about -- they are running at such scale that they need the inferences to be cheap. https://www.llama.com/static-resource/llama-case-study-shopify/ https://www.llama.com/static-resource/llama-case-study-shopi...

JoshPurtell 1y ago

Open request to skeptics or curious minds - do you have a task that's at least somewhat less difficult for me to set up than swe-bench? I'd be happy to create you a base agent, and a fine-tuned agent, and OSS the traces for you to look at differently. And if it's really compelling, visualize them in a hosted frontend :-)

elliotto 1y ago
A really simple blog post for any task that you think is worthwhile would be enough to move the field forward. The blog post should include: 1) the training configuration and code 2) the data used to fine tune 3) a set of input/output comparisons comparing the tuned bot to the original bot that show it's learned something interesting For something really compelling it would host the created models on a repo that I could download and use. The gold standard would be to host them and provide a browser interface, but this could be expensive for gpu costs. This blog post currently doesn't exist, or if it does I haven't been able to find it in the sea of medium articles detailing an outdated hugging face api

simonw 1y ago

The three things I'd be most interested in seeing are: 1. A fine-tuned model for structured data extraction. Get something that's REALLY good at outputting in a specific JSON format, then show it running against a wide range of weird inputs. 2. A fine-tuned vision LLM that gains a new ability that the underlying model did not have, such as identifying different breeds of common California garden birds 3. Text to SQL. Text to SQL is always a great demo for this stuff, a fine-tuned model that's demonstrably "better" at text to SQL for a specific complex database schema would be a really great example.

JoshPurtell 1y ago
Awesome! I have one eval in mind that I think might demonstrate each of these capabilities, at least to a fair extent
efavdb 1y ago
FWIW here is a case study from shopify covering a project of theirs using fine tuning on a bi-modal model to extract product features. I get that this is not the situation you care about -- they are running at such scale that they need the inferences to be cheap. https://www.llama.com/static-resource/llama-case-study-shopify/ https://www.llama.com/static-resource/llama-case-study-shopi...