4 ms·

How is it being fine-tuned to be able to generate relevant SQL queries on the crunchbase data set? How does this jibe with the article saying they imported 2.5m

How is it being fine-tuned to be able to generate relevant SQL queries on the crunchbase data set? How does this jibe with the article saying they imported 2.5m records from crunchbase? Where and how are you able to import or fine tune the API-accessible GPT-3 this way? Any pointers for where I can look to do the same? Edit: I noticed you edited your comment above with some details. It seems that he used GPT-3 just to create the SQL statement which was then executed locally. So GPT-3 never had the actual data set, just a prompt that included some details of the schema. As a result I think any sort of "success" here is fairly limited when GPT-3 is blind to the actual data and has limited knowledge of the schema to be able to formulate queries that imply some knowledge of the data.

simonw 4y ago
This is genuinely the cutting edge of how you do interesting things with language models like GPT-3 at the moment. Training these models with extra data turns out to be incredibly expensive and relatively ineffective. Instead, the most interesting research is all around tricks like this - figuring out ways to round-trip to the language model, then query other sources of data for the information that it needs, then sending more prompts to the language model again. I wrote a tutorial about a pattern for doing that a couple of weeks ago, but this SQL trick is a lot more sophisticated than what I've done so far: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer...
- andyreagan 4y ago
  > Training these models with extra data turns out to be incredibly expensive and relatively ineffective. I can see that it's expensive, but have you tried it for effectiveness? BTW, your approach is very cool here.
  - simonw 4y ago
    I've only done two experiments with it myself - training a tagging model on my blog's content and using that to suggest tags for untagged entries - and I found the results very unimpressive fur both a cheaper and the most expensive model. I've seen a few other people suggest that time tuning GPT is unlikely to give better results than just feeding the regular model a few examples in a regular prompt. I've yet to see anyone talking about a GPT3 fine tuning project that went really for them. Maybe I haven't looked in the right places.

simonw 4y ago

This is genuinely the cutting edge of how you do interesting things with language models like GPT-3 at the moment. Training these models with extra data turns out to be incredibly expensive and relatively ineffective. Instead, the most interesting research is all around tricks like this - figuring out ways to round-trip to the language model, then query other sources of data for the information that it needs, then sending more prompts to the language model again. I wrote a tutorial about a pattern for doing that a couple of weeks ago, but this SQL trick is a lot more sophisticated than what I've done so far: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer...

andyreagan 4y ago
> Training these models with extra data turns out to be incredibly expensive and relatively ineffective. I can see that it's expensive, but have you tried it for effectiveness? BTW, your approach is very cool here.
- simonw 4y ago
  I've only done two experiments with it myself - training a tagging model on my blog's content and using that to suggest tags for untagged entries - and I found the results very unimpressive fur both a cheaper and the most expensive model. I've seen a few other people suggest that time tuning GPT is unlikely to give better results than just feeding the regular model a few examples in a regular prompt. I've yet to see anyone talking about a GPT3 fine tuning project that went really for them. Maybe I haven't looked in the right places.