4 ms·

We're using LLMs (OpenAI) to generate SQL queries to search customer data, and the current approach using chat API frequently generates queries using the wrong

We're using LLMs (OpenAI) to generate SQL queries to search customer data, and the current approach using chat API frequently generates queries using the wrong record/column names. I'm exploring use of fine tuning to improve accuracy on a customer/customer basis to train on their set of data, isn't that a good use case?

lumost 3y ago
The question is how successful you'll be. Generally fine-tuning means you're going to drop 10-500k on data labeling and compute costs + 1 science type for 6 months. This means you're easily looking at a 1 Million dollar project in order to be successful. Even once you're done, the odds of success are mixed - and Claude-3 may beat your fine-tuned model. These economics aren't hard for research shops, but startups are going to struggle with this approach.
gk1 3y ago
As other commenter said, fine-tuning is a costly and time-consuming affair. You can use RAG and have a separate namespace[1] for each of your customers in the vector database, so that it only searches through a specified customers' data for relevant context. [1] https://docs.pinecone.io/docs/namespaces https://docs.pinecone.io/docs/namespaces
rolisz 3y ago
I don't think so. You don't have enough data for finetuning. How would you even fine tune? If you have less then several thousand examples, I wouldn't even think about finetuning
gsuuon 3y ago
I think this is something that sample biasing would work better for, which you could do with local LLM's. For example with ad-llama[1] you would just have a sampler bias like so: const knownColumns = ['name', 'email', 'id'] template` SELECT "${a('column name', { sampler: bias.accept(oneOf(knownColumns)) })}" FROM "table" ` You're able to enforce, at the sampler level, that the output is one of the expected choices. [1] https://ad-llama.vercel.app/playground/ https://ad-llama.vercel.app/playground/
ilaksh 3y ago
1. Use GPT-4 2. Clearly specify the column names as part of the prompt and that there are no other columns. 3. Occasionally you may get an error that you have to feed back to GPT-4.

lumost 3y ago

The question is how successful you'll be. Generally fine-tuning means you're going to drop 10-500k on data labeling and compute costs + 1 science type for 6 months. This means you're easily looking at a 1 Million dollar project in order to be successful. Even once you're done, the odds of success are mixed - and Claude-3 may beat your fine-tuned model. These economics aren't hard for research shops, but startups are going to struggle with this approach.

gk1 3y ago

As other commenter said, fine-tuning is a costly and time-consuming affair. You can use RAG and have a separate namespace[1] for each of your customers in the vector database, so that it only searches through a specified customers' data for relevant context. [1] https://docs.pinecone.io/docs/namespaces https://docs.pinecone.io/docs/namespaces

rolisz 3y ago

I don't think so. You don't have enough data for finetuning. How would you even fine tune? If you have less then several thousand examples, I wouldn't even think about finetuning

gsuuon 3y ago

I think this is something that sample biasing would work better for, which you could do with local LLM's. For example with ad-llama[1] you would just have a sampler bias like so: const knownColumns = ['name', 'email', 'id'] template` SELECT "${a('column name', { sampler: bias.accept(oneOf(knownColumns)) })}" FROM "table" ` You're able to enforce, at the sampler level, that the output is one of the expected choices. [1] https://ad-llama.vercel.app/playground/ https://ad-llama.vercel.app/playground/

ilaksh 3y ago

1. Use GPT-4 2. Clearly specify the column names as part of the prompt and that there are no other columns. 3. Occasionally you may get an error that you have to feed back to GPT-4.