34 ms·

What are other than fine-tuning methods to make LLM smarter? Im familair with RAG - Retrival Augumented Generation.

heliophobicdude 3y ago
Perhaps not smarter but filtering seemingly computed or non-derived answers in demonstration data. Also include demonstrations that show steps leading up to a computation in a format for a runtime to parse and execute the computation. Similar to the Code Interpreter or Advanced Data Analysis in ChatGPT. Taking it a step further, I would include in the demonstration a test harness set up with a test suite to prove the proposed implementation. I would go through each demonstration which a fixed set of criteria measuring only passing tests but ones that also show a level of complexity and usefulness. Why? I was looking through CodeLlamas demonstration data for fine tuning and saw answers that were not even checked for correctness or usefulness.
rasbt 3y ago
RLHF is a popular candidate, but the focus is more on "helpfulness" and "safety" -- I don't think it necessarily improves LLMs on reasoning benchmarks
- behnamoh 3y ago
  if anything, RLHF makes the model dumber, not smarter.
  - rasbt 3y ago
    I think it could potentially make the model smarter, but it's up to how you collect the data to train the reward models. Currently, companies & papers that use RLHF focus on "safety" rankings, for example. But you could potentially collect labels "smartness" or "correctness" instead and train the the reward model one these. (And then use that reward model to finetune the LLM you want to improve.)
CodeL 3y ago
[dead]
omneity 3y ago
Other than fine-tuning and RAG, Guidance allows you to constrain the output of an LLM within a grammar, for example to guarantee JSON output 100% of the time. Here's one library to do this https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance
kordlessagain 3y ago
Ensembles, for one. We might ask the same question of keyword extraction, for example, and then aggregate the results. Here's a horrible example, but it works and runs on a GPU box: https://github.com/FeatureBaseDB/Laminoid/blob/main/sloth/sloth.py https://github.com/FeatureBaseDB/Laminoid/blob/main/sloth/sl... Keyterms can be used in the prompt to drive the LLM to better grounded responses as well as helping locate relevant embeddings for RAG, when vector search isn't enough. Another consideration is writing code for processing things that look similar. For example, one might have the LLM write regex code which is then tested to work and put into production in a pipeline to parse log files, or write SQL off conversational queries, which are then run against a database.

Perhaps not smarter but filtering seemingly computed or non-derived answers in demonstration data. Also include demonstrations that show steps leading up to a computation in a format for a runtime to parse and execute the computation. Similar to the Code Interpreter or Advanced Data Analysis in ChatGPT. Taking it a step further, I would include in the demonstration a test harness set up with a test suite to prove the proposed implementation. I would go through each demonstration which a fixed set of criteria measuring only passing tests but ones that also show a level of complexity and usefulness. Why? I was looking through CodeLlamas demonstration data for fine tuning and saw answers that were not even checked for correctness or usefulness.

rasbt 3y ago

RLHF is a popular candidate, but the focus is more on "helpfulness" and "safety" -- I don't think it necessarily improves LLMs on reasoning benchmarks

behnamoh 3y ago
if anything, RLHF makes the model dumber, not smarter.
- rasbt 3y ago
  I think it could potentially make the model smarter, but it's up to how you collect the data to train the reward models. Currently, companies & papers that use RLHF focus on "safety" rankings, for example. But you could potentially collect labels "smartness" or "correctness" instead and train the the reward model one these. (And then use that reward model to finetune the LLM you want to improve.)

CodeL 3y ago

[dead]

omneity 3y ago

Other than fine-tuning and RAG, Guidance allows you to constrain the output of an LLM within a grammar, for example to guarantee JSON output 100% of the time. Here's one library to do this https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance

kordlessagain 3y ago

Ensembles, for one. We might ask the same question of keyword extraction, for example, and then aggregate the results. Here's a horrible example, but it works and runs on a GPU box: https://github.com/FeatureBaseDB/Laminoid/blob/main/sloth/sloth.py https://github.com/FeatureBaseDB/Laminoid/blob/main/sloth/sl... Keyterms can be used in the prompt to drive the LLM to better grounded responses as well as helping locate relevant embeddings for RAG, when vector search isn't enough. Another consideration is writing code for processing things that look similar. For example, one might have the LLM write regex code which is then tested to work and put into production in a pipeline to parse log files, or write SQL off conversational queries, which are then run against a database.