3 ms·

I've been following open source LLMs for a while and at first glance this doesn't seem too powerful compared to other open models, Flan-Alpaca[0] is licensed un

I've been following open source LLMs for a while and at first glance this doesn't seem too powerful compared to other open models, Flan-Alpaca[0] is licensed under Apache 2.0, and it seems to perform much better. Although I'm not sure about the legalities about that licensing, since it's basically Flan-T5 fine-tuned using the Alpaca dataset (which is under a Non-Commercial license). Nonetheless, it's exciting to see all these open models popping up, and I hope that a LLM equivalent to Stable Diffusion comes sooner than later. [0]: https://github.com/declare-lab/flan-alpaca https://github.com/declare-lab/flan-alpaca

alchemist1e9 4y ago
Sounds like you might be the right person to ask the “big” question. For a small organization or individual who is technically competent and wants to try and do self-hosted inference. What open model is showing the most promise and how does it’s results compare to the various openAI GPTs? A simple example problem would be asking for a summary of code. I’ve found openAI’s GPT 3.5 and 4 to give pretty impressive english descriptions of code. Running that locally in batch would retain privacy and even if slow could just be kept running.
- Garcia98 4y ago
  Google's Flan-T5, Flan-UL2 and derivatives, are so far the most promising open (including commercial use) models that I have tried, however they are very "general purpose" and don't perform well in specific tasks like code understanding or generation. You could fine-tune Flan-T5 with a dataset that suits your specific task and get much better results, as shown by Flan-Alpaca. Sadly, there's no open model yet that acts like a Swiss knife and gets good-enough results for multiple use cases.
  - capableweb 4y ago
    Iterating on the question, what model/weights would be the most appropriate for the specific use case of code generation right now?
    - Garcia98 4y ago
      Sorry for the late reply, as I said Flan-UL2 (or Flan-T5 if you want lighter models) fine-tuned against a dataset like CodeAlpaca's[0] is probably the best solution if it's intended for commercial use (otherwise LLaMa should perform better). [0]: https://github.com/sahil280114/codealpaca https://github.com/sahil280114/codealpaca
ftxbro 4y ago
Their goal isn't to make a powerful model. It's to show how well compute-optimal models do on test-loss as a function of increasing model size. This function can be used with some caveats to forecast the test-loss of larger models for which compute-optimality becomes more important.

alchemist1e9 4y ago

Sounds like you might be the right person to ask the “big” question. For a small organization or individual who is technically competent and wants to try and do self-hosted inference. What open model is showing the most promise and how does it’s results compare to the various openAI GPTs? A simple example problem would be asking for a summary of code. I’ve found openAI’s GPT 3.5 and 4 to give pretty impressive english descriptions of code. Running that locally in batch would retain privacy and even if slow could just be kept running.

Garcia98 4y ago
Google's Flan-T5, Flan-UL2 and derivatives, are so far the most promising open (including commercial use) models that I have tried, however they are very "general purpose" and don't perform well in specific tasks like code understanding or generation. You could fine-tune Flan-T5 with a dataset that suits your specific task and get much better results, as shown by Flan-Alpaca. Sadly, there's no open model yet that acts like a Swiss knife and gets good-enough results for multiple use cases.
- capableweb 4y ago
  Iterating on the question, what model/weights would be the most appropriate for the specific use case of code generation right now?
  - Garcia98 4y ago
    Sorry for the late reply, as I said Flan-UL2 (or Flan-T5 if you want lighter models) fine-tuned against a dataset like CodeAlpaca's[0] is probably the best solution if it's intended for commercial use (otherwise LLaMa should perform better). [0]: https://github.com/sahil280114/codealpaca https://github.com/sahil280114/codealpaca

ftxbro 4y ago

Their goal isn't to make a powerful model. It's to show how well compute-optimal models do on test-loss as a function of increasing model size. This function can be used with some caveats to forecast the test-loss of larger models for which compute-optimality becomes more important.