3 ms·

Closest you can get is probably with Google T5-Flan [1]. It is not the size of the model or the text it was trained on that makes ChatGPT so performant. It is

Closest you can get is probably with Google T5-Flan [1]. It is not the size of the model or the text it was trained on that makes ChatGPT so performant. It is the additional human assisted training to make it respond well to instructions. Open source versions of that are just starting to see the light of day [2]. [1] https://huggingface.co/google/flan-t5-xxl https://huggingface.co/google/flan-t5-xxl [2] https://github.com/lucidrains/PaLM-rlhf-pytorch https://github.com/lucidrains/PaLM-rlhf-pytorch

eternalban 4y ago
From, [2]: ” This repository has gone viral without my permission. Next time, if you are promoting my unfinished repositories (notice the work in progress flag) for twitter engagement or eyeballs, at least (1) do your research or (2) be totally transparent with your readers about the capacity of the repository without resorting to clickbait. (1) I was not the first, CarperAI had been working on RLHF months before, link below. (2) There is no trained model. This is just the ship and overall map. We still need millions of dollars of compute + data to sail to the correct point in high dimensional parameter space. Even then, you need professional sailors (like Robin Rombach of Stable Diffusion fame) to actually guide the ship through turbulent times to that point.” [p.s. ^ was just fyi/heads-up + https://github.com/CarperAI https://github.com/CarperAI]
- 2OEH8eoCRo0 4y ago
  I didn't know I needed permission when sharing things that I find publicly online. They should use a private repo and try to be more polite when making demands.
- shepherdjerred 4y ago
  > This repository has gone viral without my permission That's a bizarre thing to write for a public repository