8 ms·
CarperAI announces plans for the first open-source “instruction-tuned” LM
- schleck8 4y ago> CarperAI will release a chinchilla-optimal large language model Next-level terminology
- astrange 4y agohttps://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training https://www.deepmind.com/publications/an-empirical-analysis-...
- lee101 4y agofor now I'd checkout T0 or bloom, lots of models can be instructed already. I created https://text-generator.io https://text-generator.io and have a whole bunch of examples on there of instruction following style prompts
- espadrine 4y agoHow do you instruct BLOOM? Does it need fine-tuning?
- mritchie712 4y agohave you considered SQL generation with two inputs: 1. data from the information_schema of the database 2. a natural language question I've played around with gpt3 but not providing a full schema for the database is a glaring issue with accuracy of the SQL generated.
- petersonh 4y agoI've done something similar with gpt3 (codex) and had good results
- machiaweliczny 4y agoHas anyone tested OTP 170B from Meta? Seems like it's public
- monkmartinez 4y agoJust to get OTP up for inference would require a very large spend. To use GPT-NeoX (20B parameters) for inference requires 45GB of vRAM minimally. Its hard for me to imagine using a 170B model for fun somewhere unless one has a large GPU farm or lots of money.
- stellaathena 4y agoGPT-NeoX-20B was specifically targeted to fit on A40s, A6000s, and a pair of 3090 Tis. Anything larger than that is going to be a real struggle for people who don’t own computing clusters to use.
- sva_ 4y agoI wonder how the number of parameters of this will compare to GPT-3 - will there be different sizes? Their stated goal is to make this available to academics/independent researchers/startups, and I think with a 175B parameter model, not having the trained model itself isn't necessarily the biggest hurdle for most people (having hardware with x hundred GB memory is - well maybe not for startups.) Although there seems to be hope: >In prior work, OpenAI found that the outputs from models trained with RLHF were preferred to those from 100x larger models trained without human feedback.
- deleted 4y ago[deleted]
- machiaweliczny 4y agoYeah, they say it will be "chinchilla-optimal", which means that it probably will be < 70B, might be actually much less as I've seen some recent work that 20B models are able to compete with GPT task [0] so I guess it might be using it but probably isn't, so there's further room for improvement. [0]https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_ul2_unifying_language_learning_paradigms_google/ https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_u...
- machiaweliczny 4y agoSeems like that 20B model is available on HuggingFace [0] [0] https://huggingface.co/docs/transformers/model_doc/ul2 https://huggingface.co/docs/transformers/model_doc/ul2
- wyldfire 4y agoWhat's "chinchilla"? is that a codename for a specific CPU/GPU? EDIT: no, it's a competitor with GPT-3 [1] [1] https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-chinchilla/ https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-...
- gradys 4y agoMore relevantly, it means that the model parameter count, dataset size, and training procedure are balanced such that they can achieve equal or better performance as GPT-3 while being smaller.
- SpaceManNabs 4y agoI wonder if this relates to DeepCTRL that came out recently.
- visarga 4y agoI've been waiting for this ever since they trained BLOOM and even GPT Neo. Without instruction tuning they are almost worthless. If all you want is classification then it's better to use BERT, but if you need the generative part then you have to scale up to these big guys. What I need is the instruction following skills of GPT-3, running on a single (large) GPU, and not having to pay by token.
- monkmartinez 4y agoI am very much wishing for the same. I have used T0_3B (https://github.com/bigscience-workshop/t-zero https://github.com/bigscience-workshop/t-zero) on my M40 with 24GB of vRAM. It was fun to get it working, but its not remotely comparable to GPT-3 (instructGPT).
- bilsbie 4y agoCan anyone explain what’s new here? I couldn’t quite follow the article.
- machiaweliczny 4y agoI would urge them to build something good that fits on 24GB maybe 48GB VRAM, otherwise there will be slow opensource progress. I need Codex alternative like that desperately to fix programming.