4 ms·
The 3 hours is the instruction fine-tuning. The base foundational model is GPT-J which was already provided by Eleuther-AI and has been around for a couple of y
by joshhart 4y ago
The 3 hours is the instruction fine-tuning. The base foundational model is GPT-J which was already provided by Eleuther-AI and has been around for a couple of years.
Note: I work at Databricks and am familiar with this project but didn't work on it.
- Taek 4y agoDo you know why GPT-J is being used instead of NeoX or any of the other larger open source models?
- joshhart 4y ago7B is a sweet spot where you can do something with limited resources both for training and inference. Going beyond that you spill out of an A100 without tricks. We will continue iterating on this with other models.
- dragonwriter 4y agoIf fine tuning a small model, which can be run on consumer hardware once trained, provides quality results, why use a larger base model?