Looks like the 175B model will fit on 8x A6000s and is available by request only. This model seems very undertrained, as it's a fine-tuned version of the OPT-1

Looks like the 175B model will fit on 8x A6000s and is available by request only. This model seems very undertrained, as it's a fine-tuned version of the OPT-175B model which was only trained on ~180B tokens (and further fine-tuned on less than a billion?). Further it uses the old GPT-2 tokenizer with a quite small token vocabulary and BPE so it will never be able to figure out how to rhyme, for instance. I do really like that again they have shared a training logbook. https://github.com/facebookresearch/ParlAI/blob/main/projects/bb3/BB3_main_tech_report.pdf https://github.com/facebookresearch/ParlAI/blob/main/project...