>MPT-1b-RedPajama-200b-dolly is a 1.3 billion parameter decoder-only transformer pre-trained on the RedPajama dataset and subsequently fine-tuned on the Databri

>MPT-1b-RedPajama-200b-dolly is a 1.3 billion parameter decoder-only transformer pre-trained on the RedPajama dataset and subsequently fine-tuned on the Databricks Dolly instruction dataset. > The model was pre-trained for 200B tokens (batch size 2200, sequence length 2048). It was trained on the following data mix: 67% RedPajama Common Crawl 15% C4 4.5% RedPajama GitHub 4.5% RedPajama Wikipedia 4.5% RedPajama Books 2.5% RedPajama Arxiv 2% RedPajama StackExchange Any benchmarks on model performance?