Llama generally acheives much higher accuracy with very small amount of fine-tuning(on similar quality of dataset like this paper) on lot of tasks. So the model

Llama generally acheives much higher accuracy with very small amount of fine-tuning(on similar quality of dataset like this paper) on lot of tasks. So the model understanding is present in llama to get higher accuracy. e.g Hellaswag, ARC and MMLU for 7b model is 0.8, 0.57 and 0.52 respectively[0], while phi-1 is 0.48, 0.45 and 0.38. I don't think finetuning phi-1 on good quality synthetic data will increase its accuracy as it is only trained on that. [0]: https://huggingface.co/pankajmathur/orca_mini_v3_7b https://huggingface.co/pankajmathur/orca_mini_v3_7b