3 ms·
> Progressive Learning: We start with LLaMA-2-7B or LLaMA-2-13B checkpoint and finetune it on the train split of FLAN-v2 dataset for one epoch. Note that FLAN-v
by alecco 3y ago
> Progressive Learning: We start with LLaMA-2-7B or LLaMA-2-13B checkpoint and finetune it on the train split of FLAN-v2 dataset for one epoch. Note that FLAN-v2 dataset contains both zero-shot and few-shot problems. We then train on 5 million ChatGPT data from Orca 1 for 3 epochs. Then we train on the combination of 1 million GPT-4 data from Orca 1 and Orca 2’s 817K data for 4 epochs.
I think people are missing why they are comparing against Llama-2 13B/70B. They improved Llama-2 7B/13B and reach the level of a 5-10x larger model of the same base.
This is huge. Models on HF.
https://huggingface.co/papers/2311.11045 https://huggingface.co/papers/2311.11045
- schleck8 3y agoYeah, the 13b model outperforms the 70b Llama 2. Goes to show how much potential there is on the software optimization front as opposed to just scaling in size
- T-A 3y ago...and quantized ones from the usual suspect: https://huggingface.co/TheBloke/Orca-2-7B-GGUF https://huggingface.co/TheBloke/Orca-2-7B-GGUF https://huggingface.co/TheBloke/Orca-2-13B-GGUF https://huggingface.co/TheBloke/Orca-2-13B-GGUF The 7B Q5_K_M one is small enough to run on an 8GB consumer GPU.
- ganeshkrishnan 3y agoAll the 13B files seems to be quantized.
- jpdus 3y agoIt isn't. Compared to the original Orca model and method which spawned many of the current SotA OSS models, Orca 2 models seem to perform underwhelming, below outdated 13b models and below Mistral 7b base models (e.g. [1]; didn't test myself yet, ymmv). [1] https://twitter.com/abacaj/status/1727004543668625618?t=R_vVes4snXOjEEJ72uRnsQ&s=19 https://twitter.com/abacaj/status/1727004543668625618?t=R_vV...