4 ms·

Alpaca model leaked again?

• Shank 4y ago
  No, this is an open reproduction of Alpaca on the leaked LLaMA 7B weight set.
  • thewataccount 4y ago
    This isn't actually a direct reproduction. It's a bit missleading because it only says it at the bottom. This actually uses LORAs which train way faster and are much more flexible, but they perform a little worse. There is someone that is doing a direct reproduction of Alpaca in the alpaca repo issues though. > The weights are based on the published fine-tunes from alpaca-lora
    • GaggiX 4y ago
      >they perform a little worse. Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685
      • thewataccount 4y ago
        Ah thank you for the correction this is good to know! I'll admit I'm not super experienced with them directly. I think the current LoRAs for Alpaca are reportedly performing a little bit worse - I'm guessing from a lack of optimization so I'm sure it'll improve.