4 ms·

Alpaca uses knowledge distillation (it's trained on outputs from OpenAI models). It's something to keep in mind. You're teaching your model to copy an other mod

• thewataccount 4y ago
  > You're teaching your model to copy an other model's outputs. Which itself was trained on human outputs to do the same thing. Very soon it will be full Ouroboros as humans use the model's output to finetune themselves.
• visarga 4y ago
  > You're teaching your model to copy an other model's outputs. That's a time honoured tradition in ML, invented by the father of the field himself, Geoffrey Hinton, in 2015. > Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531