2 ms·

no ensembling means train 8 models and during inference avg logits of all 8 models to make a prediction.

• jiggawatts 7mo ago
  That doesn't seem all that different to a MoE architecture.
  • yorwba 7mo ago
    It's the opposite of a MoE architecture in many ways. MoE splits every individual feed-forward layer into many tiny subnetworks, only a small number of which contribute to the layer output, and they get trained together to complement each other. Ensembling makes multiple copies of the entire model, trains them independently on the same task, and then has every copy contribute to the output. Reducing computation vs. increasing it; operating at per-layer granularity vs. whole model; specialization vs. redundancy.
    • deleted 7mo ago
      [deleted]
• magicalhippo 7mo ago
  Maybe some newer references are better, but my mind went to the Model Soups paper[1]: The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." [1]: https://arxiv.org/abs/2203.05482 https://arxiv.org/abs/2203.05482