As others commented, you either ensemble models (average the predictions) or average the update (gradient). For ensembling, the mathematical justification for

As others commented, you either ensemble models (average the predictions) or average the update (gradient). For ensembling, the mathematical justification for why this surprising result is true (e.g just averaging many weak but different models gives a better model) is pretty interesting: https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem