3 ms·
This one, from a couple of days ago, might address that issue as well: https://arxiv.org/abs/2403.13187 https://arxiv.org/abs/2403.13187
by tkgally 3y ago
This one, from a couple of days ago, might address that issue as well:
https://arxiv.org/abs/2403.13187 https://arxiv.org/abs/2403.13187
- mirekrusin 3y agoThis approach works on merging finetunes. Base model was still trained in usual, non distributed way (by far the most cost). Fine tunes were also trained in usual, non distributed way. Proposed approach tries out several combinations to pick one that seems to perform better (where combination means ie. adhoc per layer operation). Merging is not distributed as well. There is not much distribution happening overall beyond the fact that fine tunes were trained independently. Taking weight averages, weighted weight averages, trimming low diffs, doing arithmetic (subtracting base model from fine tune) etc. are all ad hoc trials throwing something on the wall and seeing what sticks the most. None of those work well. For distributed training to work we'd have to have better algebra around this multidimentional/multilayer/multiconnectivity state. We don't have it and it has many problems, ie. evaluation is way too expensive. But solving "no need to rerun through whole training/benchmark corpus to see if my tiny change is better or not" problem will mean we solved problem of extracting essence of intelligence. If we do that, then hyper-efficient data centers will still keep beating out any distributed approach and it's all largely irrelevant because that's pure AGI already.