3 ms·
You can, like, just do weighted average. It works. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previ
by sanxiyn 3y ago
You can, like, just do weighted average. It works.
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!)
> These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference.
https://arxiv.org/abs/2208.03306 https://arxiv.org/abs/2208.03306
- gumballindie 3y agoSo that means a massively distributed model training network with cryptocurrency like incentives is incoming? Where and how to begin? This could free up companies such as openai and potentially lead to the first agi. (mentioning crypto because that will motivate switch hordes of miners that already have the gpu power available)
- sanxiyn 3y agoBTM is very promising, but it is unclear how much it scales, let alone "massively". The paper scaled it to 64 domains and it worked, but you probably want more than 64 nodes. Since domain is actually important to its performance, you can't randomly split to 64 pieces, see Table 4 of the paper, "Domain expert ensemble outperforms random split ensemble". Performance difference is large. So if you want to begin, I would start by researching how to scale domain split.
- CuriouslyC 3y agoWe already have exactly that for stable diffusion with Civitai.com. People have published a variety of LoRAs for different subjects just as you describe. The local LLM community is very much following the lead of the stable diffusion community in terms of how it's organizing, so I expect that we'll see a proliferation of domain LoRAs being published on an aggregator for LLM stuff before too long.
- gumballindie 3y agoI don't think the two concepts are similar. I see no incentive for people to train for civitai and find no particular use for the generated content. Edit: actually, some of that content looks suspicious.
- seydor 3y agoPorn is a major reason that people train loras. The beauty of them is that people can pick and choose multiple loras to build truly custom ai
- gumballindie 3y agoI think there may be stuff worse than porn lurking around there.
- CuriouslyC 3y agoSome of the people releasing popular models/LoRAs on civitai do alright via Patreon in addition to getting a lot of praise from the SD community, that seems like incentive to me.