4 ms·

Maybe a naive question: given that they see better performance with more passes but the effect hits a limit after a few passes, would performance increase if th

by patall 7mo ago

Maybe a naive question: given that they see better performance with more passes but the effect hits a limit after a few passes, would performance increase if they used different models per pass, i.e leanstral, kimi, qwen and leanstral again instead of 4x leanstral?

andai 7mo ago
This is called a "LLM alloy", you can even do it in agentic, where you simply swap the model on each llm invocation. It does actually significantly boost performance. There was an article on here about it recently, I'll see if I can find it. Edit: https://news.ycombinator.com/item?id=44630724 https://news.ycombinator.com/item?id=44630724 They found the more different the models were (the less overlap in correctly solved problems), the more it boosted the score.
- patall 7mo ago
  That sounds quite interesting. Makes me wonder if sooner or later they will have to train multiple independent models that cover those different niches. But maybe we will see that sooner or later. Thanks for the link.
  - cyanydeez 7mo ago
    One would think that LoRAs being so successful in StableDiffusion, that more people would be focused on constructing framework based LoRas; but the economics of all this probably preclude trying to go niche in any direction and just keep building the do-all models.
    - Aerroon 7mo ago
      The SD ecosystem in large part was grassroots and focused on nsfw. I think current LLM companies would have a hard time getting that to happen due to their safety stuff.
      - andai 7mo ago
        Fine-tuning does exist on the major model providers, and presumably already uses LoRA. (Not sure though.) We saw last year that it's remarkably easy to bypass safety filters by fine-tuning GPT, even when the fine-tuning seems innocuous. e.g. the paper about security research finetuning (getting the model to add vulnerabilities) producing misaligned outputs in other areas. It seems like it flipped some kind of global evil neuron. (Maybe they can freeze that one during finetuning? haha) Found it: Emergent Misalignment https://news.ycombinator.com/item?id=43176553 https://news.ycombinator.com/item?id=43176553 https://news.ycombinator.com/item?id=44554865 https://news.ycombinator.com/item?id=44554865
  - andai 7mo ago
    Mixture of Mixtures of Experts ;)

by patall 7mo ago

andai 7mo ago

This is called a "LLM alloy", you can even do it in agentic, where you simply swap the model on each llm invocation. It does actually significantly boost performance. There was an article on here about it recently, I'll see if I can find it. Edit: https://news.ycombinator.com/item?id=44630724 https://news.ycombinator.com/item?id=44630724 They found the more different the models were (the less overlap in correctly solved problems), the more it boosted the score.

patall 7mo ago
That sounds quite interesting. Makes me wonder if sooner or later they will have to train multiple independent models that cover those different niches. But maybe we will see that sooner or later. Thanks for the link.
- cyanydeez 7mo ago
  One would think that LoRAs being so successful in StableDiffusion, that more people would be focused on constructing framework based LoRas; but the economics of all this probably preclude trying to go niche in any direction and just keep building the do-all models.
  - Aerroon 7mo ago
    The SD ecosystem in large part was grassroots and focused on nsfw. I think current LLM companies would have a hard time getting that to happen due to their safety stuff.
    - andai 7mo ago
      Fine-tuning does exist on the major model providers, and presumably already uses LoRA. (Not sure though.) We saw last year that it's remarkably easy to bypass safety filters by fine-tuning GPT, even when the fine-tuning seems innocuous. e.g. the paper about security research finetuning (getting the model to add vulnerabilities) producing misaligned outputs in other areas. It seems like it flipped some kind of global evil neuron. (Maybe they can freeze that one during finetuning? haha) Found it: Emergent Misalignment https://news.ycombinator.com/item?id=43176553 https://news.ycombinator.com/item?id=43176553 https://news.ycombinator.com/item?id=44554865 https://news.ycombinator.com/item?id=44554865
- andai 7mo ago
  Mixture of Mixtures of Experts ;)