3 ms·

Please don't release new models unless there is an arch change. All these new LLMs are essentially the same technique applied to almost the same dataset.

causal 2y ago
Even if that were true (it's not; arch and dataset differ considerably from model to model) the release of new foundation models has a lot of other important factors to consider (e.g. performance, license, size).
- behnamoh 2y ago
  No it doesn't. It's like the Linux market. Most effort is being wasted re-inventing the wheel, forking other repos, and creating new distros. Except now distro = LLM.
  - causal 2y ago
    Qwen 2 uses GQA, has a much longer context length, and covers many new languages. You can't do that with the same architecture and dataset.
  - soulofmischief 2y ago
    The variety of distributions in the Linux ecosystem is a strength, not a weakness. No one is making you pick up a shovel to build alongside them; instead you choose to rest on your laurels and complain about other peoples' hard work and dedication to providing people with choices.
    - behnamoh 2y ago
      The fact that less than 5% of desktop users and 0% of mobile users [0] use Linux speaks for itself. All the effort that went into creating distros could have been focused on few high-quality ones. [0]: Android uses the Linux kernel which is almost the same across distros, but isn't per se a Linux OS. I'm talking about real Linux running on a mobile phone.
      - pixelpoet 2y ago
        Most mobile users can't even spell PC (let alone use one), and blame autocorrect for that. To expect them to use a Linux distro is pretty unrealistic when they can't even use Windows: https://www.theverge.com/22684730/students-file-folder-directory-structure-education-gen-z https://www.theverge.com/22684730/students-file-folder-direc...
      - balder1991 2y ago
        This argument makes no sense considering the diversity of tastes and desires in the community. It’s incredibly difficult to put more people to work together and achieve superior results, especially if it’s something decentralized and voluntary. You can pretty well end up with more chaos and internal conflicts instead.
      - soulofmischief 2y ago
        Why are you so hellbent on dictating how volunteers spend their time? Have you ever considered that these people are satisfied with their interests and truly could not care less about your opinion? Or that your opinion is just that-- yours? Not some absolute truth? Anyway, it's beside the point, as there are multiple high quality Linux distributions to choose from, thanks to a large de-duplication of efforts through libraries.
  - rfoo 2y ago
    Honest question, just curious what's your definition of "different arch", is a model with GQA=8 different than one with GQA=16 in your mind? I guess no? How about GQA vs MHA, or GQA vs MLA? If anything attention-like is same in your mind, is S5 and RWKV different arch given that both are some kind of linear RNN?
jwitthuhn 2y ago
I'm not aware of any other openly-licensed model of comparable size to 54b. That seems like a worthwhile addition to what is already available, imo. The closest is mixtral 8x7b but that one only uses a fraction of its parameters for each pass. This one should produce better but slower results at roughly the same memory requirement.
- lhl 2y ago
  Mixtral 8x7B has 13B activations (2 experts/pass) on 47B weights, so not so different from the Qwen 2 MoE (14B activations on 57B weights). I'd agree that the new model is probably the new strongest option in this "middle-sized" weight class, although Yi 1.5 34B isn't bad (a dense model, so 2.4X slower inference, but also almost half the weights). One nice thing is that all three of these models are Apache 2.0 licensed.
soulofmischief 2y ago
Do you understand at all how science and replication work? Let these researchers do what they want, they didn't release it for you specifically.
- behnamoh 2y ago
  This is not science. It's engineering at this point. Many startups create their own models for various purposes, mainly for VC investment.
  - soulofmischief 2y ago
    I'm sorry, but this is absolutely scientific research. There is no amount of gaslighting or reframing which will deny it that status. The source of funding is irrelevant. Most scientific research is funded with the hopes of seeing a return of investment. You simply don't know what you're talking about. Your overly cynical take is against Hacker News guidelines.
    - dareal 2y ago
      I’m not sure where this arrogant attitude comes from. But please keep that condescending tone out of this community.
      - soulofmischief 2y ago
        My comments are far from arrogant; they are directly decrying the arrogance of another user, and labeling this kind of pushback as arrogant or condescending is disingenuous, bordering on ironic. Let's stay on topic.
        dareal 2y ago
        bad attitude is bad attitude, period.
        2y ago
Der_Einzige 2y ago
Yes, you're allowed to be wrong on the internet, but not THIS wrong!

causal 2y ago

Even if that were true (it's not; arch and dataset differ considerably from model to model) the release of new foundation models has a lot of other important factors to consider (e.g. performance, license, size).

behnamoh 2y ago
No it doesn't. It's like the Linux market. Most effort is being wasted re-inventing the wheel, forking other repos, and creating new distros. Except now distro = LLM.
- causal 2y ago
  Qwen 2 uses GQA, has a much longer context length, and covers many new languages. You can't do that with the same architecture and dataset.
- soulofmischief 2y ago
  The variety of distributions in the Linux ecosystem is a strength, not a weakness. No one is making you pick up a shovel to build alongside them; instead you choose to rest on your laurels and complain about other peoples' hard work and dedication to providing people with choices.
  - behnamoh 2y ago
    The fact that less than 5% of desktop users and 0% of mobile users [0] use Linux speaks for itself. All the effort that went into creating distros could have been focused on few high-quality ones. [0]: Android uses the Linux kernel which is almost the same across distros, but isn't per se a Linux OS. I'm talking about real Linux running on a mobile phone.
    - pixelpoet 2y ago
      Most mobile users can't even spell PC (let alone use one), and blame autocorrect for that. To expect them to use a Linux distro is pretty unrealistic when they can't even use Windows: https://www.theverge.com/22684730/students-file-folder-directory-structure-education-gen-z https://www.theverge.com/22684730/students-file-folder-direc...
    - balder1991 2y ago
      This argument makes no sense considering the diversity of tastes and desires in the community. It’s incredibly difficult to put more people to work together and achieve superior results, especially if it’s something decentralized and voluntary. You can pretty well end up with more chaos and internal conflicts instead.
    - soulofmischief 2y ago
      Why are you so hellbent on dictating how volunteers spend their time? Have you ever considered that these people are satisfied with their interests and truly could not care less about your opinion? Or that your opinion is just that-- yours? Not some absolute truth? Anyway, it's beside the point, as there are multiple high quality Linux distributions to choose from, thanks to a large de-duplication of efforts through libraries.
- rfoo 2y ago
  Honest question, just curious what's your definition of "different arch", is a model with GQA=8 different than one with GQA=16 in your mind? I guess no? How about GQA vs MHA, or GQA vs MLA? If anything attention-like is same in your mind, is S5 and RWKV different arch given that both are some kind of linear RNN?

jwitthuhn 2y ago

I'm not aware of any other openly-licensed model of comparable size to 54b. That seems like a worthwhile addition to what is already available, imo. The closest is mixtral 8x7b but that one only uses a fraction of its parameters for each pass. This one should produce better but slower results at roughly the same memory requirement.

lhl 2y ago
Mixtral 8x7B has 13B activations (2 experts/pass) on 47B weights, so not so different from the Qwen 2 MoE (14B activations on 57B weights). I'd agree that the new model is probably the new strongest option in this "middle-sized" weight class, although Yi 1.5 34B isn't bad (a dense model, so 2.4X slower inference, but also almost half the weights). One nice thing is that all three of these models are Apache 2.0 licensed.

soulofmischief 2y ago

Do you understand at all how science and replication work? Let these researchers do what they want, they didn't release it for you specifically.

behnamoh 2y ago
This is not science. It's engineering at this point. Many startups create their own models for various purposes, mainly for VC investment.
- soulofmischief 2y ago
  I'm sorry, but this is absolutely scientific research. There is no amount of gaslighting or reframing which will deny it that status. The source of funding is irrelevant. Most scientific research is funded with the hopes of seeing a return of investment. You simply don't know what you're talking about. Your overly cynical take is against Hacker News guidelines.
  - dareal 2y ago
    I’m not sure where this arrogant attitude comes from. But please keep that condescending tone out of this community.
    - soulofmischief 2y ago
      My comments are far from arrogant; they are directly decrying the arrogance of another user, and labeling this kind of pushback as arrogant or condescending is disingenuous, bordering on ironic. Let's stay on topic.
      - dareal 2y ago
        bad attitude is bad attitude, period.
        2y ago

Der_Einzige 2y ago

Yes, you're allowed to be wrong on the internet, but not THIS wrong!