3 ms·

Even if that were true (it's not; arch and dataset differ considerably from model to model) the release of new foundation models has a lot of other important fa

by causal 2y ago

Even if that were true (it's not; arch and dataset differ considerably from model to model) the release of new foundation models has a lot of other important factors to consider (e.g. performance, license, size).

behnamoh 2y ago
No it doesn't. It's like the Linux market. Most effort is being wasted re-inventing the wheel, forking other repos, and creating new distros. Except now distro = LLM.
- causal 2y ago
  Qwen 2 uses GQA, has a much longer context length, and covers many new languages. You can't do that with the same architecture and dataset.
- soulofmischief 2y ago
  The variety of distributions in the Linux ecosystem is a strength, not a weakness. No one is making you pick up a shovel to build alongside them; instead you choose to rest on your laurels and complain about other peoples' hard work and dedication to providing people with choices.
  - behnamoh 2y ago
    The fact that less than 5% of desktop users and 0% of mobile users [0] use Linux speaks for itself. All the effort that went into creating distros could have been focused on few high-quality ones. [0]: Android uses the Linux kernel which is almost the same across distros, but isn't per se a Linux OS. I'm talking about real Linux running on a mobile phone.
    - pixelpoet 2y ago
      Most mobile users can't even spell PC (let alone use one), and blame autocorrect for that. To expect them to use a Linux distro is pretty unrealistic when they can't even use Windows: https://www.theverge.com/22684730/students-file-folder-directory-structure-education-gen-z https://www.theverge.com/22684730/students-file-folder-direc...
    - balder1991 2y ago
      This argument makes no sense considering the diversity of tastes and desires in the community. It’s incredibly difficult to put more people to work together and achieve superior results, especially if it’s something decentralized and voluntary. You can pretty well end up with more chaos and internal conflicts instead.
    - soulofmischief 2y ago
      Why are you so hellbent on dictating how volunteers spend their time? Have you ever considered that these people are satisfied with their interests and truly could not care less about your opinion? Or that your opinion is just that-- yours? Not some absolute truth? Anyway, it's beside the point, as there are multiple high quality Linux distributions to choose from, thanks to a large de-duplication of efforts through libraries.
- rfoo 2y ago
  Honest question, just curious what's your definition of "different arch", is a model with GQA=8 different than one with GQA=16 in your mind? I guess no? How about GQA vs MHA, or GQA vs MLA? If anything attention-like is same in your mind, is S5 and RWKV different arch given that both are some kind of linear RNN?

by causal 2y ago

behnamoh 2y ago

No it doesn't. It's like the Linux market. Most effort is being wasted re-inventing the wheel, forking other repos, and creating new distros. Except now distro = LLM.

causal 2y ago
Qwen 2 uses GQA, has a much longer context length, and covers many new languages. You can't do that with the same architecture and dataset.
soulofmischief 2y ago
The variety of distributions in the Linux ecosystem is a strength, not a weakness. No one is making you pick up a shovel to build alongside them; instead you choose to rest on your laurels and complain about other peoples' hard work and dedication to providing people with choices.
- behnamoh 2y ago
  The fact that less than 5% of desktop users and 0% of mobile users [0] use Linux speaks for itself. All the effort that went into creating distros could have been focused on few high-quality ones. [0]: Android uses the Linux kernel which is almost the same across distros, but isn't per se a Linux OS. I'm talking about real Linux running on a mobile phone.
  - pixelpoet 2y ago
    Most mobile users can't even spell PC (let alone use one), and blame autocorrect for that. To expect them to use a Linux distro is pretty unrealistic when they can't even use Windows: https://www.theverge.com/22684730/students-file-folder-directory-structure-education-gen-z https://www.theverge.com/22684730/students-file-folder-direc...
  - balder1991 2y ago
    This argument makes no sense considering the diversity of tastes and desires in the community. It’s incredibly difficult to put more people to work together and achieve superior results, especially if it’s something decentralized and voluntary. You can pretty well end up with more chaos and internal conflicts instead.
  - soulofmischief 2y ago
    Why are you so hellbent on dictating how volunteers spend their time? Have you ever considered that these people are satisfied with their interests and truly could not care less about your opinion? Or that your opinion is just that-- yours? Not some absolute truth? Anyway, it's beside the point, as there are multiple high quality Linux distributions to choose from, thanks to a large de-duplication of efforts through libraries.
rfoo 2y ago
Honest question, just curious what's your definition of "different arch", is a model with GQA=8 different than one with GQA=16 in your mind? I guess no? How about GQA vs MHA, or GQA vs MLA? If anything attention-like is same in your mind, is S5 and RWKV different arch given that both are some kind of linear RNN?