7 ms·

Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model

Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however. Then another "routing model" decides which model is most suitable for the given user prompt. Given they use relatively few experts, each one is likely similarly capable to the others on many tasks. I assume this make deployment easier and is a "more conservative" less risky approach. Even if the wrong model is chosen by the router, answers should still tend to be somewhat acceptable, for instance.

ta988 3y ago
That's interesting because that's more or less on more level above the multi-head attention.
refulgentis 3y ago
Source?
- ShamelessC 3y ago
  Just theorizing from the top-level post here. No clue if it's legitimate.
  - toxik 3y ago
    To be clear, you just made up MoE details while MoE is actually well established and hails from decades old research?
    - natpalmer1776 3y ago
      This is common behavior for inference based learners who don’t hail from strong academic backgrounds. Many developers who are self taught utilize a similar method of learning, essentially using pattern recognition to make “educated guesses” that are then internalized as potential facts and tested at the earliest opportunity. In this instance the test was to project the incorrect information out onto a public forum containing experts and using the presence of contradiction as weak evidence to the validity of their newfound knowledge. Yes, this is done in lieu of actually looking up extended details on what something means. It has it’s advantages though.
      - reaperman 3y ago
        > It has its advantages though. Seems the advantage is somewhat localized to the individual inference-based learner; it doesn't seem like a pro-social strategy which would optimize benefit to the group. Overall this seems like it would generalize to widespread misinformation if the majority of uses adopted this behavior. I'm guessing it's in the best interests of the wider group to try to minimize the occurrence of this type of participation.
        natpalmer1776 3y ago
        The advantages in a social setting lie in the introduction of entropy, that is _creativity_, to a community. In a rigorous academic setting and with proper training these individuals are more likely identify links between ideas or information that may not seem obvious at first, and tend to be your more 'eccentric' academics. For the interests of the wider group, the best outcome is to help these individuals refine their communication to make it clear when information they present is unsubstantiated inference as opposed to verified knowledge. Once the proper 'rules of engagement' are outlined the contributions of these individuals is an oftentimes useful 'ingredient' to the success of many enterprises.
        chaos_emergent 3y ago
        Given that this is an online forum, another advantage is that a conversational trail is left for others to discover. The inferences these types of individuals make are often based on a structure of knowledge and reality that others share, so the most common preconceived and incorrect notions tend to have the most documentation on how to ameliorate the incorrectness (given that these individuals are allowed to state their inferences out loud).
        doctor_eval 3y ago
        This had got to be the best thread I’ve ever (inadvertently) started.
    - deleted 3y ago
      [deleted]
    - ShamelessC 3y ago
      Can you detail what mistakes were made? I’m brushing up on it currently but having trouble grokking it.
    - rdlecler1 3y ago
      I believe it’s called ‘hallucination’
  - drumttocs8 3y ago
    I would have presented it as a question versus a statement, in that case.
why_only_15 3y ago
This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
- ShamelessC 3y ago
  Oh I’m happy to admit if I’m wrong in the details. My bad. So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?
nazka 3y ago
Please don’t post assumptions while making them look like you know 100% what you are talking about…

ta988 3y ago

That's interesting because that's more or less on more level above the multi-head attention.

refulgentis 3y ago

Source?

ShamelessC 3y ago
Just theorizing from the top-level post here. No clue if it's legitimate.
- toxik 3y ago
  To be clear, you just made up MoE details while MoE is actually well established and hails from decades old research?
  - natpalmer1776 3y ago
    This is common behavior for inference based learners who don’t hail from strong academic backgrounds. Many developers who are self taught utilize a similar method of learning, essentially using pattern recognition to make “educated guesses” that are then internalized as potential facts and tested at the earliest opportunity. In this instance the test was to project the incorrect information out onto a public forum containing experts and using the presence of contradiction as weak evidence to the validity of their newfound knowledge. Yes, this is done in lieu of actually looking up extended details on what something means. It has it’s advantages though.
    - reaperman 3y ago
      > It has its advantages though. Seems the advantage is somewhat localized to the individual inference-based learner; it doesn't seem like a pro-social strategy which would optimize benefit to the group. Overall this seems like it would generalize to widespread misinformation if the majority of uses adopted this behavior. I'm guessing it's in the best interests of the wider group to try to minimize the occurrence of this type of participation.
      - natpalmer1776 3y ago
        The advantages in a social setting lie in the introduction of entropy, that is _creativity_, to a community. In a rigorous academic setting and with proper training these individuals are more likely identify links between ideas or information that may not seem obvious at first, and tend to be your more 'eccentric' academics. For the interests of the wider group, the best outcome is to help these individuals refine their communication to make it clear when information they present is unsubstantiated inference as opposed to verified knowledge. Once the proper 'rules of engagement' are outlined the contributions of these individuals is an oftentimes useful 'ingredient' to the success of many enterprises.
        chaos_emergent 3y ago
        Given that this is an online forum, another advantage is that a conversational trail is left for others to discover. The inferences these types of individuals make are often based on a structure of knowledge and reality that others share, so the most common preconceived and incorrect notions tend to have the most documentation on how to ameliorate the incorrectness (given that these individuals are allowed to state their inferences out loud).
        doctor_eval 3y ago
        This had got to be the best thread I’ve ever (inadvertently) started.
  - deleted 3y ago
    [deleted]
  - ShamelessC 3y ago
    Can you detail what mistakes were made? I’m brushing up on it currently but having trouble grokking it.
  - rdlecler1 3y ago
    I believe it’s called ‘hallucination’
- drumttocs8 3y ago
  I would have presented it as a question versus a statement, in that case.

why_only_15 3y ago

This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.

ShamelessC 3y ago
Oh I’m happy to admit if I’m wrong in the details. My bad. So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?

nazka 3y ago

Please don’t post assumptions while making them look like you know 100% what you are talking about…