6 ms·

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning techniqu

I had to ask GPT what MoE means: "MoE" in the context of artificial intelligence typically stands for "Mixture of Experts". This is a machine learning technique that is based on the idea of dividing a problem into sub-problems, solving each sub-problem with a specialized "expert" (or model), and then combining their outputs.

ShamelessC 3y ago
Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however. Then another "routing model" decides which model is most suitable for the given user prompt. Given they use relatively few experts, each one is likely similarly capable to the others on many tasks. I assume this make deployment easier and is a "more conservative" less risky approach. Even if the wrong model is chosen by the router, answers should still tend to be somewhat acceptable, for instance.
- ta988 3y ago
  That's interesting because that's more or less on more level above the multi-head attention.
- refulgentis 3y ago
  Source?
  - ShamelessC 3y ago
    Just theorizing from the top-level post here. No clue if it's legitimate.
    - toxik 3y ago
      To be clear, you just made up MoE details while MoE is actually well established and hails from decades old research?
      - natpalmer1776 3y ago
        This is common behavior for inference based learners who don’t hail from strong academic backgrounds. Many developers who are self taught utilize a similar method of learning, essentially using pattern recognition to make “educated guesses” that are then internalized as potential facts and tested at the earliest opportunity. In this instance the test was to project the incorrect information out onto a public forum containing experts and using the presence of contradiction as weak evidence to the validity of their newfound knowledge. Yes, this is done in lieu of actually looking up extended details on what something means. It has it’s advantages though.
        reaperman 3y ago
        > It has its advantages though. Seems the advantage is somewhat localized to the individual inference-based learner; it doesn't seem like a pro-social strategy which would optimize benefit to the group. Overall this seems like it would generalize to widespread misinformation if the majority of uses adopted this behavior. I'm guessing it's in the best interests of the wider group to try to minimize the occurrence of this type of participation.
        natpalmer1776 3y ago
        The advantages in a social setting lie in the introduction of entropy, that is _creativity_, to a community. In a rigorous academic setting and with proper training these individuals are more likely identify links between ideas or information that may not seem obvious at first, and tend to be your more 'eccentric' academics. For the interests of the wider group, the best outcome is to help these individuals refine their communication to make it clear when information they present is unsubstantiated inference as opposed to verified knowledge. Once the proper 'rules of engagement' are outlined the contributions of these individuals is an oftentimes useful 'ingredient' to the success of many enterprises.
    - drumttocs8 3y ago
      I would have presented it as a question versus a statement, in that case.
- why_only_15 3y ago
  This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
  - ShamelessC 3y ago
    Oh I’m happy to admit if I’m wrong in the details. My bad. So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?
- nazka 3y ago
  Please don’t post assumptions while making them look like you know 100% what you are talking about…
aaron695 3y ago
[flagged]
- doctor_eval 3y ago
  If the answer is wrong, perhaps you could post a correction so that we are all better off, instead of just insulting me. Honestly, I've had a fairly rough day, and your answer has made me a bit more upset than perhaps I should be. At least GPT doesn't act like a jerk when I ask it a stupid question.
  - natpalmer1776 3y ago
    I think the complaint most folks have for people posting ChatGPT responses is that it adds nothing 'human' to the conversation. It's sort of like copy and pasting a wikipedia article into a comment, which in itself isn't necessarily wrong. The 'wikipedia' comment however does comes off as an impersonal PSA that also happens to be citing a newfangled encyclopedia that makes an easy target for ire. If you care to accommodate those who seemingly don't care to accommodate you, try phrasing your messages connect it in some way to the conversation with a couple of sentences about how that information changed your perspective or helped you understand something. That way, rather than making an announcement containing the definition of a word or concept, you're participating in the discourse as an active participant.
- kristiandupont 3y ago
  You should consider asking ChatGPT for help in clarifying your point(s).
- ikekkdcjkfke 3y ago
  [flagged]

ShamelessC 3y ago

Yep they (would) basically have 8-16 "experts" that are each about the size of GPT-3. Since they each see different batches of the dataset, they learn to model those distributions independently rather than the distribution of the whole dataset. Some of the attention is shared between them however. Then another "routing model" decides which model is most suitable for the given user prompt. Given they use relatively few experts, each one is likely similarly capable to the others on many tasks. I assume this make deployment easier and is a "more conservative" less risky approach. Even if the wrong model is chosen by the router, answers should still tend to be somewhat acceptable, for instance.

ta988 3y ago
That's interesting because that's more or less on more level above the multi-head attention.
refulgentis 3y ago
Source?
- ShamelessC 3y ago
  Just theorizing from the top-level post here. No clue if it's legitimate.
  - toxik 3y ago
    To be clear, you just made up MoE details while MoE is actually well established and hails from decades old research?
    - natpalmer1776 3y ago
      This is common behavior for inference based learners who don’t hail from strong academic backgrounds. Many developers who are self taught utilize a similar method of learning, essentially using pattern recognition to make “educated guesses” that are then internalized as potential facts and tested at the earliest opportunity. In this instance the test was to project the incorrect information out onto a public forum containing experts and using the presence of contradiction as weak evidence to the validity of their newfound knowledge. Yes, this is done in lieu of actually looking up extended details on what something means. It has it’s advantages though.
      - reaperman 3y ago
        > It has its advantages though. Seems the advantage is somewhat localized to the individual inference-based learner; it doesn't seem like a pro-social strategy which would optimize benefit to the group. Overall this seems like it would generalize to widespread misinformation if the majority of uses adopted this behavior. I'm guessing it's in the best interests of the wider group to try to minimize the occurrence of this type of participation.
        natpalmer1776 3y ago
        The advantages in a social setting lie in the introduction of entropy, that is _creativity_, to a community. In a rigorous academic setting and with proper training these individuals are more likely identify links between ideas or information that may not seem obvious at first, and tend to be your more 'eccentric' academics. For the interests of the wider group, the best outcome is to help these individuals refine their communication to make it clear when information they present is unsubstantiated inference as opposed to verified knowledge. Once the proper 'rules of engagement' are outlined the contributions of these individuals is an oftentimes useful 'ingredient' to the success of many enterprises.
  - drumttocs8 3y ago
    I would have presented it as a question versus a statement, in that case.
why_only_15 3y ago
This is not how mixture of experts works at all. The experts are chosen on each layer, not for the whole network, and attention is shared between all of them.
- ShamelessC 3y ago
  Oh I’m happy to admit if I’m wrong in the details. My bad. So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?
nazka 3y ago
Please don’t post assumptions while making them look like you know 100% what you are talking about…

aaron695 3y ago

[flagged]

doctor_eval 3y ago
If the answer is wrong, perhaps you could post a correction so that we are all better off, instead of just insulting me. Honestly, I've had a fairly rough day, and your answer has made me a bit more upset than perhaps I should be. At least GPT doesn't act like a jerk when I ask it a stupid question.
- natpalmer1776 3y ago
  I think the complaint most folks have for people posting ChatGPT responses is that it adds nothing 'human' to the conversation. It's sort of like copy and pasting a wikipedia article into a comment, which in itself isn't necessarily wrong. The 'wikipedia' comment however does comes off as an impersonal PSA that also happens to be citing a newfangled encyclopedia that makes an easy target for ire. If you care to accommodate those who seemingly don't care to accommodate you, try phrasing your messages connect it in some way to the conversation with a couple of sentences about how that information changed your perspective or helped you understand something. That way, rather than making an announcement containing the definition of a word or concept, you're participating in the discourse as an active participant.
kristiandupont 3y ago
You should consider asking ChatGPT for help in clarifying your point(s).
ikekkdcjkfke 3y ago
[flagged]