4 ms·

Isn't GPT-4 already over 1T parameters? And GPT-5 should be even "an order of magnitude" bigger than GPT-4...

• gammalost 3y ago
  Yes, the article even mentions that. Guess they went with that headline to attract readers.
• MichaelRazum 3y ago
  Was thinking the same, so 1T might bring you to the league of GPT-4. Actually in the best case, since it seems that meta, google, openai and so on have the most talent. Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?
  • lossolo 3y ago
    > Maybe 10T? 100T? I don't think we have enough training data to train models so big in a way to efficiently use all the params. We would need to generate training data, but then I don't know how effective it would be.
    • MichaelRazum 3y ago
      Have seen an interview with someone from openai. At least they claimed, that they are far away from running out of data and they are able to generate data if needed.
      • mrkeen 3y ago
        > they are able to generate data if needed. If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.
        • astrange 3y ago
          The point of generating data is so you can teach the model its distribution, not so it can memorize it.
• checkyoursudo 3y ago
  Was it ever confirmed whether GPT-4 is a Mixture of Experts or not?
  • cschmidt 3y ago
    I just asked GPT-4, and it denies being a MoE model. User: Are you an MoE model? ChatGPT: No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach. In a GPT model like mine, the entire model works in a unified manner on each task, without dividing it among specialized sub-models or experts. The model is trained on a diverse range of data and tasks, enabling it to generate responses across a wide array of topics and questions. While this approach lacks the specialization that MoE models offer, it provides broad, general-purpose language understanding and generation capabilities.
    • Jensson 3y ago
      That is almost surely a hallucination.
      • rkagerer 3y ago
        Curious, why do people gravitate to the term "hallucination" (as if it's a person not a tool) instead of just plainly stating that it's wrong?
        • therealdrag0 3y ago
          It’s a chat bot so a lot of language will be anthropomorphic. Even still this definition fits well “3. a false belief or impression; illusion; delusion.”
        • pacificmint 3y ago
          If I ask it what color an orange it and it says blue, that would be wrong. If you ask it a question and it makes up a completely fabricated story, like for example the case files in that recent legal case [1], then saying it was “wrong” doesn’t really seem to capture it. Calling it a hallucination is a great analogy, because the model made up a plausible sounding, but completely fabricated story. It saw things that were never real. [1] https://apnews.com/article/artificial-intelligence-chatgpt-courts-e15023d7e6fdf4f099aa122437dbb59b https://apnews.com/article/artificial-intelligence-chatgpt-c...