5 ms·

It's also just (deliberately) misleading. It's based on the 13B Vicuna/Llama model, not 175B GPT-3 or 1T GPT-4. There is zero justification for calling it MiniG

It's also just (deliberately) misleading. It's based on the 13B Vicuna/Llama model, not 175B GPT-3 or 1T GPT-4. There is zero justification for calling it MiniGPT-4. A more honest name would be Visual-Vicuna or Son-of-BLIP.

sebzim4500 4y ago
I don't see how it's misleading. MiniGPT-4 makes it sound like a smaller alternative to GPT-4, if it was based on GPT-4 there would be nothing 'mini' about it.
- HarHarVeryFunny 4y ago
  It has more in common with GPT-3 than GPT-4 in terms of size, but in reality it's based on Vicuna/Llama which is 10x smaller than either, so as far as the LLM part of it goes its not mini-anything - it's just straight-up Vicuna 13B. The model as a whole is just BLIP-2 with a larger linear layer, and using Vicuna as the LLM. If you look at their code it's literally using the entire BLIP-2 encoder (Salesforce code). https://arxiv.org/pdf/2301.12597.pdf https://arxiv.org/pdf/2301.12597.pdf
  - fire 4y ago
    vicuna was done with sharegpt transcripts, right? did they ever say if those transcripts were while users were using gpt3.5 or gpt4.0?
    - HarHarVeryFunny 4y ago
      I haven't read the details of how they created the training data.
- deleted 4y ago
  [deleted]
Tepix 4y ago
> 1T GPT-4 The number of parameters used for GPT-4 is unknown.
- HarHarVeryFunny 4y ago
  I got the 1T GPT-4 number from here - this is the video that goes with the Microsoft "Sparks of AGI" paper, by a Microsoft researcher that had early access to GPT-4 as part of their relationship with OpenAI. https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c
  - sandkoan 4y ago
    Bubeck has clarified that the "1 trillion" number he was throwing around was just a hypothetical metaphorical—it was in no way shape or form implying that GPT-4 has 1 trillion parameters [0]. [0] https://twitter.com/SebastienBubeck/status/1644151579723825154 https://twitter.com/SebastienBubeck/status/16441515797238251...
    - HarHarVeryFunny 4y ago
      OK - thanks! So we're back to guessing ... A couple of years ago Altman claimed that GPT-4 wouldn't be much bigger than GPT-3 although it would use a lot more compute. https://news.knowledia.com/US/en/articles/sam-altman-q-and-a-gpt-and-agi-lesswrong-aa6293dde7b95b537f5f50e37c861645fd9b4dbb https://news.knowledia.com/US/en/articles/sam-altman-q-and-a... OTOH, given the massive performance gains scaling from GPT-2 to GPT-3, it's hard to imagine them not wanting to increase the parameter count at least by a factor of 2, even if they were expecting most of the performance gain to come from elsewhere (context size, number of training tokens, data quality). So in 0.5-1T range, perhaps ?
    - HarHarVeryFunny 4y ago
      FWIW, Stephen Gou, Manager of ML at Cohere, is currently doing a Reddit AMA, and is also guessing at 1T params for GPT-4. https://www.reddit.com/r/IAmA/comments/12rvede/im_stephen_gou_manager_of_ml_founding_engineer_at/ https://www.reddit.com/r/IAmA/comments/12rvede/im_stephen_go...
EntrePrescott 4y ago
> Son-of-BLIP maybe even add an "a" for extra spice: Son-of-a-BLIP

sebzim4500 4y ago

I don't see how it's misleading. MiniGPT-4 makes it sound like a smaller alternative to GPT-4, if it was based on GPT-4 there would be nothing 'mini' about it.

HarHarVeryFunny 4y ago
It has more in common with GPT-3 than GPT-4 in terms of size, but in reality it's based on Vicuna/Llama which is 10x smaller than either, so as far as the LLM part of it goes its not mini-anything - it's just straight-up Vicuna 13B. The model as a whole is just BLIP-2 with a larger linear layer, and using Vicuna as the LLM. If you look at their code it's literally using the entire BLIP-2 encoder (Salesforce code). https://arxiv.org/pdf/2301.12597.pdf https://arxiv.org/pdf/2301.12597.pdf
- fire 4y ago
  vicuna was done with sharegpt transcripts, right? did they ever say if those transcripts were while users were using gpt3.5 or gpt4.0?
  - HarHarVeryFunny 4y ago
    I haven't read the details of how they created the training data.
deleted 4y ago
[deleted]

Tepix 4y ago

> 1T GPT-4 The number of parameters used for GPT-4 is unknown.

HarHarVeryFunny 4y ago
I got the 1T GPT-4 number from here - this is the video that goes with the Microsoft "Sparks of AGI" paper, by a Microsoft researcher that had early access to GPT-4 as part of their relationship with OpenAI. https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c
- sandkoan 4y ago
  Bubeck has clarified that the "1 trillion" number he was throwing around was just a hypothetical metaphorical—it was in no way shape or form implying that GPT-4 has 1 trillion parameters [0]. [0] https://twitter.com/SebastienBubeck/status/1644151579723825154 https://twitter.com/SebastienBubeck/status/16441515797238251...
  - HarHarVeryFunny 4y ago
    OK - thanks! So we're back to guessing ... A couple of years ago Altman claimed that GPT-4 wouldn't be much bigger than GPT-3 although it would use a lot more compute. https://news.knowledia.com/US/en/articles/sam-altman-q-and-a-gpt-and-agi-lesswrong-aa6293dde7b95b537f5f50e37c861645fd9b4dbb https://news.knowledia.com/US/en/articles/sam-altman-q-and-a... OTOH, given the massive performance gains scaling from GPT-2 to GPT-3, it's hard to imagine them not wanting to increase the parameter count at least by a factor of 2, even if they were expecting most of the performance gain to come from elsewhere (context size, number of training tokens, data quality). So in 0.5-1T range, perhaps ?
  - HarHarVeryFunny 4y ago
    FWIW, Stephen Gou, Manager of ML at Cohere, is currently doing a Reddit AMA, and is also guessing at 1T params for GPT-4. https://www.reddit.com/r/IAmA/comments/12rvede/im_stephen_gou_manager_of_ml_founding_engineer_at/ https://www.reddit.com/r/IAmA/comments/12rvede/im_stephen_go...

EntrePrescott 4y ago

> Son-of-BLIP maybe even add an "a" for extra spice: Son-of-a-BLIP