11 ms·

> Kenn explained what multimodal AI is about, which can translate text not only accordingly into images, but also into music and video. Interesting that they a

by oidar 4y ago

> Kenn explained what multimodal AI is about, which can translate text not only accordingly into images, but also into music and video. Interesting that they are broadening instead of deepening the text side of GPT - maybe they are running into problems on that side and are building the feature set with low hanging fruit on adjacent areas.

xen2xen1 4y ago
Or it matches their search needs?
thatguyman 4y ago
it is interesting. i believe they may be hitting a wall as well
- flangola7 4y ago
  A paper from a week ago found that models trained on multiple data modes perform an order of magnitude better than text-only models of the same or even larger size.
  - dwallin 4y ago
    Can you share a link to this paper?
  - gwoolhurme 4y ago
    Genuinely curious, what does it mean in this context to perform better? (hopefully that doesn't come across as snarky as text sometimes does).
    - jerrygenser 4y ago
      Some of these large models are able to do zero shot learning and perform tasks they weren't explicitly trained on since the training objective is very general. Being able to perform more advanced types of zero shot learning tasks would be comparable and further the accuracy on those tasks can be evaluated
      - dmix 4y ago
        Any chance this could improve coding LLMs like Co-pilot? Or would that sort of thing be limited to source code feeds (not that Github has a shortage).
        refulgentis 4y ago
        The next big step for coding LLMs will be context window increases, leaked docs have OpenAI pricing for up to 16K I believe, 4x the current maximum. Now you're talking "write a class" instead of this line and maybe sometimes a method
        williamcotton 4y ago
        It can already reliably write a todo web browser application. With 16k and some other techniques, I’m guessing it could write a custom CMS database backed web application.
        flangola7 4y ago
        Not 16k, 32k. 8x the current window.
        refulgentis 4y ago
        Nice
        dmix 4y ago
        What is 16k referring to here
        refulgentis 4y ago
        I’ve begun to grok it as “the amount of ram I have to play in before I have to start sharding work” more literally and correctly, it’s the maximum number of tokens in the input and output, combined, where a token is 4/3 of a word So we’re shifting from 5K words maximum to 40K (per sibling comment, who pointed out 32K context leaked as well)
  - eob 4y ago
    I figure it’s a stand-in for embodiment.
  - fiso64 4y ago
    Are you referring to PALM-E? It didn't have any positive transfer for NLP tasks, in fact the unfrozen model performed slightly worse after the finetune. That being said, PALM-E wasn't really a multimodal model from the start, it's still basically just a text model with a visual one glued on top. Whether a truly multimodal model will be better at reasoning and data efficiency is still an open question though.
lumost 4y ago
There is substantial evidence that cross training on multiple tasks improves performance. ChatGPT struggles with intuition about the world, if it has image/video knowledge than it can potentially reason about what happens when you flip a plate upside down.
- mortenjorck 4y ago
  I wonder if there’s any reason this wouldn’t also apply in reverse: could a corpus of textual training data from which the role of a hand can be derived (including the fact that it naturally has five fingers) help DALL-E reliably render hands correctly?
  - lumost 4y ago
    Almost certainly. There is a vast corpus of text which references hands and what they are used for. Likewise there is a vast corpus of images which include hands. A sufficiently advanced statistical model can infer that any image with 6 fingers is a fabrication provided that it has learned how to classify fingers and has enough linguistic knowledge. Contrast this situation with a primarily image based model which will struggle to differentiate what situations detailed hands appear in compared with a patterned rug. There is an interesting “takeoff” point where models are able to generate enough “interesting” data to train themselves. Asking chatGPT for ml training data on various tasks makes me think we are close to that inflection point.
    - noah_buddy 4y ago
      Can’t wait for AI to tell people with atypical bodies that they don’t exist.
      - lumost 4y ago
        Definitely a bias, chatGPT will likely believe that only individuals with figures similar to models are real.
    - IIAOPSW 4y ago
inconceivable 4y ago
i think it's likely that they hired a diversified AI team with expertise in multiple areas and they're all working on their own areas. parallel efforts.
fizx 4y ago
There's no more text to train on. Also, amazingly, transfer learning works better than anyone could have hoped! As an analogy, imagine that you could show that football players learned musical instruments 50% quicker than non-athletes. That would be amazing! But LLMs can write music and training them on music makes them better writers. It's almost like intelligence & skills are completely general!
- CuriouslyC 4y ago
  That's just not true. There's no more super cheap data, but there's still probably about 5x the data we are currently feeding these models just waiting to be scanned and OCRd, and most of it is higher quality (if less topical).
- osigurdson 4y ago
  The fact that AI requires so much data does make me think that we are somehow on the wrong track. Evolution didn't scan every atom on earth in order to produce a brain. FSD is a good example of this. It is ridiculous to me that we need to show the car videos of every possible situation in order for it to know how to drive.
  - MattRix 4y ago
    It’s not the wrong track, it’s just very early and specialized. As you get more multi-modal models with better ability to learn/remember on the fly, they will require less and less data to train.
    - osigurdson 4y ago
      I'm pretty sure we will look back and laugh about how this was done in 50 years. No disrespect to anyone working in the area now of course, but I think better, more natural approaches will emerge.
  - ihatepython 4y ago
    > The fact that AI requires so much data does make me think that we are somehow on the wrong track. I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.

by oidar 4y ago

xen2xen1 4y ago

Or it matches their search needs?

thatguyman 4y ago

it is interesting. i believe they may be hitting a wall as well

flangola7 4y ago
A paper from a week ago found that models trained on multiple data modes perform an order of magnitude better than text-only models of the same or even larger size.
- dwallin 4y ago
  Can you share a link to this paper?
- gwoolhurme 4y ago
  Genuinely curious, what does it mean in this context to perform better? (hopefully that doesn't come across as snarky as text sometimes does).
  - jerrygenser 4y ago
    Some of these large models are able to do zero shot learning and perform tasks they weren't explicitly trained on since the training objective is very general. Being able to perform more advanced types of zero shot learning tasks would be comparable and further the accuracy on those tasks can be evaluated
    - dmix 4y ago
      Any chance this could improve coding LLMs like Co-pilot? Or would that sort of thing be limited to source code feeds (not that Github has a shortage).
      - refulgentis 4y ago
        The next big step for coding LLMs will be context window increases, leaked docs have OpenAI pricing for up to 16K I believe, 4x the current maximum. Now you're talking "write a class" instead of this line and maybe sometimes a method
        williamcotton 4y ago
        It can already reliably write a todo web browser application. With 16k and some other techniques, I’m guessing it could write a custom CMS database backed web application.
        flangola7 4y ago
        Not 16k, 32k. 8x the current window.
        refulgentis 4y ago
        Nice
        dmix 4y ago
        What is 16k referring to here
        refulgentis 4y ago
        I’ve begun to grok it as “the amount of ram I have to play in before I have to start sharding work” more literally and correctly, it’s the maximum number of tokens in the input and output, combined, where a token is 4/3 of a word So we’re shifting from 5K words maximum to 40K (per sibling comment, who pointed out 32K context leaked as well)
- eob 4y ago
  I figure it’s a stand-in for embodiment.
- fiso64 4y ago
  Are you referring to PALM-E? It didn't have any positive transfer for NLP tasks, in fact the unfrozen model performed slightly worse after the finetune. That being said, PALM-E wasn't really a multimodal model from the start, it's still basically just a text model with a visual one glued on top. Whether a truly multimodal model will be better at reasoning and data efficiency is still an open question though.

lumost 4y ago

There is substantial evidence that cross training on multiple tasks improves performance. ChatGPT struggles with intuition about the world, if it has image/video knowledge than it can potentially reason about what happens when you flip a plate upside down.

mortenjorck 4y ago
I wonder if there’s any reason this wouldn’t also apply in reverse: could a corpus of textual training data from which the role of a hand can be derived (including the fact that it naturally has five fingers) help DALL-E reliably render hands correctly?
- lumost 4y ago
  Almost certainly. There is a vast corpus of text which references hands and what they are used for. Likewise there is a vast corpus of images which include hands. A sufficiently advanced statistical model can infer that any image with 6 fingers is a fabrication provided that it has learned how to classify fingers and has enough linguistic knowledge. Contrast this situation with a primarily image based model which will struggle to differentiate what situations detailed hands appear in compared with a patterned rug. There is an interesting “takeoff” point where models are able to generate enough “interesting” data to train themselves. Asking chatGPT for ml training data on various tasks makes me think we are close to that inflection point.
  - noah_buddy 4y ago
    Can’t wait for AI to tell people with atypical bodies that they don’t exist.
    - lumost 4y ago
      Definitely a bias, chatGPT will likely believe that only individuals with figures similar to models are real.
  - IIAOPSW 4y ago

inconceivable 4y ago

i think it's likely that they hired a diversified AI team with expertise in multiple areas and they're all working on their own areas. parallel efforts.

fizx 4y ago

There's no more text to train on. Also, amazingly, transfer learning works better than anyone could have hoped! As an analogy, imagine that you could show that football players learned musical instruments 50% quicker than non-athletes. That would be amazing! But LLMs can write music and training them on music makes them better writers. It's almost like intelligence & skills are completely general!

CuriouslyC 4y ago
That's just not true. There's no more super cheap data, but there's still probably about 5x the data we are currently feeding these models just waiting to be scanned and OCRd, and most of it is higher quality (if less topical).
osigurdson 4y ago
The fact that AI requires so much data does make me think that we are somehow on the wrong track. Evolution didn't scan every atom on earth in order to produce a brain. FSD is a good example of this. It is ridiculous to me that we need to show the car videos of every possible situation in order for it to know how to drive.
- MattRix 4y ago
  It’s not the wrong track, it’s just very early and specialized. As you get more multi-modal models with better ability to learn/remember on the fly, they will require less and less data to train.
  - osigurdson 4y ago
    I'm pretty sure we will look back and laugh about how this was done in 50 years. No disrespect to anyone working in the area now of course, but I think better, more natural approaches will emerge.
- ihatepython 4y ago
  > The fact that AI requires so much data does make me think that we are somehow on the wrong track. I know the entire AI industry on the wrong track, but there's nothing that can be done about it. Sometimes you just have to wait for the crash and burn, and then things can change. It's like being at the ocean and trying to stop a wave from crashing after it has already formed, you just have to let it resolve itself of its own volition. As Einstein said, stupidity is infinite.