3 ms·
It is largely a text-only model, but tiny amounts of information from other modalities are indeed represented in the text. The best example is GPT-4's drawing a
by golol 3y ago
It is largely a text-only model, but tiny amounts of information from other modalities are indeed represented in the text. The best example is GPT-4's drawing ability.
From text only it might theoretically be possible to infer to some degree how our physical world works, but it should be extremely difficult in practice. That's why we we'll need multimodal models obv. But my intuition is that understanding captioned youtubes with a transformer should work just as well as LLMs do, it's just a matter of scale. I think a 100-1000x GPT-4 sized video model could model let's say 50ms frame rate videos pretty well, in the sense that you can watch minutes of video (vs read minutes of text) and the video looks reasonably realistic and preserves coherence/context over the duration. We just don't have the amount of compute necessary yet, so we have to be smarter with multimodality.
- flir 3y agoYou mean the SVG-generation stuff? I'm not convinced that's not just another form of text. It can "speak SVG" like it can "speak German". I spent a fair amount of time asking it to draw me SVGs of fish, and the results suggested "there's always a triangle, there's always a circle", not "it's got a tail and an eye, and they're connected to the body". It didn't seem to understand the relationships between the components. Could be wrong though. > But my intuition is that understanding captioned youtubes with a transformer should work just as well as LLMs do Wouldn't that be amazing. Bring it on, Google.
- golol 3y agoI know what you mean, but at some point when I played with it it could reasonably modify an image based on queries. Like, if I asked it to add smoke to the house's chimney, it would put grey blobs above the house, or if I asked it to give the cat a tail, it would put it roughly at the end. For me that's kind of a very rudimentary picture understanding, although I know it's a stretch.
- flir 3y agoNow that you mention it, I had the same experience - I asked it to draw the Mona Lisa, add a background, folded hands, stuff like that. Ok, yeah, I don't know what to make of that.
- nomel 3y ago> It didn't seem to understand the relationships between the components. Could be wrong though. There is evidence that you are wrong: https://www.youtube.com/watch?v=qbIk7-JPB2c https://www.youtube.com/watch?v=qbIk7-JPB2c Unfortunately, the "alignment tax" is a heavy one.