4 ms·

Looks like, again, this doesn't have GPT-4 processing video as much as a stack of video frames, concatenated and sent as a single image. But much closer to real

Looks like, again, this doesn't have GPT-4 processing video as much as a stack of video frames, concatenated and sent as a single image. But much closer to real!

ShamelessC 3y ago
The video is an actual live demo without any editing or other tricks involved and even includes reasonable mistakes and the code used. It is not close to real, it's just real.
trescenzi 3y ago
How does a video differ from a stack of video frames? Isn’t that all a video is? A bunch of images stuck together and played back quickly?
- famouswaffles 3y ago
  I'd guess you'd miss any audio that way. But otherwise, yeah a video is a stack of images.
- zwily 3y ago
  You could say that this demo is processing a 2.4s video that is 1.25fps.
- civilitty 3y ago
  In a technical sense most video is compressed using motion prediction algorithms so the preprocessing on the data is already significantly different to static images, containing more compressed information. Only the key frames are actually full images and they only make up 1-5% of the frames. On top of that the video container usually provides synchronized audio packets.
fortunefox 3y ago
Since audio is processed separately this isn't just close to real. it is real. After all what is video if not a stack of frames! :D
adtac 3y ago
Is "it's just processing frames" the new "it's just predicting the next token"?
- topspin 3y ago
  Nothing new here at all: trivializing the intellectual achievements of machines is SOP. This will continue until machines have surpassed every conceivable benchmark. At that point we'll be left with only our epic hubris.
riwsky 3y ago
I just found out it gets worse: turns out GPT-4 isn't processing images so much as arrays of pixels! And worse: turns out GPT-4 isn't processing pixels so much as integers representing in a position in some color space like RGB! And worse! turns out GPT-4 isn't processing integers so much as series of ones and zeroes! Now that this is public knowledge, I'm willing to bet this was the ugly "less than candid" truth that the board sacked Sam Altman over.
- zarzavat 3y ago
  As they say, quantity has a quality all of its own. If the framerate of a video is so slow as to be a slideshow, then it’s arguably not video anymore. Video has a connotation of appearing temporally continuous to the naked eye.
- omneity 3y ago
  There is a significant difference. Video has a temporal component (frames tend to be correlated with previous ones), and vision LLMs do not have some sort of hidden states to keep track of that temporal component. Using captions to bridge this only works to a certain extent (you're giving text descriptions of what happened in the past, not what had _actually_ happened).

Looks like, again, this doesn't have GPT-4 processing video as much as a stack of video frames, concatenated and sent as a single image. But much closer to real!

ShamelessC 3y ago

The video is an actual live demo without any editing or other tricks involved and even includes reasonable mistakes and the code used. It is not close to real, it's just real.

trescenzi 3y ago

How does a video differ from a stack of video frames? Isn’t that all a video is? A bunch of images stuck together and played back quickly?

famouswaffles 3y ago
I'd guess you'd miss any audio that way. But otherwise, yeah a video is a stack of images.
zwily 3y ago
You could say that this demo is processing a 2.4s video that is 1.25fps.
civilitty 3y ago
In a technical sense most video is compressed using motion prediction algorithms so the preprocessing on the data is already significantly different to static images, containing more compressed information. Only the key frames are actually full images and they only make up 1-5% of the frames. On top of that the video container usually provides synchronized audio packets.

fortunefox 3y ago

Since audio is processed separately this isn't just close to real. it is real. After all what is video if not a stack of frames! :D

adtac 3y ago

Is "it's just processing frames" the new "it's just predicting the next token"?

topspin 3y ago
Nothing new here at all: trivializing the intellectual achievements of machines is SOP. This will continue until machines have surpassed every conceivable benchmark. At that point we'll be left with only our epic hubris.

riwsky 3y ago

I just found out it gets worse: turns out GPT-4 isn't processing images so much as arrays of pixels! And worse: turns out GPT-4 isn't processing pixels so much as integers representing in a position in some color space like RGB! And worse! turns out GPT-4 isn't processing integers so much as series of ones and zeroes! Now that this is public knowledge, I'm willing to bet this was the ugly "less than candid" truth that the board sacked Sam Altman over.

zarzavat 3y ago
As they say, quantity has a quality all of its own. If the framerate of a video is so slow as to be a slideshow, then it’s arguably not video anymore. Video has a connotation of appearing temporally continuous to the naked eye.
omneity 3y ago
There is a significant difference. Video has a temporal component (frames tend to be correlated with previous ones), and vision LLMs do not have some sort of hidden states to keep track of that temporal component. Using captions to bridge this only works to a certain extent (you're giving text descriptions of what happened in the past, not what had _actually_ happened).