3 ms·

Speculating on How GPT-4 Changes Computer Vision

4 points by yeldarb 4y ago

ggm 4y ago
The key word here is "speculating" because I don't see ANY evidence that the visual processing is available in GPT-4 to perform semantically driven tracking of the order "follow the footballer" implied in the article. The linguistic processing to understand "footballer" exists as a distinct category, and amass the image set of footballers, sure. But, to synthesise a tracker which goes from generic model in linguistic terms to visual processing? I don't see it. If nothing else, the primary elements of what makes an individual footballer distinctive, are really constrained. There's 20,000 fans in identically coloured 'fan grade' strip, not to mention 10 other soccer players in the actual same strip in the same visual field to dis-ambiguate from: They're all in motion, focussed on the ball, performing the same class of movement. Disambiguating the gorilla behind the basketball game is a low-threshold test. Visual processing for e.g. waste sorting in recycling, parts alignment on production line feeds, fruit sorting at point of collection, there's plenty of visual processing tasks which can be improved upon, but flinging "GPT" at it in the current form? Hang on: the training set here isn't necessarily in the same domain of need. I understand this as a statistical model of a project space of inputs. What qualities in the training net for 2D images at large will mimic discrimination of moving objects on a feed belt at video speed feed rates? So maybe a GPT "like" solution to visual problems is out there but it demands training. A LOT of inputs. A lot MORE than will be culled from farming the public domain image spaces and textual inputs which make the 65B inputs for what we're all playing with. Basically, where is the farm of youtube videos of garbage belt feedstock to train on? (It's an n-times problem: this training set won't help with processing ripe or unripe apples in the orchard. each niche demands it's own training until we have a more generalised model and again, I don't see how GPT per se as we have it, will derive that. )

4 points by yeldarb 4y ago

ggm 4y ago

The key word here is "speculating" because I don't see ANY evidence that the visual processing is available in GPT-4 to perform semantically driven tracking of the order "follow the footballer" implied in the article. The linguistic processing to understand "footballer" exists as a distinct category, and amass the image set of footballers, sure. But, to synthesise a tracker which goes from generic model in linguistic terms to visual processing? I don't see it. If nothing else, the primary elements of what makes an individual footballer distinctive, are really constrained. There's 20,000 fans in identically coloured 'fan grade' strip, not to mention 10 other soccer players in the actual same strip in the same visual field to dis-ambiguate from: They're all in motion, focussed on the ball, performing the same class of movement. Disambiguating the gorilla behind the basketball game is a low-threshold test. Visual processing for e.g. waste sorting in recycling, parts alignment on production line feeds, fruit sorting at point of collection, there's plenty of visual processing tasks which can be improved upon, but flinging "GPT" at it in the current form? Hang on: the training set here isn't necessarily in the same domain of need. I understand this as a statistical model of a project space of inputs. What qualities in the training net for 2D images at large will mimic discrimination of moving objects on a feed belt at video speed feed rates? So maybe a GPT "like" solution to visual problems is out there but it demands training. A LOT of inputs. A lot MORE than will be culled from farming the public domain image spaces and textual inputs which make the 65B inputs for what we're all playing with. Basically, where is the farm of youtube videos of garbage belt feedstock to train on? (It's an n-times problem: this training set won't help with processing ripe or unripe apples in the orchard. each niche demands it's own training until we have a more generalised model and again, I don't see how GPT per se as we have it, will derive that. )