6 ms·
How far are we from intelligent visual deductive reasoning?
- minimaxir 3y agoGPT-4-Vision is a surprisingly lot better than I expected for a variety of practical tasks. OpenAI released a demo notebook with practical use cases for tagging/captioning a few weeks ago: https://cookbook.openai.com/examples/tag_caption_images_with_gpt4v https://cookbook.openai.com/examples/tag_caption_images_with... It will be much better once it gets function calling/structured data support so you don't have to the formatting hacks that notebook abuses to get a specific output format. Just don't ask it to describe a real person in the image or it gets fussy.
- WhitneyLand 3y agoGpt-4 will describe people, just don’t ask it to be judgmental. Prompt: Describe all objective details you can glean from the person in this photo. Result: The person in the photo appears to be a woman with fair skin. She has long, straight, light blonde hair, and blue eyes. Her eyebrows are well-shaped and lighter in color, matching her hair. She is wearing mascara and possibly eyeliner, which highlight her eyes, and she has a natural, possibly light pink, lipstick on. She is smiling, showing her teeth, which are white and well-aligned. The woman is wearing what seems to be a black outfit, although only a small portion is visible. She appears to be indoors, with a nondescript background. There are no visible tattoos, piercings (other than possibly in her ears, which are not fully visible), or distinctive marks that can be seen in the photograph.
- sandspar 3y agoChatGPT is happy to describe people if it has the option to give a flattering description. Our current chatbots' basic relation to humanity is flattery.
- carlossouza 3y ago> We observed that all the VLMs tend to be confident while being wrong. Interestingly, we observed that even when the entropy was high, models tried to provide a nonsensical rational, instead of acknowledging their inability to perform the task It looks like all current models suffer from an incurable case of Dunning–Kruger effect cognitive bias. All are at the peak of Mount Stupid.
- leereeves 3y agoHow can a neural network evaluate "confidence"? The parameters don't store any information about what inputs were seen in the training data (vs being interpolated) or how accurate the predictions were for those specific inputs. And even if they did, the training data was usually gathered voraciously, without much preference for quality reasoning.
- habitue 3y agoI don't know for sure, but here's a plausible mechanism for how: Multiple sub-networks detect the same pattern in different ways, and confidence is the percent of those sub-networks that fire for a particular instance. There's a ton of overlap and redundancy with so many weights, so there are lots of ways this could work
- brookst 3y agoThat’s good. Also maybe an architecture that runs the query through multiple times and then evaluates similarity of responses, then selects (or creates) the most-generated one, along with a confidence level of how many of the individual responses were aligned.
- montjoy 3y agoIIRC confidence in video is related to predicting what happens next vs what actually happens. If the two seem to correlate to the model it would give it a higher confidence ranking, which would then be used further for self-reinforced learning.
- pbhjpbhj 3y agoPhind gives me ChatGPT answers with relatively authoritative references to works on the web that (usually!) support the answer. Could it have a post-filter to fact check against the references? I guess that is a slight variation of the sibling (@habitue's) answer; both are checks against external material. I wonder if best resources could be catalogued as the corpus is processed, giving a document vector space to select resources for such 'sense' checking.
- CamperBob2 3y agoWhen self-driving cars were first becoming a thing, I was one of the skeptics who would say things like, "Call me when the AI can avoid situations like https://i.imgur.com/cmR2PDm.jpeg https://i.imgur.com/cmR2PDm.jpeg ." At the time, my thinking was that the software would have to recognize both fixed and moving objects nearby, capture their position and velocity, infer or look up their inertia tensors, and run at least a basic dynamics simulation to anticipate and prevent accidents like the one that's about to happen in the photo. However, I didn't consider the possibility that such a simulation might emerge from a model driven by nothing but visual input, given a large-enough data set and the right training. At this point my old argument seems like a losing one, even if present-day systems can't answer the "What happens next?" question reliably enough to trust in all driving situations. It won't exactly have to be perfect in order to outperform most humans. As a matter of fact, I just checked, and the situation in the meme is already partially recognized by ChatGPT 4: https://i.imgur.com/wLSBSkJ.png https://i.imgur.com/wLSBSkJ.png , even if it misses the comedic implications of the truck heading for the overpass. Whether it was somehow trained to give a useful, actionable answer for this particular question, I don't know, but the writing's on the proverbial wall. ------- Edit due to rate-limiting: note that I intentionally cropped out the WAIT FOR IT frame when I submitted the pic, so that it wouldn't see it as a meme loaded with comedy value. When I explicitly asked it what was funny about the image, ChatGPT4 eventually answered: "When the truck hits the overpass, given that the convertible is following closely behind, the occupants could be in for quite a shock. The impact might cause the portable toilets to become dislodged or damaged, potentially resulting in the contents being spilled. For the people in the convertible, this could mean an unexpected and unpleasant shower, especially because the convertible's top is down. The humor perceived here is in the anticipation of the unexpected and the sudden reversal of fortune for the individuals in what appears to be a fancy car – it's a form of slapstick humor." But I did have to provide a couple of strong hints, including telling it that the truck was not towing the car ( https://i.imgur.com/AlPRgEQ.png https://i.imgur.com/AlPRgEQ.png ). I was going to feed Claude the same hints, but it repeatedly failed to recognize the SMS code for registration.
- minimaxir 3y agoVideo data doesn't work well with these multimodal LLMs because images are still somewhat intensive for them process. Both the GPT-4-Vision demos and Gemini take a few frames from each video (~1 frame per second). Self-driving cars use much more efficient approaches for the real-time outputs needed.
- cgearhart 3y agoThe Georgia Tech Knowledge Based AI course involved building a program to answer ravens progressive matrix questions. The course was offered in the online MS program, so thousands of students have taken the course. The most impressive result I saw was one student who got nearly perfect results in about 25 lines of Python code. This may be a case where humans do well on the test, but you can do very well on the test without doing anything the way a human would. The fact that GPTs aren’t very good at the test isn’t probably evidence that they’re not really very smart, but it doesn’t really mean that if we fix them to do very well on the test that they’ve gotten any smarter.
- mjburgess 3y agoThere are an infinite number of algorithms to compute A from Q, given a se tof (Q, A). Almost none, surely, are intelligent. These proxy measure of intelligence are just arguments from ignorance, "I don't know how the machine computed A from Q, therefore...". But of course some of us do know how the machine did it; we can quite easily describe the algorithm. It just turns out no one wants to because its really dumb. Esp. if the alg is, as in all ML, "start with billions/trillions of data points in the (Q, A) space; generate a compressed representation ZipQA; and for novel Q' find decompressed A located close to Q similar to Q'" There are no theories of intelligence which would label that intelligence. And let me say, most such "theories" are ad-hoc PR that are rigged to make whatever the latest gizmo "intelligent". Any plausible theory begins form the initial intuition, "intelligence is what you do when you dont know what you're doing".
- thfuran 3y agoBut does the fact that we don't really know how human intelligence works actually suggest that it is categorically smarter?
- mjburgess 3y agoThat's a great argument and way of reversing the argument-from-ignorance line. That said, I think people who argue from ignorance suppose we don't know how AI works either. Since the admen selling it tell them that. We know exactly and precisely how AI works; we can fully explain it. What we don't know are circumstantial parts of the explanation (eg., what properties of the training data + alg gave rise to the specific weight w1=0.01). This is like knowing why the thermometer reads 21 deg C (since the motion of the molecules in the water, etc. etc.) -- but not knowing which molecules specifically bounced off it. This confusion about "what we dont know" allows the prophetic tech grifter class to prognosticate in their interest. Since "we dont know how AI works, then it might work such that i'll be very rich, so invest with me!" All this said, it's a great observation.
- GaggiX 3y agoNowadays VLMs are kind of Frankenstein models where the vision model is separated from the LLM and the communication is only one way where the vision embeddings are projected into textual embeddings and aligned; I think that this method severely limits the visual capability of the model and I think that it would be fixed by training a single decoder-only transformer on both textual and image tokens, I think that this method would allow some new incredible emergent behaviors like accurately translating images of logos to SVG, actually good ascii art, more meaningful openscad scripts and the ability to generate images with more meaningful composition and text (for now models can only generate meaningful text if it's available in the prompt, this is true even with Dalle 3 as the prompt is rewritten by ChatGPT).
- famouswaffles 3y agoFor now, LLMs are a lot better when you express this kind of task in text, preferably as a 1D arrangement. GPT-4's performance on the Abstraction and Reasoning Corpus. https://arxiv.org/abs/2305.18354 https://arxiv.org/abs/2305.18354 Also, this model trained to predict image tokens seemed to be able to perform Raven's Matrices type questions when expressed as an inpainting task in context. https://yutongbai.com/lvm.html https://yutongbai.com/lvm.html
- andoando 3y agoCheck out the ARC benchmark. https://lab42.global/arc/ https://lab42.global/arc/ Iirc, humans get something like 80% on the test, and the best performing AI models are at 25%
- iraqmtpizza 3y ago[flagged]
- deleted 3y ago[deleted]
- 2snakes 3y agoTo answer the question: How far from visual deductive reasoning? Consider G. Spencer Brown's Calculus of Indications. If you read his book it shows how using the mark/cross is trivial for visual deductions (if well-formed in translation). There is also work on this from Kaufmann at University of Illinois and William Bricken. Though this is a bit different than Raven's matrices.
- levocardia 3y agoA little surprised that neither the paper nor the discussion here cite Francois Chollet's 2019 paper on defining intelligence for machines [1], which talks explicitly about visual reasoning and proposed an RPM-like visual task as a benchmark. I found it very useful for thinking about what it means for an AI to be generally intelligent. From what I remember (having not read it in a while), his idea is basically that general intelligence is the efficiency with which you can learn, after appropriately "penalizing" any baked-in priors or knowledge beyond what a typical animal is born with. As an example, Magnus Carlsen's chess skill is a testament to his general intelligence because he became skilled at chess in one human lifetime, starting out with only basic priors about the world like objectness, agentness, and some others I can't remember. In contrast, DeepBlue is not as intelligent, because it started with many more priors specific to chess, and AlphaZero is also not as intelligent, because it acquired its chess skill much less efficiently, requiring the equivalent of thousands of human lifetimes. I don't remember if the paper considers whether 3.5 billion years of "pre-training" counts in this regard, though. [1] https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547
- SmooL 3y agoIsn't that a bit of a cop-out though? They're just redefining the word "intelligence" to be something else (here, the ability to learn new things). That's fine and all, but that doesn't answer the question we actually care about, which is the absolute magnitude of the ability. It doesn't matter how "long" an entity took to learn, or how "efficient" they were in learning - at the end of the day, AlphaZero will crush Magnus.
- deleted 3y ago[deleted]
- aaron695 3y ago[dead]
- mynameisnoone 3y agoHoly shit. I was thinking about this use-case the other day. I was thinking about whether a deep learning model could draw on the internet to identify objects in film screen captures and determine what they were or if they were unique. For example, I was watching the news and wondered if a mostly submerged vehicle was either a modern Jeep or an older Land Rover. Certainly AI/ML is the right tool for this task.