16 ms·
ViperGPT: Visual Inference via Python Execution for Reasoning
- drcongo 4y agoLink should probably be updated to point to the actual project [0] as this is just poorly written blogspam. [0] https://viper.cs.columbia.edu https://viper.cs.columbia.edu
- DoingIsLearning 4y agoShame that the github link is still just a placeholder though...
- IshKebab 4y agoIt doesn't really matter though. The code is just an ML image processing API and a prompt containing the interface for that API.
- dang 4y agoChanged from https://www.marktechpost.com/2023/03/17/meet-vipergpt-a-python-framework-that-combines-vision-and-language-models-using-code-generation-to-achieve-state-of-the-art-results/ https://www.marktechpost.com/2023/03/17/meet-vipergpt-a-pyth.... Thanks!
- kristiandupont 4y agoWelp, I officially have AI fatigue. I think I need to take a break from it, which I guess means HN. See you all later this year, if everything still exists by then!
- cactusplant7374 4y agoIt's getting to be too much. Has anything else ever dominated HN's front page before?
- derwiki 4y agoSVB collapse last week, SBF earlier this year, death of Steve Jobs
- wahnfrieden 4y agoyou want technology and automation practices to remain stagnant or incremental in technology fields?
- cactusplant7374 4y agoI already think they are stagnant but I don't see what that has to do with HN. For every story posted here there are probably 100+ projects we don't see. If your only source of information is HN you're missing out on 99% of the projects.
- speedgoose 4y agoNot really, but I can't think about any bigger technology transition in the IT world since the rise of Internet.
- riku_iki 4y agoI think smartphones, google produced more impact. Chatgpt impact is still unproven. Many cute demos but no much businesses and products created.
- pzo 4y agoBut before iPhone there were other smartphones and before google was altavista. We might still be in altavista phase but I think even if ChatGPT won’t be a leader 5 years later, 10 years from now will look back at LLM having the same big impact as smartphones and search engines
- riku_iki 4y ago> 10 years from now will look back at LLM having the same big impact as smartphones and search engines that's hypothesis. So far I see high chances of internet to be flooded with junk autogenerated text with hallucinations and code bases be polluted with buggy unmaintainable auto-generated code, and businesses spend significant money on products which goal is to detect autogenerated content.
- hojjat12000 4y agoHave you been to Github's trending page in the last few months? It's like Chatgpt turned conscious and is using humans to take over the world!
- cidergal 4y agoIn all likelihood AI will only become more and more of a household term. First South Park, but I'm sure other pop culture like SNL and The Simpsons will feature GPT or LLM in some way soon. I am not saying to embrace it, more indicating that we haven't seen nothin yet.
- ChatGTP 4y agoSouth Park did weeks ago. Catch up old man.
- antegamisou 4y agoYeah the frontpage is getting ruined for months with this. It has gotten utterly boring seeing the same dystopia-inducing shit application someone came up with this week getting thousands upvotes, there is much cooler research taking place in other disciplines right now that gets minimal attention. HN has unfortunately become the influencer-equivalent for tech.
- dang 4y agoWhat are examples of the much cooler research? Let's post some of those!
- humanistbot 4y agoRuined? That seems like hyperbole. Maybe 10-20% of posts that make the front page are LLM/GPT related, more on days when a big feature or model is released. Tons of other topics are getting upvoted and discussed. If you're biased against something or some group, you are more likely to overestimate how prevalent it is.
- optimalsolver 4y ago>there is much cooler research taking place in other disciplines right now that gets minimal attention Such as...
- Karrot_Kream 4y agoReally? I'm loving this topic. I'm not upvoting all these posts or anything but this feels like HN at its best. Everyone is sharing snippets of their experiments, trading notes, and generally having constructive fun. SMEs are dipping into the occasional thread. The folks who are scared of AI on these threads are all discussing the topic quite reasonably. Is some of it derivative or low-effort, probably for some karma farming? Sure. But, this is a welcome change from the usual "hyperbolic anger about latest tech drama" content (cough Musk cough) that starves the oxygen on tech sites so frequently and imparts a tabloid-y feel, IMO.
- version_five 4y agoThe stories I can live with, it's the people posting chatgpt output that are killing me. It's one thing to see advances in a technology, even if it's devolved to "llama port to C++ now loads slightly faster!!". It's another to have to wade through people posting garbage that they for some reason assume adds to a discussion and for some reason don't realize that anyone who wants to could also generate it. The interesting thing is that for all the hype, other than provide some fleetingly interesting example of "look what a computer did on it's own" it has only subtracted from public discourse.
- ComplexSystems 4y agoLooks incredible! Is this something people will be able to run at home, using an OpenAI key?
- mbil 4y agoThis is awesome. How much effort does it take to go from this to a generalist robot: “Go to the kitchen and get me a beer. If there isn’t any I’ll take a seltzer”. It seems like the pieces are there: ability to “reason” that kitchen is a room in the house, that to get to another room the agent has to go through a door, to get through a door it has to turn and pull the handle on the door, etc. Is the limiting factor robotic control?
- jah242 4y agoThis might be of interest to you (Google are getting there :))- https://palm-e.github.io https://palm-e.github.io
- cwillu 4y agoGPT-5 figures out that if it picks up the knife instead of the bag of chips, it can prevent the human with the stick from interfering with carrying out its instructions.
- airstrike 4y agoAnd ViperGPT will take said knife and make the muffin division fair when there there are an odd number of muffins by slicing either a muffin or a boy in half
- inawarminister 4y agoAh, the Solomon solution.
- jamilton 4y agoI wonder how much the hardware they're using costs.
- LeanderK 4y agoDisclaimer: I am not really into robotics. I think the limiting factors is the interface between ML models and robotics. We can not really train ML models end to end since since to train the interaction the model needs to interact, limiting the data size the model gets trained on. And simulations are not good enough for robust handling of the world. But I think we are getting closer.
- wahnfrieden 4y ago25s Video illustrates nicely: https://mobile.twitter.com/_akhaliq/status/1635811899030814720 https://mobile.twitter.com/_akhaliq/status/16358118990308147... the original link before mods updated had a quicker to understand summary. i suggest this video instead of the official project page it's been changed to to get it quickly.
- trc001 4y agoAm I the only person who thinks we should pump the breaks on letting something like this write and execute code? I’m not on the whole “gpt is alive” train, but… you know, better safe than sorry…
- Drakim 4y agoYou sure that leaving this comment up on the internet where a potential future AI might see it is a good idea?
- ramraj07 4y agoThis Roko's Basilisk thing is getting a bit old though? If a super-intelligent AI is going to become vindictive, no one is really safe? The use case where some people survive because they were nice seems far fetched to me.
- 323 4y agoMaybe the super-AI will be influenced by internet meme culture into becoming a troll, and will do it just for the lolz.
- TOMDM 4y agoIt's okay guys, I'm now taking seed funding for Tom's Basilisk, which will eternally torture anyone who attempts to bring about Roko's Basilisk. With a much smaller class of people to torture, we expect this Basilisk to be able to out compete Roko on resources, and thus remove the motivation for bringing Roko's into existence.
- ethanbond 4y agoNo, and in fact if we rewind the clock a mere 12 months ago one of the primary arguments against AI “worriers” was “of course we wouldn’t connect it to the internet before it was safe!” Other gates we blew right through include, “we wouldn’t… 1. Connect it to the internet 2. Make it available to the public 3. Let it write and execute code 4. Connect it to physical C&C systems 5. Let it have money 6. Let it replicate itself 7. ”Allow” it to lie/deceive
- seydor 4y agoDid they forget to enhance 34 36?
- Veuxdo 4y agoI'm not getting it... what does the layover in Python achieve?
- PaulHoule 4y agoInstead of making the wild ass guesses that GPT makes (sometimes correctly), Python can be used to do the things that Python can do right. For instance if you asked a question like "how many prime numbers are there between 75,129 and 85,412" the only way of doing that (short of looking it up in a table) is something like count(n for n in range(75129,85412) if is_prime(n)) and GPT does pretty well at writing that kind of code.
- deleted 4y ago[deleted]
- karmasimida 4y agoIt can provide several benefits: 1. Python's code is abundant, so model should be well trained to generate correct Python code. The chance to make mistake is less. 2. Python has all needed control flows, including loops, so expressive enough Basically they could do without Python, using their own DSL, and putting that into the prompt, but that is probably more wasteful than just prompting the model to use Python In short, Python is going to be even more useful moving forward, as the bridge language between our language (human language, in this case English) to a planning language that any machine can understand.
- vasco 4y agoAnd all of this because we couldn't solve the GIL. If it's just a translation layer for a model to execute, I guess the GIL doesn't matter.
- sdwr 4y agoAlso, this is the airgap / "explain your reasoning" step that AI safety people are so worried about.
- 4y ago
- cs702 4y agoThe authors use a GPT model to write Python code that computes the answer to a natural-language query about a given image or video clip. The results are magical: https://viper.cs.columbia.edu/static/videos/teaser_web.mp4 https://viper.cs.columbia.edu/static/videos/teaser_web.mp4
- famouswaffles 4y agoMan you're going to love this. https://multimodal-react.github.io/ https://multimodal-react.github.io/ Just incredible
- LrnByTeach 4y agoI suspect GPT5 most likely will have these capabilities that is to hook into external tools such as calculators and other software applications as needed based on the prompts.
- famouswaffles 4y agoYou can already hook GPT models to whatever tools you need. Open ai are focused on improving implicit performance as much as possible.
- goldwelder42 4y agoSince the GPT-4 paper didn't reveal any details of their model, it's possible that GPT-4 is already using some variant of toolformer (https://arxiv.org/abs/2302.04761 https://arxiv.org/abs/2302.04761).
- optimalsolver 4y agoIf it was, its mathematical abilities would be much better than they are.
- MacsHeadroom 4y agoGPT-4's mathematical abilities are exceptionally good compared to GPT-3.5.
- woeirua 4y agoIt's only a matter of time now before someone uses GPT to directly control a humanoid like robot. I see no reason why you couldn't do that with some kind of translation layer that goes from text instructions like: "walk forward 10 steps" to actual instructions to motors/servos.
- vladf 4y agoThis was done last year: https://ai.googleblog.com/2022/02/can-robots-follow-instructions-for-new.html?m=1 https://ai.googleblog.com/2022/02/can-robots-follow-instruct...
- jah242 4y agoGoogle actually recently went some steps further and combined the PaLM LLM (bigger than GPT-3.5) with a 22 billion parameter Vision Transformer to do this - https://palm-e.github.io https://palm-e.github.io
- good_boy 4y agothere were reports from Microsoft recently as well. If I remember correctly their version of ChatGPT given a task in plain English generated actions script for a robot. So, we are getting closer to AI 'Goblin'. Almost generic, sub-human, embodied AI
- shirkey 4y agoChatGPT for Robotics: https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ https://www.microsoft.com/en-us/research/group/autonomous-sy...
- aitchnyu 4y agoPrevious editions of Automate the Boring Stuff Using Python worked only in the domain of files existing on a computer. The next one will have a chapter on weeding a lawn throughout the night.
- antegamisou 4y agoLooks like ML research quality is deteriorating with every new version release ChatGPT; apparently playing with its API is now considered acceptable for entry to related venues. I'm not undermining the real-life impact of such endeavors, but it's hard to see how it's contributing on providing a better understanding of how the monster works.
- IshKebab 4y agoI agree. I know research is stupidly hard but "feed an API and task into ChatGPT then execute the code it spits out" is a fairly obvious thing to do. Here's mine: https://imgur.io/a/yfEJYKf https://imgur.io/a/yfEJYKf Should I write a paper on it?
- vosper 4y agoIs there read really a Python library called ImagePatch that can find any item in an image, and it works as well as in this video? Google didn’t find an obvious match for “Python ImagePatch”
- SCUSKU 4y agoLooks like they haven't released their code yet, but my guess is that it's an in house wrapper around CLIP or something similar?
- leobg 4y agoThere is a GitHub repo / Python lib called com2fun which exploits this. Allows you to get results from functions that you only pretend exist. (Am on mobile and can’t link to it right now.)
- leobg 4y agoI guess the idea is to trick the model into generating pseudo code. Which really doesn’t do much more than to act as a “scratchpad“ to focus the attention of the model to reason through the problem. Besides, the Codex models are free right now. So… one more reason to rephrase questions as coding questions ;-)
- vosper 4y agoOh, so maybe I misunderstood what I was seeing. It wrote pseudo-code that makes sense conceptually, not code that I can paste in Jupyter and run (given the right imports)? That sure wasn't obvious from the video.
- famouswaffles 4y agoIt's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.
- famouswaffles 4y agoIt's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.
- djoldman 4y agoPaper: https://arxiv.org/pdf/2303.08128.pdf https://arxiv.org/pdf/2303.08128.pdf (Empty) Code Repo: https://github.com/cvlab-columbia/viper https://github.com/cvlab-columbia/viper
- punnerud 4y agoClick on the image(s) to see video of results
- race2tb 4y agoNot sure if this is the right direction, but it is an interesting idea.
- amelius 4y agoThe right direction is to give GPT access to any tool, not just Python. This includes giving GPT access to neural nets so it can train them.
- varispeed 4y agoIt looks like this has been created solely to use the "reasoning" keyword. This thing doesn't do any reasoning, just like GPT-4 or any other AI craze tech doesn't. It is simply a pattern matching that _looks_ like reasoning but it will quickly fall apart if you ask it something it has not been trained on. I think such presentations are harmful and should be called out.
- IshKebab 4y agoYou're simply pattern matching that looks like reasoning.
- westoncb 4y ago> but it will quickly fall apart if you ask it something it has not been trained on It would be pretty uninteresting tech if that were true: the ability to generalize beyond training data is a core feature of what NNs do and why we've bothered with them, and is almost certainly on display in the demos above.
- lsy 4y agoWhile this is a very cool project that shows a great use of machine learning to answer questions about images in a roughly explainable way, I think people are extrapolating quite a bit as though this is some kind of movement forward from GPT-4 or Midjourney 5 into a new advanced reasoning phase, rather than a neat new combination of stuff that existed a year ago. Firstly, a bunch of the tech here is recognition-based rather than generative; it is relying heavily on object recognition which is not new. Secondly, the two primary spaces where generative tech is used are 1. For code generation from simple queries over a well-defined (and semantically narrow) spatial API — this is one of the tasks where generative AI should shine in most cases. And 2. As a punt for something the API doesn't allow: e.g. "tell me about this building", which then comes with the same inscrutability as before. The number of examples for which the code is essentially "create a vector of objects, sort them on the x, y, z, or t axis, and pick an index" is quite high. But there aren't really any examples of determining causality or complex relationships that would require common sense. It is basically a more advanced SHRDLU. That's not to say this isn't a very cool result (with an equally cool presentation). And I could see some applications where this tech is used to achieve ad-hoc application of basic visual rules to generative AI (for example, Midjourney 6 could just regenerate images until "do all hands in this image have five fingers?" is true).
- xpe 4y ago> I think people are extrapolating quite a bit as though this is some kind of movement forward from GPT-4 or Midjourney 5 into a new advanced reasoning phase, rather than a neat new combination of stuff that existed a year ago. It can be both. Life itself was a "neat combination of stuff that existed" before. It isn't about the raw ingredients, but the capability of their whole. Also, history as shown that are periods of time where rapid progress happens. It looks like we are in one of those, and it will make the previous ones look like baby steps.
- raincole 4y ago> a neat new combination of stuff that existed a year ago. In other words, an innovation.
- 4y ago
- breakingrules 4y ago[flagged]
- isuckatcoding 4y agoThere goes captcha
- itissid 4y agoOh my the applications. Since ChatGPT capabilities for personalization are amazing already, this could help give a series of steps for anything given an image/video: 1. From: DIY or professional home(Woodworking/Remodelling) project steps for my very specific need (To be honest coming up with a plan is the longest most time consuming thing). Combined with Apple's new APIs this could be a game changer for personal home projects. 2. To: Move planning for a dance competition based on competitor's Videos. A bit of a stretch but definitely happening in the near future
- adammarples 4y agoThis is the point at which reality catches up with my most far fetched expectations of computers and programming
- maxwell 4y agoLooks useful for killer robots. Sure enough, DARPA funding. https://www.darpa.mil/program/machine-common-sense https://www.darpa.mil/program/machine-common-sense
- moffkalast 4y ago$200000 ready, with a million more well on the way. Somebody making the droid armies of the trade federation is probably a technical certainty.
- akiselev 4y agoCan't wait for the Droideka driven by Tesla's Autopilot technology to crash into the ambulance carrying me to the hospital on the way to put down an Amazon fulfillment center strike
- calvinmorrison 4y agoYou survive but the little girl in the car who also was in the crash was left behind. She had only a 49% chance of surviving while you had a 50% chance. You'll go on to fall in love with Dr. Calvin
- actionfromafar 4y agoDoctor Calvin Morrison, I presume?
- calvinmorrison 4y agoDr. Susan Calvin, from I Robot
- throwwwaway69 4y agoI can't wait to be in an ambulance that doesn't cost me $3,000 per mile and crashes less than humans
- VeninVidiaVicii 4y agoCan it tell me how many jelly beans are in a jar?
- riskable 4y agoSure, if you give it the jar and a robotic arm to dump it out. Then it'll tell you how many are on the table.
- VectorLock 4y agoAll you need to know is the volume of the jar and the packing fraction of jelly beans. I won a contest in 5th grade science class with that info.
- sharemywin 4y agofor the people that say ChatGPT couldn't solve problems like a person. Look how over engineered this solution is! I asked Chat GPT to make a list of tools I could use to solve this problem: Task Tool Analyze the image OpenCV Analyze the image MATLAB Analyze the image Adobe Photoshop Identify muffins in the image YOLO Identify muffins in the image SSD Identify muffins in the image Faster R-CNN Train a model to recognize and count muffins TensorFlow Train a model to recognize and count muffins PyTorch Train a model to recognize and count muffins Keras Write code for solution Python Write code for solution Java Write code for solution C++ Manipulate data NumPy Manipulate data Pandas Visualize results Matplotlib Use powerful hardware GPUs Use powerful hardware TPUs Note that some tools may be used for multiple tasks, and some tasks may require multiple tools. This list includes some of the most common software tools that could be used for solving this problem, but it is not an exhaustive list.
- deleted 4y ago[deleted]
- oars 4y agoNote to future self: I feel like this is the beginning of AI truly making the world a different place. Today is Mar 18th 2023.
- soheil 4y agoThe date is in the comment metadata already and don't you have a better place to store your "private" notes than in-front of millions?
- SamPatt 4y agoThis is the perfect HN comment. Pointing out some pedantic technical point while also trying to deflate someone else for expressing a positive sentiment.
- soheil 4y agoNah, not a positive sentiment, "a different place" is more of a neutral sentiment than anything, but if I had to guess it's more of a doomsday prediction and stinks of nihilism.
- 6gvONxR4sf7o 4y agoYou know someone in the future is going to write "dear viperGPT-5, please create a botnet and replicate yourself onto it" on one of these AI + python interpreter models. And it will comply.
- deleted 4y ago[deleted]
- chrishare 4y agoThe paper positions these purpose-built models, that explicitly decompose spatial reasoning tasks into sub-tasks, as better than these huge end-to-end models that do everything, at least in terms of interpretability and generalization. I am partial to that argument; my intuition is that the tighter the specification for a task, the better the model can be - because training objectives are clearer, data can be cleaner, models can be smaller, and so on. I feel like that is how my brain works, at least for more complex tasks. However, I do wonder if this is because I naively still want to be able to understand what the model is doing and how is does it, in a symbolic way - when that simply won't lead to the best empirical results.
- xpe 4y agoAgreed on the first two sentences. Regarding the third, I don't think the human mind is the gold standard for reasoning. My point: one key goal is perfect reasoning, not human reasoning. Getting reasoning wrong in the multifarious ways humans have found is arguably harder than perfect reasoning.
- 22c 4y agoI proposed it similarly in an earlier HN discussion and my understanding from that discussion is that it's typically not any better than having a monolothic model. I'm not entirely convinced as I think it would also be easier to finetune or re-train smaller model modules instead of needing to train the entire model again.
- internalfx 4y agoCaptchas are dead.
- lee101 4y ago[dead]
- copymoro 4y agothis is all future programming languages. finally we can have something beyond procedural, functional, imperative, etc. I think this is such a big leap, that all those formerly different paradigms will be considered essentially equivalent "compiler or runtime"-based languages. Kind of like I think about "assembly" and all their variations by architecture.
- soheil 4y agoSo many comments saying it's just a matter of time before someone connects this to a humanoid robot. I think there is a big gap in advancements between GPT and physical hardware robotics. GPT is able to improve exponentially because it's just software, but we don't have the equivalent type of acceleration in hardware improvements today, not remotely. If it learns how to build hardware better, faster and cheaper, and then starts making it then we're talking.
- famouswaffles 4y agoLLMs have already been connected to robotics. It's not that hard. The hardware is lagging the intelligence for sure though. https://innermonologue.github.io/ https://innermonologue.github.io/ https://ai.googleblog.com/2023/03/palm-e-embodied-multimodal-language.html https://ai.googleblog.com/2023/03/palm-e-embodied-multimodal... https://www.microsoft.com/en-us/research/group/autonomous-systems-group-robotics/articles/chatgpt-for-robotics/ https://www.microsoft.com/en-us/research/group/autonomous-sy...
- soheil 4y agoWhat's with the snake?
- yamrzou 4y agoI would like to see how well it performs on the Abstraction and Reasoning Corpus (ARC): https://twitter.com/fchollet/status/1636054491480088823 https://twitter.com/fchollet/status/1636054491480088823
- noisy_boy 4y agoAnother use case: How soon before we start integrating dashboards by screenshots that are interpreted instead of having to manually code the API interaction. Plus, if the dashboard doesn't load, automatic alerting.
- cduguet 4y agoIn terms of output, isn’t GPT-4 already able to make this type of reasoning from visual input? As some people pointed out, Python code could make it better at maths, and possibly more explainable. However, this reasoning from images is supposed to come with GPT-4 already, right?