43 ms·
HuggingGPT: Solving AI tasks with ChatGPT and its friends in HuggingFace
- amelius 4y agoCan we please stop generating bounding boxes and instead generate a proper segmentation using bitmasks?
- cameronfraser 4y agoit depends on the problem?
- amelius 4y agoTrue, but a bitmask is always better than a rectangle. And the computational resources required for this problem are not very large compared to other AI workloads, so it should be a no-brainer from a usefulness viewpoint.
- arcastroe 4y agoPage 14 of the document does exactly that.
- roddylindsay 4y agoWhat could go wrong?
- deleted 4y ago[deleted]
- maCDzP 4y agoAm I the only one with an uneasy feeling about all these advancements? Are we going fast or is it just because of the buzz? I have a hard time separating the two. It’s very exiting and I want to keep being exited. I don’t want to become terrified.
- catchnear4321 4y agoBeing terrified doesn’t serve much purpose at this point. We are going too fast. Have been for years. This is just the first clear indication. Braking is fatal, but some seem pretty hell-bent. Deceleration is complicated, and it seems highly unlikely that there would be sufficient consensus for true deceleration. Local deceleration is simply waiting for the acceleration occurring somewhere else to overcome your efforts. The math hasn’t really changed for most individuals. At some point something big will happen. Singularity. Be excited and put your efforts towards what you value. One way or another, there is very little time left for wasting.
- quonn 4y agoI disagree. ChatGPT is a clear turning point. We have not been going too fast before, at least not outside LLMs. Singularity is a belief system, it has very little to do with AI. edit: I also think if we would ever get to a point where AI gets close to a point of possibly getting out of control as you imply it would simply be banned in the US/Canada/EU/Australia. Furthermore Latin America and Africa could and would be pressured to go along if needed. Which leaves some parts of Asia. China, maybe India and Russia. Probably only China. It could be cut off from the Internet if needed. We could build up a wall just like in the Cold War. My point being: This will not happen just because it happens. It will be a choice.
- JL-Akrasia 4y agoAn AI API marketplace is going to be amazing. But each API is going to need parameters that you can run gradient descent on to tune. This needs to be the foundation
- avereveard 4y agoGpt plus langchain agents is quite scary It will use every tool you give it to reach the goal you give him. It will try forever if needed. I bet state actor are already plugging in tooling to register social account and automate credible propaganda. Maybe not with gpt itself, but privately hosted fine tuned models. This can win elections. You can plug wordpress and build infinite blogs with infinite post with the unique scope of building mass around a topic. This can alter Wikipedia, many people don't ever check sources and take it at face value. You can not only build fake research paper, but fake the whole research team and their whole interactions with the community and investor. This can fraud millions. Tools enable this today.
- user- 4y agoIts a matter of time until connecting widely different AI tools will be super seamless, very exciting. The examples in the paper are pretty cool. I predict within the next year or so will see an sort of A.I assistant that is hooked up to dozens of LLMs and similar tools, and the end user will just ask their assistant to do things for them. That sci fi moment is almost here.
- chasd00 4y agoconnecting "widely different AI tools" reminds me of the good old pipe operator. Maybe an AI can be an "anything command" for a specific domain and then we just pipe them together so they can each weigh in with their own expertise. ...like an AI team or committee more or less.
- Workaccount2 4y agoI strongly suspect the first AGI will come sooner than expected on the back of a "glue" AI that can intelligently bond together a web of narrow AIs and utilities. I got access to the wolfram plugin for chatGPT, and it turned it from a math dummy to a math genius overnight. A small step for sure, but a hint of what's to come.
- _yb2s 4y agoI agree, I suspect AGI is possible right now with a similar system only slightly more sophisticated than this one. The right "glue" for existing models, and plugins to existing data sources all coordinated in a system. GTP-4 would do the managing, and handling, and some simple template API and handler script would allow it to call instances of itself or other models, track recursion depth, and automatically remind GTP-4 to stay on track and always use the correct 'templates' for requests. It could also remind it to create a condensed summary of it's goals and state that gets repeated back automatically to act as long term memory, and increase the effective context window size (edit: this is single variable external memory). I am afraid to explain this, because I have tried a preliminary version of it that I supervised step by step, and it seems to work. I think it is obvious enough that I won't have been the only one to think of it, so it would be safer to put the information out there so people can prepare. I see a big disconnect on here between people saying GPT-4 can't do things like this and is just a "stochastic parrot" or "glorified autocomplete," and people posting logs and summaries of it solving unexpectedly hard problems outside of any conceivable training set. My theory is that this disconnect is due to three major factors: * People confusing GPT-4 and GPT-3, as both are called "chatGTP" and most people haven't actually used GPT-4 because it requires a paid subscription, and don't realize how much better it is * Most popular conceptual explanations about how these models work imply that these actually observed capabilities should be fundamentally impossible * Expectations from movies, etc. about what AGI will be like, e.g. that it will never get confused or make mistakes, or that it won't have major shortcomings in specific areas. In practice this doesn't seem to limit it because it recognizes and fixes its mistakes automatically when it sees feedback (e.g. in programming)
- basch 4y ago
- lachlan_gray 4y agoIt would be really interesting to see how a system like this competes with multimodal transformers. If it's good, systems like GPT-4 and Kosmos-1 won't need to have a built-in vision system.
- levesque 4y agoThis read more like a technical demo than a scientific paper. Wonder why they put it on arXiv.
- Imnimo 4y agoWhy are the example outputs in the Figures ungrammatical? >A text can describe the given image: a herd of giraffes and zebras grazing in a fields. In addition, there are five detected objects as giraffe with score 99.9%, zebra with score 99.7%, zebra with 99.9%, giraffe with score 97.1% and zebra with score 99.8%. I have generated bounding boxes as above image. I performed image classification, object detection and image captain on this image. Combining the predictions of nlpconnet/vit-gpt2-imagecaptioning, facebook/detr-resnet-101 and google/vit models, I get the results for you. Is it just that the in-context demonstrations are also ungrammatical and ChatGPT is copying them? It feels very unlike the way ChatGPT usually writes.
- nomel 4y agoIt's probably working from/stuck in some very sparse space, resulting in relatively poor output. It's a neural network after all. Telling it to fix its grammar, in a new thread fixes it. I also confidently assume that telling it to fix its grammar, within the original, topic specific, conversation, would noticeably harm the quality of subsequent output for that topic.
- zoba 4y agoThese sort of agent-architecture AIs are where I think things will head and also where things will get more dangerous. When an AI has a goal and an ability to break the goal down, and make progress towards the goal… the only thing stopping the AI from misalignment is whatever it’s creator has specified as the goal. Things get even more tricky when the agent can take in new information and deprioritize goals. I have been curious why OpenAI haven’t discussed Agent-based AIs.
- jazzyjackson 4y ago> the only thing stopping the AI from misalignment is whatever it’s creator has specified as the goal. And the API providers willingness/capability to provide the service being requested. I'm with those who don't see the danger, besides humans' own stupidity being amplified and someone trusting "AI" with something life critical, what should I be watching out for?
- amelius 4y agoWhere can I find a good, constantly-up-to-date overview of what is possible with AI?
- meghan_rain 4y agoYou're already here :-)
- amelius 4y agoAre you sure? There's a lot of overhyping here, and of course the reaction to that. Too much noise for me, tbh. Just give me the scientifically established facts.
- deleted 4y ago[deleted]
- roflyear 4y agoI agree with you
- roflyear 4y agoThis is an awfully biased and inaccurate source of what these models can do and what they will probably be able to do. HN is not full of experts here. It's fanboys.
- abrichr 4y agohttps://library.phygital.plus/ https://library.phygital.plus/
- oars 4y agoThanks for sharing this link to AI resources: https://library.phygital.plus/ https://library.phygital.plus/
- tracyhenry 4y agoReminds me of VisualChatGPT (https://github.com/microsoft/visual-chatgpt https://github.com/microsoft/visual-chatgpt), which also uses a LLM to decide what vision models to run.
- ed 4y agoHere’s the (empty, for now) GitHub repo - https://github.com/microsoft/JARVIS https://github.com/microsoft/JARVIS
- karmasimida 4y agoIt is possible to actually revolution the glue layer of computation in a lot of organizations. GhatGPT is the ultimate and last glue layer we will ever need.
- catchnear4321 4y agoA lot of LLM/ AI research feels like a big “duh.” The language model, being trained on essentially the primary way in which humanity communicates, might be a good means of managing integration of less language-focused models. …duh?
- baq 4y agoanimals which use tools: great apes (gorillas, chimpanzees, orangutans), monkeys, otters, dolphins, some bird species, octopus, and a few crocodilian species. now LLMs are on this list. a thing which isn't born, doesn't experience, can be copied and instantiated a million times, and a single improvement can be basically immediately propagated to all instances. a very unfamiliar thing with very familiar capabilities. so, technically, it was obvious. socially and psychologically, we'll be dealing with it for the rest of our civilization's lifetime.
- fatherzine 4y ago"We'll be dealing with it for the rest of our civilization's lifetime." Exactly.
- catchnear4321 4y agoThat’s a steal!
- roflyear 4y agoIf you think LLMs are an animal I'd hope you'd be cautious of enslaving them.
- nemo44x 4y agoI always wondered what it was like for people that lived through the discovery of relativity. It must have just been world changing to understand what it meant and how entire perspectives and theories were dead overnight. Just a massive change in the understanding of our existence and the potential it unlocked. These types of things must feel similar.
- imjonse 4y agothat was much more gradual and did not affect everyday life at all. Even today most people don't know/care about the implications of relativity :)
- arlcode 4y agoI don't think relativity had the same hype associated with it. I'm finding it very difficult to differentiate between (sometimes involuntary) sales people and genuine usefulness (of which there is plenty to be clear). It is yet unclear where the conceptual and fundamental limitations of the current approach lie and I'm immediately suspicious about anyone telling me there aren't any in particular when this sentiment is based on a few hours of GPT-4 usage. We need more hard science on this. Great claims require great proofs and "AGI is imminent" is a very very very great claim. We know that these tool are pretty useful as they are right now and will have massive influence on society.
- jacobrussell 4y ago> Therefore, in this paper, we propose a system called HuggingGPT to connect LLMs (i.e., ChatGPT) and ML community (i.e., HuggingFace), which can process inputs from different modalities and solve numerous complex AI tasks. More specifically, for each AI model in HuggingFace, we use its corresponding model description from the library and fuse it into the prompt to establish the connection with ChatGPT. Afterward, in our system, LLMs (i.e., ChatGPT) will act as the brain to determine the answers to the questions of users. I can already feel the HuggingFace ChatGPT plugin coming. The only problem is it would be very slow and use a bunch of tokens.
- simon_000666 4y agoChatGTP/4 is to AGI what pepper’s ghost is to holography. It’s a parlor trick, even if you add plugins or the ability to call other hugging face ML models - it’s just a parlor trick with fancier bells and whistles. All it is doing is using stochastic gradient descent to predict the next word in a sequence based on an enormous sophisticated training set designed to amaze people. Thinking it has advanced because it can now get calculations correct is a fallacy. It’s still just predicting the next word, it’s just that it’s now got a post processing step that is converting those next words into code and parroting the output. It maybe be able to now answer 4567*9876 correctly (using the human hardcoded wolfram alpha engine) but it still does not fundamentally comprehend why 1+1=2 - like my 5 year old can. Until it can generate its own internal neural networks to for example learn to logically reason about calculations we are still far from AGI. Also those calling for more data are misguided - less data, more sophisticated architectures than transformers are the only way to avoid the stochastic parrot trap.
- bjackman 4y agoI find this so bizarre. Every time someone demonstrates a new way in which models are capable of a wider array of tasks than expected someone goes "it's just predicting tokens". It's such a big "just". You are just firing neurons. The stock market is just supply and demand. The internet is just a bunch of computers talking through 50 year old protocols that don't work very well. Everything is just something else! I wonder if the first tribe to be annihilated by bronze weapons were like "that stuff is just like stone but more malleable, don't see what the big deal is".
- duanyy 4y ago[dead]
- duanyy 4y ago[dead]
- duanyy 4y ago[dead]
- duanyy 4y ago[dead]
- duanyy 4y agosay something