12 ms·
Introducing Agents in Haystack: Make LLMs resolve complex tasks
- deleted 4y ago[deleted]
- j0e1 4y agoCould someone in the know compare this with LangChain (https://github.com/hwchase17/langchain https://github.com/hwchase17/langchain)?
- teruakohatu 4y agoLooks like the same general idea: https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack
- antti909 4y agoSee above - Haystack started a few years ago as a result of us working with some large enterprise clients on implementing extractive QA at scale. Now evolving to also allow the backend builders to mimic what's available from, e.g. OpenAI+plugins, but with their own set of models, and being able to mix&match best available components and technology.
- nextworddev 4y agoMost of the core ideas came from a paper called React, they all kind of riff on the idea of self-inspection / introspection to augment the context or plan action
- antti909 4y agoFor the Agents? Yes, indeed. Referred in the article.
- ukuina 4y agoI would consider Haystack to be the more batteries-included, easier to use (but harder to customize) of the two. They have a good emphasis on local model use.
- antti909 4y agoThanks :) Working on it. Re local models - indeed, all started with using the Transformer models for extractive QA and semantic search. With the Promptnode, and/or the Agents it's also now possible to combine local models/pipelines & 'LLMs' freely.
- antti909 4y agoLangChain is very cool tho :))
- antti909 4y agoHaystack has been around for a while now, and we've been mostly specializing in the extractive QA. The focus has been indeed on making the use of local Transformer models most easy and convenient for a backend application builder. You can build very reliable and sometimes quite elaborate NLP pipelines with Haystack (e.g., extractive or generative QA, summarization, document similarity, semantic search, FAQ-style search, etc. etc.) with either Transformer models, LLMs, or both. With the Agents you can also put an Agent on top of your pipelines and use a prompt-defined control to find the best underlying tool and pipeline for the task. Haystack has always included all the necessary 'infrastructure' components - pre-processing, indexing, several document stores to choose from (ES/OS, Pinecone, Weavite, Milvus, now Qdrant, etc.) and the means to evaluate and fine-tune Transformer models.
- d4rkp4ttern 4y agoThanks for clarifying. The support for local LLMs seems very interesting — would a haystack agent call out to a separately “running” self-hosted LLM via an API (REST, etc) or would it need to actually load up the model and directly query it (e.g model.generate(<prompt>) ) ? Also it seems like the functionality of haystack subsumes those of langchain and llama-index (fka GPT-index) ?
- antti909 4y agoTo be precise - I don't think I'm saying 'local LLMs' above :) But technically possible, I guess, just hasn't been part of what's officially available. (There are also licensing issues still.) To answer your question about the APIs - the Agent itself queries OpenAI via REST to break the prompt down into tasks, then works with the underlying tools/pipelines using Python API (and then, e.g., a Transformer model that's part of the pipeline has to be 'loaded' into a GPU). Part of those pipelines might be using Promptnode (that can work with hosted LLMs via REST, but could also work with a local LLM). Re 'subsume' - well, that depends :) But arguably, you can build an NLP Python backend with Haystack only, of course.. Regardless of how complex your underlying use case is, or whether it's extractive, generative or both.
- tholor 4y ago
- brofallon 4y agoSomewhat OT, but I feel like this is such an underappreciated aspect of recent LLMs: Not just their ability to generate text - but their apparent effectiveness in making use of arbitrary tools to interact with their environment to achieve some goal. It seems like we're just at the beginning of this with the MRKL and ReAct papers, there will be a ton more awesomeness coming in this area I'm sure.
- bjackman 4y agoYes I think if this works it's one of the strongest signals yet that LLMs as they currently exist have a fairly general form of intelligence. I am pretty sympathetic to the field of "AI Safety" and I worry a lot about the implications of agent-like general intelligences. This post gives me a lot to ponder. What are the implications of the fact that even AIs that are not agent-like at heart can apparently be told "please simulate an agent-like AI"? I really don't know. Should we consider it as an "inner AI" with its own objectives? How can we determine what those objectives are? Instinctively it feels much less concerning than an AI with a direct action->perception feedback loop but who knows. AI is fucking weird. What a thrilling time to be alive!
- jcims 4y agoFrom the Github project readme >Agent: (since 1.15) An Agent is a component that is powered by an LLM, such as GPT-3. It can decide on the next best course of action so as to get to the result of a query. It uses the Tools available to it to achieve this. While a pipeline has a clear start and end, an Agent is able to decide whether the query has resolved or not. It may also make use of a Pipeline as a Tool. Emphasis mine. Having tinkered with LangChain I think the idea of a recursive and/or graph-oriented model is going to yield interesting phenomena in the overall feel of these language models. LangChain agents are already super impressive.
- Art9681 4y agoI've been tinkering with LangChain for a few days and I agree. Is there a resource that collects the agents so we can experiment? I'd love to see an aggregated list of the most impressive agents and use cases if anyone knows of any. This stuff is the future of computing no doubt.
- nico 4y agoThere is a LangChainHub mentioned in their docs, but the repo for it seems dead. Is there any sort of marketplace/AppStore for agents/tools/plug-ins for LLMs via LangChain? Or some other library like haystack?
- fire 4y agoclosest i've seen so far is https://llamahub.ai https://llamahub.ai, but afaict it's only for loaders atm
- deleted 4y ago[deleted]
- antti909 4y agoThanks for the emphasis :) Accurate!
- lsy 4y agoIs there some way of holding the LLM response to a given prompt constant? It sounds like a lot of this relies on the LLM getting the right answer in sequence, so I'm guessing they do something like keep the temperature at 0? Otherwise you are going to wind up with possibly different behavior run-to-run. And even if they do have something like the above, don't we end up with potentially breaking changes once models are updated? Basically the issue is that even if you can guarantee response format X for prompt A, a slightly modified prompt A' has no guarantee that its response will be in the same format as X, even in the same model. You can also imagine that the more "Tools" are available, the lower the chance that the model will pick the right one based on its English text description. Would be interesting to know how this is being addressed.
- amrrs 4y agoThere is a library called guardrails . I've not played with it extensively but that seems to address issues like this one.
- Cyphase 4y agohttps://shreyar.github.io/guardrails/ https://shreyar.github.io/guardrails/ https://github.com/shreyar/guardrails https://github.com/shreyar/guardrails
- Ozzie_osman 4y agoEven with temperature 0 and the same model, the models are slightly non-deterministic and may diverge with the same input. You probably don't want to treat them as deterministic (at least, not now). But there are many applications where slightly non-deterministic behavior is OK.
- computerex 4y agoThe output is stochastic, so if the response doesn't decode to your format you can try the request again. Gpt3.5-turbo is pretty good at tool selection and use, but it often messes up with difficult tasks. gpt-4 is on another level when it comes to tool use. It is very reliable in my testing. You ofc can't guarantee the output so defensive programming, retries are a must in my opinion. We are all learning how to work with this technology.
- yosito 4y agoI haven't yet figured out how to get an LLM to accurately determine whether it actually knows something or is making it up. I wonder how they handle that. They may get to that at some point in the article, but the page eventually breaks for me on mobile and I can't read past the first code block.
- djtriptych 4y agoi agree. LLMs are not built for structured reasoning or even citations.
- ramraj07 4y agoGpt-4 does a reasonable job citing things. It can’t cite every paper out there but definitely the well cited ones.
- PostOnce 4y agoDoes it cite papers that don't exist, or cite papers when the paper it cites doesn't actually contain the information being cited? I would bet it does, at least some percent of the time.
- yosito 4y agoI've been trying to get GPT-4 to give me accurate links to predictable websites. It gives me very plausible links, that even have the right domain and path format but often the plausible link is not the correct link and GPT-4 seems to have no awareness of the correct link.
- ramraj07 4y agoThe latter, yes. Interestingly I’m not surprised at all. This is what many researchers themselves do lol. I never take a reference at face value from any human being and I apply the same standard to gpt-4 as well. But all its references are real. Just 20-40% of time it might not exactly say the same as what I asked it for (though it’s related, and mostly there).
- Art9681 4y agoAre there any projects similar to this that use other languages? I don't have anything against Python. But I would prefer using Go if there are alternatives. I experimented with LangChain and this appears to be a similar idea but they are all based on iPython notebooks and that ecosystem.
- ukuina 4y agoPython seems the go-to language for most ML work. Are there any wrappers you could use to call these frameworks from within Golang?
- birdiesanders 4y agoThat's the heartbreak in ml dev, an entire company of golang code, no way to avoid tossing in a pile of python I don't want to look at.
- arthurcolle 4y agoDoesn't this remind anyone else of Bitcoin autonomous agents? https://en.bitcoin.it/wiki/Agent#:~:text=An%20agent%20is%20an%20autonomous,additional%20instances%20on%20other%20servers https://en.bitcoin.it/wiki/Agent#:~:text=An%20agent%20is%20a.... It would be interesting if you could somehow combine a wallet plugin to give the GPT something to work to increase. That + that AutoGPT repo from a day ago + maybe Mitogen for self-replication might be a cool combo to really kick off a fun "dumb" AI on the loose
- alangou 4y agoThis is cool. Some of the weaknesses they point out in software agents can be obviated with the latest AI advancements—give a chatbot the tools of the courts, and maybe the autonomous agent won’t be at a disadvantage in scams anymore. Interesting to see if an AI agent given the objective to generate profit would resort to running scams. Encoding moral boundaries becomes important. What if you asked it to raise money for a good cause and it uses “whatever means necessary?”
- Bjartr 4y ago> give a chatbot the tools of the courts, and maybe the autonomous agent won’t be at a disadvantage in scams anymore. And now we have the beginnings to a plot point from Accelerando. Self propelled AI businesses as a distinct entity.
- arthurcolle 4y agoIf anyone wants to start hacking on this, I just started working on this today. What I am having trouble understanding is how do I have the "agent" (a GPT-like LLM) maintain it's own "state" (balances, accounts receivable, accounts payable, etc) - would I just run a webservice and then have a ChatGPT Plugin access the "agent state" via a webhook? Is it as simple as that? Or would it just be like a recursive function where you continue to pass in your state with every "iteration" I started working on a design last night and I think this is a really cool idea. All the parts are basically "there" EDIT: I asked ChatGPT-4 what it thinks. It seems to like the idea... *thinking emoji* perhaps it seeks a human acolyte to spread its gospel... Arthur Collé: I want to combine the concept of Bitcoin agents (autonomous programs) with GPT-4 and mitogen, to be able to distribute complex AI agents throughout the internet ChatGPT-4: Combining Bitcoin agents, GPT-4, and Mitogen to create and distribute complex AI agents throughout the internet is an ambitious and innovative idea. Here's a high-level outline to guide you through this process: Define the purpose and functionality of the AI agents: Before proceeding, clearly define the purpose of these AI agents. Will they serve as decentralized oracles, provide AI-driven services, or facilitate complex transactions on the blockchain? Understanding their purpose will help you design and develop the agents more efficiently. Develop the AI agents using GPT-4: Design and train your AI agents using GPT-4 to achieve the desired functionality. Depending on the complexity of the agents, you might need to fine-tune the GPT-4 model to tailor it to your specific use case. Be mindful of the computational resources required to run the AI agents, as this will impact their performance in a decentralized environment. Integrate Bitcoin agents: Incorporate the concept of Bitcoin agents, or autonomous programs that interact with the Bitcoin blockchain, into your AI agents. This will enable your agents to autonomously perform tasks such as sending and receiving payments, signing transactions, and updating their internal state based on blockchain data. You can use existing libraries and tools for this purpose, such as bitcoinj or BTCPay Server. Utilize Mitogen for distribution and communication: Mitogen is an Ansible extension that simplifies distributed computing by enabling communication between remote Python interpreters over SSH or other transports. Use Mitogen to facilitate communication between your AI agents and distribute them across various nodes on the internet. Ensure that the AI agents can communicate effectively and perform their tasks even in a decentralized setting. Implement security measures: Securing your AI agents is crucial, as they will be handling sensitive information and potentially financial transactions. Implement strong encryption, authentication, and access controls to protect the integrity and confidentiality of your agents. Testing and deployment: Thoroughly test your AI agents in a controlled environment to ensure they perform as expected. Once you are satisfied with their functionality, deploy them throughout the internet using the Mitogen framework. Monitor and maintain: Continuously monitor the performance and security of your AI agents. Periodically update the GPT-4 models to maintain their accuracy and relevance. Be prepared to address any issues or vulnerabilities that may arise. Keep in mind that this is a complex project with many potential challenges, including security, scalability, and regulatory compliance. Additionally, the ethical implications of creating and distributing autonomous AI agents should be carefully considered. Nonetheless, if executed properly, this concept has the potential to unlock innovative use cases and services in the realm of AI and blockchain technology.
- Ozzie_osman 4y agoIf you (like me) were wondering how these works, the LLM is given a prompt like: Answer the following questions as best you can. You have access to the following tools: Search: Use this to search the internet. Calculator: Use this to do math. Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of [{tool_names}] Action Input: the input to the action Observation: the result of the action ... (this Thought/Action/Action Input/Observation can repeat N times) Thought: I now know the final answer Final Answer: the final answer to the original input question Question: What is the age of the president of Egypt squared? Thought: To which the LLM will generate a completion like: Thought: I need to find the age of the president of Egypt. Action: Search Action Input: Age of president of Egypt Observation: At which point, the code (langchain, haystack, etc) will parse out the requested tool (Search) and input (Age of president of Egypt), and then call the right tool or API, then append the output of that action into the prompt. This all happens in a loop, at each step, the LLM is given the entire past prompt history, and given the opportunity to do a completion to choose the next tool and input to the next tool, after which the code parses those out, executes the tool, and repeats until the LLM decides it has the final answer and returns.
- Ozzie_osman 4y ago(also tbc I took these example prompts from LangChain.. not sure if Haystack uses different prompts (LangChain actually has a bunch of versions, this is probably the easiest one)
- ryanwaggoner 4y agoI did a manual version of this where I played a dispatch controller in a robot, relaying inputs and outputs from GPT4, which I told was the reasoning brain in this robot. It was very remarkable to watch its train of thought in considering sensor inputs and then giving me actions to take in response.
- sr-latch 4y agoThis looks similar to the WebGPT paper, is that referenced in any of langchain or haystack's publications? Introducing the mechanism of internal thought is very interesting, I wonder if there's a way to make it implicit in the model's architecture.
- Vecr 4y agoHas anyone had experience with text-davinci-003 vs Code-DaVinci-002? Apparently code-davinci-002 is better at statistical reasoning, as RTHF/fine tuning has made the text versions as well as the later models as bad as humans are or worse. If this is true code-davinci-002 is probably the most competent model available that could form the basis of a reasonably rational system, using a chaining or step-by-step DAG method similar to the submitted article.
- eob 4y agoAnyone have requests for an agent framework that adds some of the capabilities Yohei on Twitter has been tinkering with? Longer term planning, memory, etc?
- dovlex 4y agoHey, I work on Haystack Agents; we haven't seen Yohei's types of requests yet but I'm closely following his ideas and work.
- bottlepalm 4y agoComputer viruses in the future really will be ‘ghosts in the machine’
- ActorNightly 4y agoSure, lets build more and more applications that all have a single point of failure that is Open AI.
- thomashop 4y agoThe framework let's you plugin many different LLMs not just OpenAI's
- antti909 4y agoVery accurate observation :) So basically, a bit more freedom in picking the right tools for the job, connecting an LLM to proprietary data in a safe way, using multiple models simultaneously, and leveraging custom extractive/generative pipelines.
- vrglvrglvrgl 4y ago[dead]
- anonu 4y agoI wonder on the implications for portfolio management. Experts could be optimizers with historical pricing.
- antti909 4y agoThanks for the spotlight :) We've spent quite a lot of time working on the Agents lately, and it's definitely a big focus. Couple of extra points to reflect on some of the comments here. It's quite straightforward to build a hybrid NLP backend with Haystack combining either hosted LLM (e.g., OpenAI or Cohere), or local, smaller Transformer models, or both. Agents add another level of control on top of that, as described in the article and in the comments. This provides more flexibility wrt bridging it to the relevant data and extract/generate accurate non-hallucinatory answers. Join our Discord too :) https://haystack.deepset.ai/community https://haystack.deepset.ai/community
- d4rkp4ttern 4y agoHaystack looks very interesting, just found out about it today :) Is there some overlap with the functionality of langchain? Could you highlight some differences? Thanks
- antti909 4y agoThanks :) Answered a similar one somewhere else here - looks like you've found it already. Feel free to ask more in Discord https://haystack.deepset.ai/community https://haystack.deepset.ai/community
- tuanacelik 4y agoHey, thanks for posting our article. I see a few comments here about memory: This is indeed a challenge which we're working on. And some other topics at hand are making oss models available for our implementation of the agent as well
- kacperlukawski 4y agoThat's a great news! Especially since we implemented the integration between Haystack and Qdrant: https://github.com/qdrant/qdrant-haystack/ https://github.com/qdrant/qdrant-haystack/
- andre-z 4y agoGreat move! Congrats to the deepset team! Glad there is now a Qdrant integration in place to power large-scale vector search needs.