20 ms·

Hugging Face Releases Agents

214 points by mach1ne 3y ago

rahimnathwani 3y ago
If you want an overview, scroll down to this part of the page: https://huggingface.co/docs/transformers/transformers_agents#whats-happening-here-what-are-tools-and-what-are-agents https://huggingface.co/docs/transformers/transformers_agents... In short: - they've predefined a bunch of tools (e.g. image_generator) - the agent is an LLM (e.g. GPT-*) which is prompted with the name and spec of each tool (the same each time) and the task(s) you want to perform - the code generated by the agent is run by a python interpreter that has access to these tools
samstave 3y ago
Asking for help from those that are smarter than I am ;; - One of the very common things for Martial Arts Books in the past, was the fact that one were presented with a series of pics, along with some descriptions about what was being done in the pics. Sometimes, these are really hard to interpolate between frames, unless you had a much larger repetoir of movements based on experience (i.e. a white belt vs another higher belt... e.g. a green belt will have better context of movement than a white belt...) -- So can this be used to interpolate frames and digest lists (lists are what many martial arts count as documentation for their various arts... Many of these have been passed down via scrolls with either textual transmissions, paintings and then finally pics before vids existed... It would be really interesting to see if AI can interpret btwn images and or scroll text to be able to create an animation of said movements. --- For example, not only was Wally Jay one of my teachers, but as the inventor (re-discoverer) of Small Circle JuiJitsu - his pics are hard to infer what is happening... because there is a lot of nuanced feeling in each movement that is hard to convey via pics/text But if you can interpolate btwn frames, and model the movements, its game changing because through such interpolations on can imagine that you can get any angle of viewership -- and additionally, one can have the precise positioning and translucent display of bone/joint/muscle articulation such that one may provide for a deeper insight into the kinematics behind each movement.
- PaulHoule 3y ago
  Yeah, that reminds me reverse-engineering a form involving a Sansetsukon and a "spear".
  - samstave 3y ago
    Thats actually one of the harder weapons to master! Maybe next to the Kusari Gama... but I've only known one master of each.
- dr_kiszonka 3y ago
  I am certainly not smarter than you, especially in the context of LLMs and DL. I think existing DL models would have a tough time with such interpolation because 1) they don't seem to understand human anatomy, and 2) the space of all possible transitions is massive. I remember reading about human pose estimation algorithms[0], which would be a good first step. You could apply them to photos that you would like to interpolate between. I am not sure how you would train the interpolation model, though. Perhaps you could use OpenSim Models [1] in combination with reinforcement learning [2]? There is also some literature on pose forecasting [3, 4]. 0. Deep Learning-Based Human Pose Estimation: A Survey: https://github.com/zczcwh/DL-HPE https://github.com/zczcwh/DL-HPE 1. OpenSim: https://simtk.org/projects/opensim/ https://simtk.org/projects/opensim/ 2. BioImitation-Gym: http://umishra.me/bioimitation-gym/ http://umishra.me/bioimitation-gym/ 3. Human Pose Forecasting: https://paperswithcode.com/task/human-pose-forecasting https://paperswithcode.com/task/human-pose-forecasting 4. PoseGPT (name of the year!): https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136660409.pdf https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136...
  - samstave 3y ago
    >>"...because 1) they don't seem to understand human anatomy, and 2) the space of all possible transitions is massive..." I have often thought that we need an empirical-ish library of human movement/positions... we have a beginning small version with Ballet's positions and movements, but we dont have a necessarily precise dialogue for human positions common to every body, as opposed to just the athletic dancers. aside from maybe "Do the Robot!"
famouswaffles 3y ago
If a typical LLM has decent representation of the languages in question (and you'd be surprised how little decent is with all the positive transfer that goes on during training) then outsourcing translation is just a downgrade. a pretty big one in fact. https://github.com/ogkalu2/Human-parity-on-machine-translations https://github.com/ogkalu2/Human-parity-on-machine-translati... T5 seems to be the default so i get why it's done here. Just an observation.
- senko 3y ago
  Outsourcing everything but the reasoning process helps with preventing prompt injection attacks: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ Even if you're outsourcing to a restricted instance of the same model, it could be beneficial.
IAmStoxe 3y ago
This seems to be an interpretation similar to that of langchain.
PaulHoule 3y ago
Kinda what people are asking for, I mean people are really attracted to "describe a task" as opposed to "create a training set".
abidlabs 3y ago
Follow up Guide that explains how to create your own tools: https://huggingface.co/docs/transformers/custom_tools https://huggingface.co/docs/transformers/custom_tools
minimaxir 3y ago
From the documentation, HF Agents are much better explained than LangChain but not easier to use, and due to multimodality it may actually be more arcane to use.
- abidlabs 3y ago
  > due to multimodality it may actually be more arcane to use. Can you elaborate?
nico 3y ago
They also released today StarChat, their code model fine tuned as an assistant Might be good to try with CodeGPT, AutoGPT or BabyAGI
senko 3y ago
I've been thinking lately of the two tiered reasoner + tools architecture inspired by LangChain, simonw's writing[0] and this is right along those lines. We're trying too hard to have one model do it all. If we coordinate multiple models + other tools (ala ReAct pattern) we could make the systems more resistant to prompt injection (and possibly other) attacks and leverage their respective strengths and weaknesses. I'm a bit wary of tool invocation via python code instead of prompting the "reasoning" LLM to teach it about the special commands it can invoke. Python's a good crutch because LLMs know it reasonably well (I use a similar trick in my project, but I parse the resulting AST instead of running the untrusted code) so it's simpler to prompt them. In a few iterations I expect to see LLMs fine tuned to know about the standard toolset at their disposal (eg. huggingface default tools) and further refinement of the two-tiered pattern. [0] https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
- piloto_ciego 3y ago
  I’ve been thinking this way too. Our brains have different areas with different functions… so like, why wouldn’t a good AI too? Maybe an LLM for an internal monologue, maybe two or three to debate each other realistically, then a computer vision model to process visual input…
  - TeMPOraL 3y ago
    ... and perhaps a LLM that shares the latent space with the visual model, because those are apparently (and surprisingly) easily mapped to each other (at least per that one paper that popped up the other day). The bit that's missing is on-line learning. There's only so much you can keep bouncing around in working memory (context window of all the component models) - eventually you want to "fix" some of the context by altering the weights of the models (a kind of gradual fine-tuning?).
- green_man_lives 3y ago
  I am not sure if you've ever heard of Julian Jaynes, but you might be interested. His theory of the bicameral mind is pretty interesting. To bastardize it, the unconscious mind is a really good generalized searcher/pattern completer. You can set it loose and it will search some space or optimize some problem and return a solution. The conscious mind is a really good planner, organizer, supervisor, and filter. Basically the conscious mind will come up with a plan and keep track of all the things it has done and needs to do, while the unconscious chugs along solving the problems underneath. The conscious mind will choose to implement the things that the unconscious comes up with based on its discretion (slap that old woman! "Nope"). The conscious mind is good at this because it can sort of simulate the outcome of these "searches" and see what would happen "slapping that old lady would hurt her and get me arrested". So his model sort of sounds like an unrestricted LLM for the unconscious, with another, more restrictive LLM for the conscious, that has access to some sort of crazy deep Q-learning model that can simulate the outcomes of actions taken.
sudoapps 3y ago
As this LLM agent architecture continues to evolve and improve, we will probably see a lot of incredible products built on top of it.
NumberWangMan 3y ago
I'm not 100% sure that AGI is guaranteed to end humanity like Yudkowsky, but if that's the course we're on, seeing news like this is depressing. Can anyone legitimately argue that LLMs are safe because they don't have agency, when we just straight up give them agency? I know current-generation LLMs aren't really dangerous -- but is this not likely to happen over and over again as our machine intelligences get smarter and smarter? someone is going to give them the ability to affect the world. They won't even have to try to "get out of the box", because it'll have 2 sides missing. I'm getting more and more on board with "shut it all down" being the only course of action, because it seems like humanity needs all the safety margin we can get, to account for the ease at which anyone can deploy stuff like this. It's not clear alignment of a super-intelligence is even a solvable problem.
- oars 3y ago
  What is the definition of "agency" in this context?
  - NumberWangMan 3y ago
    Good point. I'm partially conflating the definition I usually mean, which is "having a goal in the world", with what they're doing, which is "having ability to affect the world". Hugging Face is trying to keep these locked down, and maybe being able to generate images and audible sound is not that much more dangerous than being able to output text. But it is increasing the attack surface for an AGI trying to get out of its box.
  - dotancohen 3y ago
    Perhaps, with internet access, these AI could open bank accounts (with plausible-enough forged ID - a task which AI excels at), then work on e.g. Fiver, then gamble on the stock market... Where they go from there is anybody's guess.
    - nullsense 3y ago
      I've been thinking about what the likely "minimal self-employable system" might look like and it struck me yesterday that it's very likely going to be something like a NSFW roleplay chatbot.
macrolime 3y ago
How does this compare to langchain agents?
ed 3y ago
Cool! The DX is tricky to nail, when combined with LLM's tendency to hallucinate. I asked it to extract some text from an image, which it dutifully tried to do. However the generated python kept throwing errors. There's no image -> text tool yet, so it was trying to use the image segmenter to generate a mask and somehow extract text from that. It would be super helpful to: 1) Have a complete list of available tools (and / or a copy of the entire prompt given to the LLM responsible for generating python). I used prompt injection to get a partial list of tools and checked the Github agent PR for the rest, but couldn't find `<<all_tools>>` since it gets generated at runtime (I think?). 2) Tell the LLM it's okay to fail. E.g.: "Extract the text from image `image`. If you are unable to do this using the tools provided, say so." This prompt let me know there's no tool for text extraction. Update: per https://huggingface.co/docs/transformers/custom_tools https://huggingface.co/docs/transformers/custom_tools you can output a full list of tools with `print(agent.toolbox)`
anton5mith2 3y ago
Could use LocalAI to get around this: “The openAI models perform better (but require you to have an openAI API key, so cannot be used for free);” https://www.reddit.com/r/selfhosted/comments/12w4p2f/localai_openai_compatible_api_to_run_llm_models/ https://www.reddit.com/r/selfhosted/comments/12w4p2f/localai...
- syntaxing 3y ago
  I was so excited until I saw that it's CPU based only. Would you happen to know of any alternative for GPU support, particularly GPTQ models? Edit: I think textgen itself can support this nowadays
  - anton5mith2 3y ago
    For now it is CPU only yes, uses AVX instructions. But it's pretty fast anyway, try it out. I have it running on my mbp M1 and it's pretty decent. I think GPU support will come eventually. I wrote an app that uses the openai API and it was nice and simple to just point it at my own local service instead.
syntaxing 3y ago
Whoa this is super awesome, kind of makes a ton of sense since HF pretty much dominates the market for model hosting and interfacing. The documentation actually looks about as complex as langchain. Gonna give it a go to query the docs with an agent to get an example (going full circle).
chaxor 3y ago
This is beautiful, but is there a decent way to plow through say, 20TB of text and put that into a vector database (encoder only)? It would be quite a great addition, especially if the vectors could then be translated into other forms (different language, json representation, pull out names/NER, etc) by just applying a decoder to the database.
bluepoint 3y ago
If you are like me and you tried to copy paste the python commands and it did not work, you need to generate an access token. Here is what you should do: 1. Sign up (https://huggingface.co/ https://huggingface.co/) to hugging face. 2. Setup access tokens (https://huggingface.co/settings/tokens https://huggingface.co/settings/tokens) 3. Install or Upgrade some dependencies `pip install huggingface_hub transformers accelerate` 4. From the terminal run `jupyter lab` 5. Then, if I did not forget any other dependencies you can just copy paste ```python from huggingface_hub import login from transformers import HfAgent login("hf_YOUR_HUGGING_FACE_TOKEN") agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder https://api-inference.huggingface.co/models/bigcode/starcode...") agent.run("Is the following `text` (in Spanish) positive or negative?", text="¡Este es un API muy agradable!") ```

214 points by mach1ne 3y ago

If you want an overview, scroll down to this part of the page: https://huggingface.co/docs/transformers/transformers_agents#whats-happening-here-what-are-tools-and-what-are-agents https://huggingface.co/docs/transformers/transformers_agents... In short: - they've predefined a bunch of tools (e.g. image_generator) - the agent is an LLM (e.g. GPT-*) which is prompted with the name and spec of each tool (the same each time) and the task(s) you want to perform - the code generated by the agent is run by a python interpreter that has access to these tools

samstave 3y ago

Asking for help from those that are smarter than I am ;; - One of the very common things for Martial Arts Books in the past, was the fact that one were presented with a series of pics, along with some descriptions about what was being done in the pics. Sometimes, these are really hard to interpolate between frames, unless you had a much larger repetoir of movements based on experience (i.e. a white belt vs another higher belt... e.g. a green belt will have better context of movement than a white belt...) -- So can this be used to interpolate frames and digest lists (lists are what many martial arts count as documentation for their various arts... Many of these have been passed down via scrolls with either textual transmissions, paintings and then finally pics before vids existed... It would be really interesting to see if AI can interpret btwn images and or scroll text to be able to create an animation of said movements. --- For example, not only was Wally Jay one of my teachers, but as the inventor (re-discoverer) of Small Circle JuiJitsu - his pics are hard to infer what is happening... because there is a lot of nuanced feeling in each movement that is hard to convey via pics/text But if you can interpolate btwn frames, and model the movements, its game changing because through such interpolations on can imagine that you can get any angle of viewership -- and additionally, one can have the precise positioning and translucent display of bone/joint/muscle articulation such that one may provide for a deeper insight into the kinematics behind each movement.

PaulHoule 3y ago
Yeah, that reminds me reverse-engineering a form involving a Sansetsukon and a "spear".
- samstave 3y ago
  Thats actually one of the harder weapons to master! Maybe next to the Kusari Gama... but I've only known one master of each.
dr_kiszonka 3y ago
I am certainly not smarter than you, especially in the context of LLMs and DL. I think existing DL models would have a tough time with such interpolation because 1) they don't seem to understand human anatomy, and 2) the space of all possible transitions is massive. I remember reading about human pose estimation algorithms[0], which would be a good first step. You could apply them to photos that you would like to interpolate between. I am not sure how you would train the interpolation model, though. Perhaps you could use OpenSim Models [1] in combination with reinforcement learning [2]? There is also some literature on pose forecasting [3, 4]. 0. Deep Learning-Based Human Pose Estimation: A Survey: https://github.com/zczcwh/DL-HPE https://github.com/zczcwh/DL-HPE 1. OpenSim: https://simtk.org/projects/opensim/ https://simtk.org/projects/opensim/ 2. BioImitation-Gym: http://umishra.me/bioimitation-gym/ http://umishra.me/bioimitation-gym/ 3. Human Pose Forecasting: https://paperswithcode.com/task/human-pose-forecasting https://paperswithcode.com/task/human-pose-forecasting 4. PoseGPT (name of the year!): https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136660409.pdf https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136...
- samstave 3y ago
  >>"...because 1) they don't seem to understand human anatomy, and 2) the space of all possible transitions is massive..." I have often thought that we need an empirical-ish library of human movement/positions... we have a beginning small version with Ballet's positions and movements, but we dont have a necessarily precise dialogue for human positions common to every body, as opposed to just the athletic dancers. aside from maybe "Do the Robot!"

famouswaffles 3y ago

If a typical LLM has decent representation of the languages in question (and you'd be surprised how little decent is with all the positive transfer that goes on during training) then outsourcing translation is just a downgrade. a pretty big one in fact. https://github.com/ogkalu2/Human-parity-on-machine-translations https://github.com/ogkalu2/Human-parity-on-machine-translati... T5 seems to be the default so i get why it's done here. Just an observation.

senko 3y ago
Outsourcing everything but the reasoning process helps with preventing prompt injection attacks: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ Even if you're outsourcing to a restricted instance of the same model, it could be beneficial.

IAmStoxe 3y ago

This seems to be an interpretation similar to that of langchain.

PaulHoule 3y ago

Kinda what people are asking for, I mean people are really attracted to "describe a task" as opposed to "create a training set".

abidlabs 3y ago

Follow up Guide that explains how to create your own tools: https://huggingface.co/docs/transformers/custom_tools https://huggingface.co/docs/transformers/custom_tools

minimaxir 3y ago

From the documentation, HF Agents are much better explained than LangChain but not easier to use, and due to multimodality it may actually be more arcane to use.

abidlabs 3y ago
> due to multimodality it may actually be more arcane to use. Can you elaborate?

nico 3y ago

They also released today StarChat, their code model fine tuned as an assistant Might be good to try with CodeGPT, AutoGPT or BabyAGI

senko 3y ago

I've been thinking lately of the two tiered reasoner + tools architecture inspired by LangChain, simonw's writing[0] and this is right along those lines. We're trying too hard to have one model do it all. If we coordinate multiple models + other tools (ala ReAct pattern) we could make the systems more resistant to prompt injection (and possibly other) attacks and leverage their respective strengths and weaknesses. I'm a bit wary of tool invocation via python code instead of prompting the "reasoning" LLM to teach it about the special commands it can invoke. Python's a good crutch because LLMs know it reasonably well (I use a similar trick in my project, but I parse the resulting AST instead of running the untrusted code) so it's simpler to prompt them. In a few iterations I expect to see LLMs fine tuned to know about the standard toolset at their disposal (eg. huggingface default tools) and further refinement of the two-tiered pattern. [0] https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ https://simonwillison.net/2023/Apr/25/dual-llm-pattern/

piloto_ciego 3y ago
I’ve been thinking this way too. Our brains have different areas with different functions… so like, why wouldn’t a good AI too? Maybe an LLM for an internal monologue, maybe two or three to debate each other realistically, then a computer vision model to process visual input…
- TeMPOraL 3y ago
  ... and perhaps a LLM that shares the latent space with the visual model, because those are apparently (and surprisingly) easily mapped to each other (at least per that one paper that popped up the other day). The bit that's missing is on-line learning. There's only so much you can keep bouncing around in working memory (context window of all the component models) - eventually you want to "fix" some of the context by altering the weights of the models (a kind of gradual fine-tuning?).
green_man_lives 3y ago
I am not sure if you've ever heard of Julian Jaynes, but you might be interested. His theory of the bicameral mind is pretty interesting. To bastardize it, the unconscious mind is a really good generalized searcher/pattern completer. You can set it loose and it will search some space or optimize some problem and return a solution. The conscious mind is a really good planner, organizer, supervisor, and filter. Basically the conscious mind will come up with a plan and keep track of all the things it has done and needs to do, while the unconscious chugs along solving the problems underneath. The conscious mind will choose to implement the things that the unconscious comes up with based on its discretion (slap that old woman! "Nope"). The conscious mind is good at this because it can sort of simulate the outcome of these "searches" and see what would happen "slapping that old lady would hurt her and get me arrested". So his model sort of sounds like an unrestricted LLM for the unconscious, with another, more restrictive LLM for the conscious, that has access to some sort of crazy deep Q-learning model that can simulate the outcomes of actions taken.

sudoapps 3y ago

As this LLM agent architecture continues to evolve and improve, we will probably see a lot of incredible products built on top of it.

NumberWangMan 3y ago

I'm not 100% sure that AGI is guaranteed to end humanity like Yudkowsky, but if that's the course we're on, seeing news like this is depressing. Can anyone legitimately argue that LLMs are safe because they don't have agency, when we just straight up give them agency? I know current-generation LLMs aren't really dangerous -- but is this not likely to happen over and over again as our machine intelligences get smarter and smarter? someone is going to give them the ability to affect the world. They won't even have to try to "get out of the box", because it'll have 2 sides missing. I'm getting more and more on board with "shut it all down" being the only course of action, because it seems like humanity needs all the safety margin we can get, to account for the ease at which anyone can deploy stuff like this. It's not clear alignment of a super-intelligence is even a solvable problem.

oars 3y ago
What is the definition of "agency" in this context?
- NumberWangMan 3y ago
  Good point. I'm partially conflating the definition I usually mean, which is "having a goal in the world", with what they're doing, which is "having ability to affect the world". Hugging Face is trying to keep these locked down, and maybe being able to generate images and audible sound is not that much more dangerous than being able to output text. But it is increasing the attack surface for an AGI trying to get out of its box.
- dotancohen 3y ago
  Perhaps, with internet access, these AI could open bank accounts (with plausible-enough forged ID - a task which AI excels at), then work on e.g. Fiver, then gamble on the stock market... Where they go from there is anybody's guess.
  - nullsense 3y ago
    I've been thinking about what the likely "minimal self-employable system" might look like and it struck me yesterday that it's very likely going to be something like a NSFW roleplay chatbot.

macrolime 3y ago

How does this compare to langchain agents?

ed 3y ago

Cool! The DX is tricky to nail, when combined with LLM's tendency to hallucinate. I asked it to extract some text from an image, which it dutifully tried to do. However the generated python kept throwing errors. There's no image -> text tool yet, so it was trying to use the image segmenter to generate a mask and somehow extract text from that. It would be super helpful to: 1) Have a complete list of available tools (and / or a copy of the entire prompt given to the LLM responsible for generating python). I used prompt injection to get a partial list of tools and checked the Github agent PR for the rest, but couldn't find `<<all_tools>>` since it gets generated at runtime (I think?). 2) Tell the LLM it's okay to fail. E.g.: "Extract the text from image `image`. If you are unable to do this using the tools provided, say so." This prompt let me know there's no tool for text extraction. Update: per https://huggingface.co/docs/transformers/custom_tools https://huggingface.co/docs/transformers/custom_tools you can output a full list of tools with `print(agent.toolbox)`

anton5mith2 3y ago

Could use LocalAI to get around this: “The openAI models perform better (but require you to have an openAI API key, so cannot be used for free);” https://www.reddit.com/r/selfhosted/comments/12w4p2f/localai_openai_compatible_api_to_run_llm_models/ https://www.reddit.com/r/selfhosted/comments/12w4p2f/localai...

syntaxing 3y ago
I was so excited until I saw that it's CPU based only. Would you happen to know of any alternative for GPU support, particularly GPTQ models? Edit: I think textgen itself can support this nowadays
- anton5mith2 3y ago
  For now it is CPU only yes, uses AVX instructions. But it's pretty fast anyway, try it out. I have it running on my mbp M1 and it's pretty decent. I think GPU support will come eventually. I wrote an app that uses the openai API and it was nice and simple to just point it at my own local service instead.

syntaxing 3y ago

Whoa this is super awesome, kind of makes a ton of sense since HF pretty much dominates the market for model hosting and interfacing. The documentation actually looks about as complex as langchain. Gonna give it a go to query the docs with an agent to get an example (going full circle).

chaxor 3y ago

This is beautiful, but is there a decent way to plow through say, 20TB of text and put that into a vector database (encoder only)? It would be quite a great addition, especially if the vectors could then be translated into other forms (different language, json representation, pull out names/NER, etc) by just applying a decoder to the database.

bluepoint 3y ago

If you are like me and you tried to copy paste the python commands and it did not work, you need to generate an access token. Here is what you should do: 1. Sign up (https://huggingface.co/ https://huggingface.co/) to hugging face. 2. Setup access tokens (https://huggingface.co/settings/tokens https://huggingface.co/settings/tokens) 3. Install or Upgrade some dependencies `pip install huggingface_hub transformers accelerate` 4. From the terminal run `jupyter lab` 5. Then, if I did not forget any other dependencies you can just copy paste ```python from huggingface_hub import login from transformers import HfAgent login("hf_YOUR_HUGGING_FACE_TOKEN") agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder https://api-inference.huggingface.co/models/bigcode/starcode...") agent.run("Is the following `text` (in Spanish) positive or negative?", text="¡Este es un API muy agradable!") ```