6 ms·

Enabling the 'terminal' and 'python-repl' tools in a langchain agent demonstrates some pretty remarkable behavior. The link below is the transcript of a sessio

by jcims 4y ago

Enabling the 'terminal' and 'python-repl' tools in a langchain agent demonstrates some pretty remarkable behavior. The link below is the transcript of a session in which I asked the agent to create a hello world script and executes it. The only input I provide is on line 17. Everything else is the langchain agent iteratively taking an action, observing the results and deciding the next action to take. This is just scratching the surface. I've seen it do some crazy stuff with the AWS CLI. And this is just with GPT-3.5, I don't have access to GPT-4 yet and it clearly has better capabilities. https://pastebin.com/qJsbufVj https://pastebin.com/qJsbufVj

deleted 4y ago
[deleted]
robbintt 4y ago
What is the aws cli stuff? I was thinking about writing a terraform agent tool.
- jcims 4y ago
  You install and configure the CLI to run locally, then ask it to do something. For example, ask it to create a website using s3 static website feature. It creates the bucket, creates the content, uploads the content, configures the bucket static website features and configures the permissions on the bucket and content. I just started tinkering with terraform, which it seems to understand fairly well.
antibasilisk 4y ago
That pastebin is mindboggling.
- badloginagain 4y ago
  I'm glad our AI overlords are as bamboozled by python vs. python3 as us lowly humans
- roflyear 4y ago
  How so?
  - antibasilisk 4y ago
    it worked through a software problem and came to a conclusion based on the computer's feedback
    - flangola7 4y ago
      Wait until it's directed to hack into power grids!
    - ChatGTP 4y ago
      I feel like it brute forced an answer? I actually think that this could've been achieved with Stack Overflow Search API?
      - antibasilisk 4y ago
        It looks to me like something a junior developer would do. It's not pure brute force
        ChatGTP 4y ago
        Looking like something is different to doing what a junior developer would do. I admit they'd probably look through a database of known existing answers...not disputing that. I'd say that a person using Linux for the first time might do what is happening in that demo. Anyway, not to diminish the bot or LLMs or whatever, but it's definitely something that could be done using the Stack Overflow Search API or similar. I'd say it's actually a very primitive / boring use of an LLM.
px43 4y ago
Can you share the code for llm.py?
- jcims 4y ago
  #!/home/ubuntu/venv/bin/python3.10 from langchain.agents import load_tools from langchain.agents import initialize_agent from langchain.chat_models import ChatOpenAI llm = ChatOpenAI(model='gpt-3.5-turbo',temperature=0) tools = load_tools(['python_repl', 'requests', 'terminal', 'wolfram-alpha', 'serpapi', 'wikipedia', 'human', 'pal-math', 'pal-colored-objects'], llm=llm) agent = initialize_agent(tools, llm, agent="chat-zero-shot-react-description", verbose=True) agent.run("Ask the human what they want to do") Note that you'll need to get api keys for openai and serpapi and a app id from wolfram-alpha.
  - SCUSKU 4y ago
    People are joking that they can't wait for the next big thing in AI to come out in a few weeks, but this seems pretty big. After fiddling with it for a while, its not perfect, but this isn't that far from being able to replace (or at least dramatically change) my job as a software engineer. For example, I asked it to write conway's game of life, and it took about 4-5 attempts but it wrote fully functional code that popped up a matplotlib window with a fully functional simulation. This would've taken me a day at least. I asked it to write a FastAPI backend that uses SQLite for storing blog Posts, and it struggled with that one a lot and couldn't quite get it right, although I think that's largely a limitation of the python REPL from langchain as opposed to GPT. On the one hand I'm excited to build all sorts of new things and projects with this, but on the other hand I'm worried my standard of living will decline because my skills will become super commodified :/
fudged71 4y ago
About how long did this take to run? It would be great if it also summarized what the error was, what was the fix, and how to run the code that it created. That’s all in the output but could be pulled out at the end.
- jcims 4y ago
  A minute or two. You can ask it to summarize things if you like. It sometimes forgets to do so, however.
ArenaSource 4y ago
If you put GPT-4 on a loop with access to the shell it manages to do whatever is needed to finish the job https://raw.githubusercontent.com/jla/gpt-shell/assets/examples/any-open-ports.png https://raw.githubusercontent.com/jla/gpt-shell/assets/examp...
- jcims 4y ago
  Yeah I can't wait to get API access to gpt-4, it is a stepwise more capable based on the stuff I've done with chatgpt on gpt-4. That said, even gpt-3.5 will try multiple routes to get to the same endpoint. It seems to get distracted pretty easily though.
  - ArenaSource 4y ago
    > It seems to get distracted pretty easily though. That’s true, gpt-4 is way more easy to guide with the system messages and it doesn’t forget the instructions as the conversation goes on.
  - hombre_fatal 4y ago
    One demo of gpt-4’s superiority over gpt-3 is to come up with a prompt that determines the language of some given text. I couldn’t figure out a gpt-3 prompt that could handle “This text is written in French” correctly (it thinks it’s written in French), but with gpt-4 you can include in the prompt to disregard what the text says and focus on the words and grammar that it uses.
- hellcow 4y ago
  My experience with GPT-4 has been really disappointing. It didn't feel like a step up from 3.5. As an example, I've been trying to use it to learn Zig since the official docs are ... spartan. And I've said, "here's my code, here's the error, what's wrong with it?" and it will go completely off the rails suggesting fixes that don't do anything (or are themselves wrong). In my case, understanding/fixing the code would have required GPT-4 to know the difference between allocating on the stack/heap and the lifetimes of pointers. It never even approached the right solution. I haven't yet gotten it to help me in even a single instance. Every suggestion is wrong or won't compile, and it can't reason through the errors iteratively to find a fix. I'm sure this has to do with a small sample of Zig code in its training set, but I reckon an expert C coder could have spotted the bug instantly.
roflyear 4y ago
And you only used the most popular example of getting started and one of if not the most popular programming languages. Amazing.

by jcims 4y ago

deleted 4y ago

[deleted]

robbintt 4y ago

What is the aws cli stuff? I was thinking about writing a terraform agent tool.

jcims 4y ago
You install and configure the CLI to run locally, then ask it to do something. For example, ask it to create a website using s3 static website feature. It creates the bucket, creates the content, uploads the content, configures the bucket static website features and configures the permissions on the bucket and content. I just started tinkering with terraform, which it seems to understand fairly well.

antibasilisk 4y ago

That pastebin is mindboggling.

badloginagain 4y ago
I'm glad our AI overlords are as bamboozled by python vs. python3 as us lowly humans
roflyear 4y ago
How so?
- antibasilisk 4y ago
  it worked through a software problem and came to a conclusion based on the computer's feedback
  - flangola7 4y ago
    Wait until it's directed to hack into power grids!
  - ChatGTP 4y ago
    I feel like it brute forced an answer? I actually think that this could've been achieved with Stack Overflow Search API?
    - antibasilisk 4y ago
      It looks to me like something a junior developer would do. It's not pure brute force
      - ChatGTP 4y ago
        Looking like something is different to doing what a junior developer would do. I admit they'd probably look through a database of known existing answers...not disputing that. I'd say that a person using Linux for the first time might do what is happening in that demo. Anyway, not to diminish the bot or LLMs or whatever, but it's definitely something that could be done using the Stack Overflow Search API or similar. I'd say it's actually a very primitive / boring use of an LLM.

px43 4y ago

Can you share the code for llm.py?

jcims 4y ago
#!/home/ubuntu/venv/bin/python3.10 from langchain.agents import load_tools from langchain.agents import initialize_agent from langchain.chat_models import ChatOpenAI llm = ChatOpenAI(model='gpt-3.5-turbo',temperature=0) tools = load_tools(['python_repl', 'requests', 'terminal', 'wolfram-alpha', 'serpapi', 'wikipedia', 'human', 'pal-math', 'pal-colored-objects'], llm=llm) agent = initialize_agent(tools, llm, agent="chat-zero-shot-react-description", verbose=True) agent.run("Ask the human what they want to do") Note that you'll need to get api keys for openai and serpapi and a app id from wolfram-alpha.
- SCUSKU 4y ago
  People are joking that they can't wait for the next big thing in AI to come out in a few weeks, but this seems pretty big. After fiddling with it for a while, its not perfect, but this isn't that far from being able to replace (or at least dramatically change) my job as a software engineer. For example, I asked it to write conway's game of life, and it took about 4-5 attempts but it wrote fully functional code that popped up a matplotlib window with a fully functional simulation. This would've taken me a day at least. I asked it to write a FastAPI backend that uses SQLite for storing blog Posts, and it struggled with that one a lot and couldn't quite get it right, although I think that's largely a limitation of the python REPL from langchain as opposed to GPT. On the one hand I'm excited to build all sorts of new things and projects with this, but on the other hand I'm worried my standard of living will decline because my skills will become super commodified :/

fudged71 4y ago

About how long did this take to run? It would be great if it also summarized what the error was, what was the fix, and how to run the code that it created. That’s all in the output but could be pulled out at the end.

jcims 4y ago
A minute or two. You can ask it to summarize things if you like. It sometimes forgets to do so, however.

ArenaSource 4y ago

If you put GPT-4 on a loop with access to the shell it manages to do whatever is needed to finish the job https://raw.githubusercontent.com/jla/gpt-shell/assets/examples/any-open-ports.png https://raw.githubusercontent.com/jla/gpt-shell/assets/examp...

jcims 4y ago
Yeah I can't wait to get API access to gpt-4, it is a stepwise more capable based on the stuff I've done with chatgpt on gpt-4. That said, even gpt-3.5 will try multiple routes to get to the same endpoint. It seems to get distracted pretty easily though.
- ArenaSource 4y ago
  > It seems to get distracted pretty easily though. That’s true, gpt-4 is way more easy to guide with the system messages and it doesn’t forget the instructions as the conversation goes on.
- hombre_fatal 4y ago
  One demo of gpt-4’s superiority over gpt-3 is to come up with a prompt that determines the language of some given text. I couldn’t figure out a gpt-3 prompt that could handle “This text is written in French” correctly (it thinks it’s written in French), but with gpt-4 you can include in the prompt to disregard what the text says and focus on the words and grammar that it uses.
hellcow 4y ago
My experience with GPT-4 has been really disappointing. It didn't feel like a step up from 3.5. As an example, I've been trying to use it to learn Zig since the official docs are ... spartan. And I've said, "here's my code, here's the error, what's wrong with it?" and it will go completely off the rails suggesting fixes that don't do anything (or are themselves wrong). In my case, understanding/fixing the code would have required GPT-4 to know the difference between allocating on the stack/heap and the lifetimes of pointers. It never even approached the right solution. I haven't yet gotten it to help me in even a single instance. Every suggestion is wrong or won't compile, and it can't reason through the errors iteratively to find a fix. I'm sure this has to do with a small sample of Zig code in its training set, but I reckon an expert C coder could have spotted the bug instantly.

roflyear 4y ago

And you only used the most popular example of getting started and one of if not the most popular programming languages. Amazing.