4 ms·
We gave GPT-3.5 tools to run, write, commit, and deploy code
- angarg12 4y agoI keep reading examples like this, but every time I used ChatGPT at work it ends up saving me a bit of time completing my task (at best), definitely nothing as impressive. I wonder if these examples are heavily cherry-picked, or if I'm terrible at using the tool.
- csomar 4y agoIt's a bit of both. The savings that ChatGPT did is impressive. And these examples are quite cherry-picked.
- RcouF1uZ4gsC 4y agoI suspect that there is a huge selection bias with regards to ChatGPT since reporting impressive results gets the most clicks.
- noahtallen 4y agoTrue, but I think this is totally fine and expected. We’re in an experimentation phase — a new discovery has been made, and everyone is experimenting, poking around, and testing it out. This is the only way we figure out what it will be good at (and what it will bad at!) Someone has an idea, tries to come up with a way for GPT to execute it and succeeds or fails. Along the way, we start developing better techniques and answers to the question, “what is it really useful for?” And in a while, we’ll know a lot more. But right now, we’re learning and discovering. IMO, this is a useful lens for viewing the news around Chat GPT. (Well, more useful than saying “it’s the future!” or “it’s useless” or “it’ll be the end of us!”)
- famouswaffles 4y agoTypical Zero shot Q/A is like coming up to a person, shouting a question and then taking the first then that comes to mind as the answer. LLMs work quite well in zero shot despite this. They undoubtedly work better with a. Chain of thought b. Feedback Gpt-3 performance on MultiArith goes from 18% zero shot to 93% with chain of thought and feedback (or 79% with just chain of thought) All of this is to say that complex chains will produce better results. Probably they cherry picked but that's definitely not the only thing going on
- nl 4y agoChain-of-thought is still zero-shot performance. It's just prompting "thinking step by step" at the start of your prompt.
- famouswaffles 4y agoIt can be that yes but the first CoT paper introduced few shot examples to induce the actual CoT and those generally produce better results. But yes you're right.
- nl 4y ago> the first CoT paper introduced few shot examples It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf https://arxiv.org/pdf/2201.11903.pdf Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.
- faizshah 4y agoWhat do you mean by Feedback here? Is there a paper you’re referencing for Chain of Thought with feedback?
- famouswaffles 4y agoFeedback is a. pasting output of code back in b. For the benchmark inporovement I was talking about, they basically had gpt generate an algebraic and python solution of. Proceed if the output of the two matched, try again if it didn't. Guess I should have added another one - Consensus
- angarg12 4y agoI'm definitely getting better at asking questions, but the tool still seems to have some very rough edges. Here is my most egregious example. I asked ChatGPT to produce code for a very specific approach. It produced ok code, but I didn't like the approach, so I asked it to do it a different way. It produced a second function, but I ended up liking the first one better. The first code had a bug though, so I asked ChatGPT to fix it. It apologized and applied the fix... to the second code (which didn't make sense). I pointed this out, and it profusely apologized... and spitted out the same code again. I corrected it once more time, and did exactly the same thing. After 3 tries I gave up, fixed the code myself, and started a new chat with this code. Had it been a person I'd think it was a troll, a blithering idiot, or gaslighting me.
- nestorD 4y agoPart of it that this is ChatGPT + tools (given via langchain). This opens the door to a lot of power. This is a good example: https://langchain.readthedocs.io/en/latest/modules/agents/getting_started.html https://langchain.readthedocs.io/en/latest/modules/agents/ge...
- BoorishBears 4y agoI spent a ton of time hacking together something like that only to just find langchain supports this: they definitely need to bring the tools use case to the top of the documentation since it's by far the most tedious part of what it offers to figure out on your own
- mlejva 4y agoIt started as cherry picked but as you come up with better solutions and handling unexpected situations it becomes magically good at building things
- noisycarlos 4y agoSame here. It sounds really impressive until i try try it. Ive been trying it as help for coding a web app this week. Sometimes it's really helpful. But often it wouldn't work, I kept correcting it and telling it which errors i got. So it keeps giving me new suggestions that also didn't work, and it keeps adding code that doesn't help. So i end up with a bunch of long code from all the attempts that still doesn't work. So i have to fix it by hand anyway. Like i said, sometimes it's really good, but either I'm also bad at asking, or they have cherry-picked examples.
- jhoelzel 4y ago100% cherry-picked usually i cant even get the complete boilerplate of a project, because it losses its context. let alone the fact that you cant get around the token limit.
- jasfi 4y agoI've found ChatGPT useful for coming up with ideas and possible solutions. Left to its own devices, however, it would fail to do anything meaningful in its current state. It just gets things wrong too often, or at least, requires constant supervision to head in the right direction. The more input is required from ChatGPT, the greater the likelihood that it will break something.
- BoorishBears 4y agoIf you're using it in the default chat UI, you aren't getting exposed to 1/10th of what it can do. Even the playground is a step up because you can modify conversations in-place, change temperature (which influences how often it needs supervision), various penalties, etc. For example, if you want it to generate creative text, but need it in JSON, you can give it an instruction with temperature set to the max (which biases it towards novel output). Then follow up with asking it to transform that output to match json schema with temperature 0 (which biases it towards plainly following instructions) And there's an edit model too now, so in my previous example you can also leave the temperature maxed out, probably get some mistakes, and feed that to the edit model telling it to fix the mistakes - The API is another step up because it makes it easy to chain different conversations. Often times you get a much better result by running different prompts for each step of a process, and in the web UI that'd be a nightmare.
- jasfi 4y agoInteresting, but why use ChatGPT in an API instead of GPT-4?
- dragonwriter 4y agoGPT-4 (depending on which model and whether you look at prompt vs. response tokens) is between 15 and 60 times the cost per token of GPT-3.5. Also, GPT-4 API is doing a slow roll-out, so unless you’ve been granted access, you may not have access to it at all.
- BoorishBears 4y ago
- mlejva 4y agoHey everyone! I’m the original Reddit poster and one of the creators of this agent. We essentially give the agent a full cloud sandboxed environment as it’s playground (it’s Firecracker). We’re working on this agent as a part of a bigger project. This has been one of the most interesting development experiences I’ve ever had. We also learned a new way of interactions with the model. For example giving it an ability to ask human for guidance when it isn’t sure what to do next - https://twitter.com/mlejva/status/1637175051828043777?s=20 https://twitter.com/mlejva/status/1637175051828043777?s=20 Or letting the model tell you what tools it thinks it needs - https://twitter.com/mlejva/status/1636721807129493511?s=46 https://twitter.com/mlejva/status/1636721807129493511?s=46 Happy to answer questions
- evanb 4y agoHave you asked it to try and break out of the sandbox?
- mlejva 4y agoNo… But it might be good idea to ask it for our penetration testing and security
- evanb 4y agoMy question was not a suggestion, it was a probe to understand the danger your project might pose.
- satvikpendem 4y agoMaybe it should be a suggestion too.
- ChatGTP 4y agoGoing to be fun when examples like this start showing up in CTO Magazine (tm)...
- awesomeMilou 4y agoI'm already prepared for the inevitable "You used ChatGPT? AI is on the rise. Why should I hire you instead of investing in AI?" in interviews.
- anonzzzies 4y agoAlready happening. And not a bad question for quite a lot of people.
- boringuser1 4y ago[dead]
- zoklet-enjoyer 4y agoRuns in the cloud. Clouds are in the sky. A network connecting clouds. A sky network. Skynet
- komposit 4y agoYeah this is the future right here. A couple of iterations over this paradigm and we can all go to the beach afaic. Software engineering in our day and age is 1% deep problem solving (coming up with novel algorithms, for example) and 99% writing glue code. The real problem with the 99%, and the reason we get paid what we do is that this requires a lot of thoughtful problem decomposition. Once the problem is adequately decomposed it becomes a ticket that can be picked up, and further decomposition follows in situ. What we should try to do is write a tool which can provide a sort of chatGPT integrated ide where a dev can - specify an overall goal of varying complexity. - ask chatGPT to split this up in smaller subtasks - iterate down the tree until chatgpt decides a task is specific enough for implementation to start - ask chatGPT to write tests verifying task completion - then initiate a feedback loop where gpt can suggest code, run the tests (in a containerized setting), evaluate if output is as expected, and amend changes - once tests pass, commit, move onto the next ticket. Programming then becomes a process of guided decomposition with humans mainly guiding the process along.
- komposit 4y agoOn a related note. Problem decomposition is also the reason why we should all be worried about AI, even with it's current capabilities. After all, every nefarious goal, once decomposed into smaller units, is not necessarily recognizably nefarious any longer. The challenges of organized crime are more in the logistics and HR departments more than anywhere else and their problems, once framed in that context, won't make any AI suspicious, however much training openAI does on its LLM.
- 1letterunixname 4y agoWe're watching the tech singularity unfold. Office jobs of all sorts are at risk, especially underscoring low-skill coders and systems engineers.