8 ms·
Last week I used ChatGPT for the first time for a real world task at work. It was a self-contained lambda function to perform some admin tasks, so it seemed lik
by angarg12 4y ago
Last week I used ChatGPT for the first time for a real world task at work. It was a self-contained lambda function to perform some admin tasks, so it seemed like an ideal fit. Although the experience was good, it's far from the end of programmers. This was my experience:
* Although ChatGPT is pretty good at generating code, it kept making simple mistakes such as calling non-existing APIs or introducing bugs. Some of them it could fix itself, some I had to fix.
* The code provided worked well for the "happy path" but failed miserably for some corner cases. I had to fix that manually.
* The code was working, but I wouldn't consider it production ready. It required some cleanup, unit tests, etc. Again, some of this with ChatGPT, some without.
* Not to mention that I was the one with the knowledge about the domain, what problem to solve, a vague idea of how...
Not to pick on OP but extracting a few seconds of video from a file is a pretty straightforward task, you can essentially do it with a bash one liner [1]. My biggest question is how ChatGPT performs with a large codebase, contributed over time by different authors, with complex domain logic and layers of abstraction.
I also had a brief existential crisis, but I just shrugged it off and got back to work.
[1] https://askubuntu.com/questions/59383/extract-part-of-a-video-with-a-one-line-command https://askubuntu.com/questions/59383/extract-part-of-a-vide...
- bmitc 4y agoYou made me think: it feels like ChatGPT is just a less accurate StackOverflow answer generator. The culture of needing to use StackOverflow is not a good one, so I'm not sure why people are considering ChatGPT to be.
- otabdeveloper4 4y agoYeah, it seems like a better search engine would be easier and more accurate to use in this case.
- sharemywin 4y agobut it can usually answer your specific question. not loke close to your question.
- politician 4y agoCounterpoint: ChatGPT will answer your question in a few moments whereas on StackOverflow, you might need up to 60 minutes for the question to be closed as "offtopic". ChatGPT never asks "why do you want to do this?"
- asdff 4y agoWho even asks on stack overflow? The exercise is the generalize your issue, and then the thread from 7 years ago with your answer appears.
- sharemywin 4y agobut what's the point if a new tool does it for you?
- asdff 4y agoIts a good exercise to be able to write generalizeable code since you can use it in various settings. Its also an exercise to get to the root of your problem, separating the emotion or bias or whatever the context might give you, that way you can talk to it with people who just know about Python and not people who must now know Python + other thing to understand your question. Helps you speak to your experience in other jobs when you are getting interviewed too, even if that experience isn't directly in the same line of work. In the end its up to people what they end up doing with their life and how well they want to understand the things they interact with. I hope people see chatgpt as any other knowledge shortcut in time. Sure, reading the documentation is always longer than asking on reddit or whatever for the immediate solution, but it makes you a better programmer over all, and maybe next time you won't need to spend time turning to reddit/stackoverflow/chatgpt and you end up saving time on the whole. Its like teaching a person to fish versus handing them a fish when they get hungry.
- circuit10 4y ago"The culture of needing to use StackOverflow is not a good one" Not being able to write code without it might be bad but it's a valuable resource and you should use it when it's available to you (for both)
- stevedonovan 4y agoYes, and at least StackOverflow will often give you some minority opinion, not just a snippet to be pasted into your code. Especially if something a little tricky like cryptography. Consider this classic: https://stackoverflow.com/questions/12122159/how-to-do-a-https-request-with-bad-certificate https://stackoverflow.com/questions/12122159/how-to-do-a-htt...
- davidhaymond 4y agoThis is one of my major concerns with ChatGPT, and I'm not sure why it hasn't been discussed more. StackOverflow is a massively useful resource, to be sure, but it takes knowledge to wade through the outdated (or just plain bad) answers. StackOverflow can be a useful starting point, but I don't think I have ever copy/pasted code directly from StackOverflow. I don't think any LLM will be able to replace the skill of reading the docs and learning your tools. I have no doubt that ChatGPT will become even better than StackOverflow at answering questions. Is this really going to make us better programmers?
- LeoPanthera 4y agoFor what it's worth, that was exactly my experience with GPT-3.5, but GPT-4 is a lot better at generating code. Almost spookily good, at least for some languages. It makes far fewer mistakes.
- ollien 4y agoMaybe the ChatGPT implementation of GPT-4 is different than the one in Bing AI, but I tried to ask Bing AI to write a fairly simple Python-based ini-parser yesterday (and by that I really mean using the built-in configparser module), and while it got a good amount of the way there, but attempted to index a string with a string-key, which was weird. After multiple notices of this mistake, it produced something that _could_ work in some cases, but was definitely brittle.
- crop_rotation 4y agoI can confirm that GPT4 is much better than Bing on such tasks (have used both extensively for same prompts.)
- d0mine 4y agoBing is backed up by GPT-4? No?
- crop_rotation 4y agoIt is but they don't have to be exactly the same. Bing might be tuned for searching real time information and maybe cost less since at search engine scale is much higher (just a guess on my part).
- HDThoreaun 4y agoMy understanding is that they're similar but not the same. I think the rlhf process was different with GPT4 receiving much more human feedback.
- LeoPanthera 4y ago
- komposit 4y agoThis skepticism absolutely baffles me. Have you been using gpt-4? To unlock gpt for real you have to be careful to prompt it correctly and find a way to improve the feedback loop for improving code. It is only a matter of time until tools arrive that integrate this into your development environment and give it access to test/console output such that it can suggest code and iterate over the result. It's not perfect yet, but I'm seriously feeling the nature of our work will change fundamentally over two years already.
- illiarian 4y agoSo... nothing changes. It will be the tool for which you will need to manually construct prompts and clean up output (including imagined non-existent APIs). The availability of a button inside an IDE doesn't make this a fundamental change in how we work
- crop_rotation 4y agoIt is so far ahead of even what the best IDEs do. For one, I have not seen GPT4 ever use non existent APIs. You don't need to carefully construct prompts. It tolerates typos to a good extent. You can just type a rough description and the output won't need cleaning manually. You might need to reiterate it to focus on some thing (like remove all heap allocations and focus on performance).
- 49531 4y agoI've seen it use non existent APIs a lot. Working on a project that uses a dialect of python it told me it knew (Starlark) was like pulling teeth. It would tell me to use a python feature Starlark didn't have, I'd ask it to rewrite it without using that specific feature and it would with another feature Starlark didn't have access to, so I'd ask it to write the solution using neither and it would just give me the first solution again.
- illiarian 4y ago> For one, I have not seen GPT4 ever use non existent APIs. Have you asked it to use any API that appeared after September 2021 (that's the cut off date for its data)? Have you asked it to write code in less popular languages (e.g. Elixir)? Have you asked it to write code for less popular or unavailable APIs (smart TV integrations)?
- crop_rotation 4y agoFor the sake of completeness, can you specify whether you used GPT4 or GPT3.5 ChatGPT. The difference is huge. I too was not too impressed by the default ChatGPT. But GPT4 is a huge improvement.
- hn_throwaway_99 4y agoMy experience was different than yours. I had an existential crisis, and it only got worse the more I used it. To be clear, I did not think ChatGPT was ready to replace me now (and, mind you, I only used the 3.5 version of ChatGPT). But it was so easy for me to see how in just a few years it can accelerate its pace of learning. I was discussing a bug with a colleague, so for curiosity's sake I decided to plug a similar question into ChatGPT. I was quite impressed with the solution it gave, and interestingly, it had the same subtle bug that our code had. What blew me away is that when I pointed out the bug, ChatGPT fixed the code by itself. On one hand I felt "phew, at least it needed me to point out the bug", but then I thought "I just (perhaps stupidly) provided training data so that down the road ChatGPT would get it right the first time."
- creeble 4y agoBut did you? Does it store and re-train on all of that input?
- groestl 4y agoIt will, rest assured. There is a reason that you have a history of chats in the free version, and it's not because it's so handy for you.
- matesz 4y agoWow how about people using paid api. Can OpenAI retrain on data provided there? ps. ChatGPT: "You should only share information that you are comfortable with being stored or potentially used as training data."
- sharemywin 4y agoThis video kind of scared me even more. https://www.youtube.com/watch?v=0dBq9sKTKTY https://www.youtube.com/watch?v=0dBq9sKTKTY
- hn_throwaway_99 4y ago
- idopmstuff 4y agoI'm a PM who is relatively technical, but I haven't written more than a few lines of code (stuff like minor modifications to my Shopify site) in a decade. I saw a bunch of people talking about how GPT helped them code stuff on Twitter, so I thought I'd give it a try. Right now I'm building a sort of simple, mock version of the type of software that integrates with my company's APIs. I've successfully managed to create a simple web application that creates a new object, hits my company's API endpoint to create a corresponding object on our software, allows me to upload a document locally and then allows me to upload that document to our software via API as well. It's all a little messy and clearly not production-ready, but it works. It would've taken me probably a few months on nights and weekends do this (mostly refreshing myself on JS and Python). Instead I've done it in <24h (would've been shorter except for GPT-4's message limit). I'm sometimes able to spot and fix GPT's bugs, but even when I'm not, it walks me through adding more logging and successfully debugs issues. Sometimes it takes a few tries and a little direction as to what I suspect the issue is, but so far it's fixed everything that's come up. I don't think this would be doable for a totally non-technical person, but I do think it'll get there pretty soon. I'm just absolutely blown away.
- devjab 4y agoOne of the reasons I'm not blown away is that everything we've tasked it to do has resulted in rather terrible answers. A lot of them outright didn't work. When they did, the way GPT created our solutions were often in a way that wouldn't work well over time. Unfortunately you wouldn't necessarily know that unless you really know your tools. In many ways, this isn't too different from people performing brute-force programming through their favorite search engine, but at least most people know that "Google programming" is sort of bad. I think we're going to be cleaning up the GPT messes for decades to come because it's very confidently incorrect and much more accessible as you point out. I think we're going to see a lot of programmers who are going to trust GPT a little too much, and I think that's sort of scary. For the most part that is going to work out just fine. Often the quality of your programming isn't actually going to matter that much, because as long as it solves the business needs okish, then it's frankly great. That's not always the case, however, imagine someone using GPT to get your healthcare software wrong. I'm still impressed with it in other areas. I think it'll do wonders in the world of office automation because it seems to have the ability to succeed at this much better than any previous "no-code" attempt where the logic would almost always end up requiring people who are basically programmers for it to work. I think GPT will help here, requiring less "superusers" for a department to move their data flows into automation. Especially in areas, where efficiency and stability aren't necessarily that important if the automation-tools mean you don't need three full time employees moving data from one system to another.
- markus_zhang 4y agoJust imagine Amazon paid someone to train GPT on all available API and tons of correct code, then your company uses another ton of private code to train it...and then provide you with 40-50 prompts.
- redmaverick 4y agoUpton Sinclair: "It is difficult to get a man to understand something, when his salary depends upon his not understanding it." This quote highlights the challenges of accepting new information or ideas when they might jeopardize one's livelihood or status quo.
- cactusplant7374 4y agoI see people solving a lot of simple problems with it. How about asking it to design a robot that makes and hands you a cup of coffee in the morning? Something that really hasn’t been done before. And afterwards it cleans the cup and puts it back in the cabinet.
- dTal 4y agoIt is not good at that kind of novelty, but my impression is that the difficulty is that it is limited to a single pass through the network - such "loops" as there are are "unrolled" within the network, into a very limited stack depth [0] (it would be very interesting to analyze these networks for self-similarity). If you want it to solve arbitrarily complex problems, you need to set up some sort of loop. People are already feeding the outputs back in as input in various primitive ways, but I suspect the real breakthrough will come when someone trains some sort of recursive transformer from scratch. (Assuming the current networks waste neurons in unrolling loops, we might possibly even see smaller models). [0] Try the following family of prompts: "_ is an example of _, which is an example of _, which is an example of _...." etc to a depth of your choosing. At some point it bottoms out and you can't get any more levels out of it.
- afro88 4y agoHuh. 2 days ago I built 3 significant internal tools for my company that automated important workflows for our growth, in a language that I rarely use (js), in 4 hours. Something we have been putting off for months because we figured it would take a week or two. It was an exhilarating experience. Yesterday I got a complex data structure out of it in 1h that we'd been talking about but not implementing because it would have taken a couple of days to get right. In all cases it made mistakes and I had to rely on my experience as an engineer to ask the right questions and fix things. But god damn it made me insanely more productive. Don't shrug this off and go back to work. You'll get left behind, and may not have a work to go back to.
- krainboltgreene 4y agoSo wait, just to be clear, you deployed production code in a language you don't use regularly? And this is a good thing? This is supposed to take programming jobs? HN is incredible.
- afro88 4y agoWho said anything about production code in a language I don't use regularly? Internal tools that automate 3 workflows we'd been doing manually. 2 node scripts and a super simple web app exposed on our private network.
- Planktonne 4y ago'Production' doesn't mean 'external'; it means 'actively in use'.
- thequadehunter 4y agoTBH, the most useful thing it does for me is write my boilerplate code and tedious statements. That's infinitely more useful than the knowledge stuff. The other day I was working with the Cisco Meraki API...I knew exactly what the script needed to do, but the calls were tedious and I didn't feel like learning the names of all the JSON columns, so I just had ChatGPT do it. I had to fix a couple mistakes, but the 20 minutes it took was better than having to read all the documentation.
- canadianfella 4y ago[dead]
- kledru 4y agoThe self-confidence of people who dismiss it after having tried it once is impressive. Having said that, I do not think it is the end of programmers, only some of them.
- mirekrusin 4y agoIt already has wider attention span than humans, if anything it'll dominate at precisely more complex input very quickly with evolution counted in weeks not millions of years. Let's wait for "And now... long term memory is all you need" paper.