11 ms·
I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my tim
by Gigachad 3y ago
I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output.
I've only found it useful for explaining basic terminology or concepts for a topic I'm not familiar with. Though this is the most dangerous use case since I'm not familiar with the answer, I can't easily fact check it.
- thomasfromcdnjs 3y agoI've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and paste code from my colleagues PR's and paste it into GPT-4, it explains it perfectly and gives tips on how to improve it (which I add as comments to my PR review)
- master_crab 3y agoI feel like using suggestions for line by line composition might lend itself to better review by programmers. As opposed to ChatGPT which spits out dozens of lines of code based on a request and therefore requires more involved editing after the fact to understand what’s happening.
- wesleywt 3y agoYes, GPT is an aid not a programmer. You should still be doing the programming while ChatGPT gives you quick solutions to problems without having to look up Stack Overflow.
- hedora 3y agoMeh. My coworkers do what you’re doing, and then I end up rewriting their stuff. (AI generated stuff generally passes code reviews because it gets the benefit of the doubt.)
- wesleywt 3y agoMeh. I started coding professionally in the age of AI. My productivity is way higher than it should be. I don't let the AI "write" the code for me. I use it as a companion that I asked question of and suggestions. It suggests solutions to problems, I add the complexity. So far my code has been both clear and performant. This should not be the case at my level.
- enumjorge 3y ago> I started coding professionally in the age of AI. What does this mean? Copilot was released in March of this year. ChatGPT 4, which the community considers to be the only version of ChatGPT to be competent enough at coding tasks, was also released in March. My guess is that you didn't mean that statement to be a fancy way of saying "I've been coding professionally for around 6 months", but I don't know what else you mean by the "age of AI".
- anon25783 3y agoMaybe they think this is LinkedIn? /j
- deleted 3y ago[deleted]
- FeepingCreature 3y agoThe community is wrong, ChatGPT 3.5 is fine.
- Philpax 3y agoMinor correction - the original version of GitHub Copilot was released in late-2021.
- redhale 3y agoThis! I wonder if it's a language difference thing (I use C# mostly), but my experience with Copilot has been nothing short of mind-blowing. It's not writing every line, and when writing truly new feature code it's less useful. But here are two patterns that I've noticed it is especially and consistently good at as a potential place to look for initial value if you are skeptical: - If you have any kind of repeated pattern, even very complex ones, like performing a set of operations on a set of objects, or initializing a set of things, or whatever, it will guess everything else after the first line 90% of the time. This stuff is almost always plumbing/wiring/boilerplate-type code, so pure time gained. Think about Excel filling incrementing numbers down a column, but for pattern-matched lines or blocks of code. - For any reasonably testable class, if I write the name of the unit test, Copilot will write the entire unit test perfectly 90% of the time, down to variable names and //Arrange//Act//Assert comments that I stylistically prefer. Seriously, it's sort of scary how good it is at this.
- toyg 3y agoLanguages like C# and Java have always required a lot of boilerplate and repetitive (sorry, "patterned") code. They are perfectly suited to GPTs.
- justinlloyd 3y agoI've used ChatGPT and Copilot with Javascript & Typescript and a bunch of frameworks. Python, C, C++, C#, Go, 6502, Z80, ARM and a few other languages so far. They've all worked pretty good. I wish it had a wider breadth of APIs and documentation, but it is pretty good so far.
- darklycan51 3y agocopilot? lol I can understand using chatgpt 4 but copilot??? copilot can barely autocomplete... and half the time it does it incorrectly when variables and other stuff are involved
- biohax2015 3y agoCopilot has been incredibly useful for me
- simonw 3y agoSounds like you would benefit from spending some time learning how to use Copilot. Which isn't easy, because there's virtually no documentation for it. Once you get the hang of it it's incredibly useful. I really feel it when I'm working in an editing environment without it now.
- enumjorge 3y agoIt's comments like these that make me wish I could stand behind the chair of the people who get these results because I feel like there's something lost in translation. I don't know if I'm taking your comment too literally but this part confuses me: > It writes lots of code before I even think about what I was going to do in that file. If you haven't even thought of what you want to do in the file, how are you prompting Copilot to write the code you need? Am I understanding correctly that Copilot not only generates the correct code, but it's also able to deduce what code you meant to ask for without any input from you? I don't see how this is possible except in the uncommon case where you are bringing a file to be more in line with other files in your code base that have a similar structure. I do think Copilot can be useful, but I'm confused at how different my experience is from what others are getting out of the tool.
- squeaky-clean 3y ago> Am I understanding correctly that Copilot not only generates the correct code, but it's also able to deduce what code you meant to ask for without any input from you? You can know what you want to program, but not how you would approach the code. "Hey ChatGPT, how do I draw a curved line plot in JUCE?" If I just ask it "Can you write me a script that gathers the current stories on the front page of hacker news and sorts them by number of comments, then prints the title and number of comments?" it writes a script in Python using requests and BeautifulSoup4.
- abootstrapper 3y agoChatGPT 4 is amazing! I use it when I’m blocked to get unstuck. I don’t need perfectly functioning code, just something to get my gears turning. I use it to review my own code. It often has pretty good suggestions, or can validate my approach. And I use it to sketch out unit tests for me. I still have fill in pieces, but it’s absolutely improved my output. It’s like having an egoless programming partner. Yeah, it makes mistakes, but I treat the AI like a programming buddy, not a flawless code wizard. It doesn’t solve problems for me, but we come to solutions together.
- thomasfromcdnjs 3y agoWhat I find useful is I write two lines of comments when starting a new file, about what it is going to do. Otherwise, even just creating a function, generally they will be filled out automatically. I suppose I'm quite big on naming (vars and functions) Maybe if your language or code is more terse, it might struggle more.
- capableweb 3y agoWhich version of GPT are you using? I've found GPT4 moderately helpful, like getting help from someone with a wide but shallow experience of lots of things. > I can't easily fact check it Why not? How did you find out things about basic terminology or concepts you're not familiar about before GPT? Apply the same methods, although you just have to fact check rather than coming up with what to check, so removes a small initial discovery, at least for me.
- Retric 3y agoIt’s a question of time investment, fact checking is slower you than actually learning the material directly. So if you need to fact check every single thing you’re better off just leaning it in the first place.
- pnpnp 3y agoIn the original comment’s defense, sometimes you need to find where to start. For example: when writing a research paper, one might hastily cast a wide net before locating and drilling into a topic from a respected source. I’m not well-versed in JavaScript (I’d like to be, but neither me nor my company have the resources right now to commit). When JS work comes up, I’ll often Google/ChatGPT to point me in a general direction before pulling up technical documentation on whatever comes back. I’m pretty good at learning, but sometimes it takes me a while to find out where to start.
- Retric 3y agoI get where you’re coming from. However it’s easy to fall into the generalist trap where muddling through works well enough that you never spend the time to actually learn anything. It’s risky to really lean the details of a web framework which you may only use for a single 6 month project. However, there’s a few things like SQL and JavaScrip that are likely to stick around mostly unchanged for your entire career. The risks are therefore much lower. IMO, a reasonable heuristic is spend around 10% of the time you expect to work on technology in the next year actually learning the fundamentals until you feel comfortable. Often what looks like a major time sink goes away when you stop fumbling around.
- malux85 3y agoYeah, {zero, one}-shot is frequently wrong, to get good results you need to ToT or GoT (Tree of thought, graph of thought) which is currently only useful in more automated codewriting systems (like I'm building with https://atomictessellator.com https://atomictessellator.com) and not really useful in a co-pilot scenario.
- lockhouse 3y agoAI for nearly everything has been very overhyped. Paintings of people have extra fingers and other strange artifacts. Using it for writing prose is hit or miss. Full self driving cars are thwarted by rogue traffic cones. I think we'll have AI some day, but today it's just not at the level that all the hype claims it is.
- rvz 3y agoThere you go. But even before that 8 years ago, ConvNets, AlexNet, etc were getting all the hype and then came the adversarial images and issues which weren't addressed and then that hype fizzled out very quickly. Now the AI bros here are attempting to sell us their new snake-oil in the form of a stochastic parrot which promises to be the solution to everything. Well, unsurprisingly it is another unexplainable AI black box which still requires the human to check every single output so that it doesn't hallucinate something incorrect, which it does almost all the time with a lack of transparent explainability as to why it hallucinated. So the fact is, it is already overpromising and under-delivering for serious use-cases. Just like FSD (Fools Self Driving) was when that was over-hyped and with little to no Tesla Robo-taxis on the road.
- Retric 3y agoYea, I had a thought early on that programmers might start poisoning the well with these LLM by uploading a lot of broken code. But I think people have been unintentionally doing so for years.
- ianbutler 3y agoI don't relate, I'm currently designing a lower level data structure based on some papers and its been a pretty valuable rubber duck. I've only had one real instance of it being incorrect that wasted my time, higher ranked trait bounds in Rust can only be applied over lifetimes and instead it was suggesting I try to use them for non lifetime generics as well. That said I'm not asking ChatGPT to write code that I use directly, I'm basically asking clarifying questions about the papers and other implementations I have on hand, and that's similar to how I use it in most cases. For code generation I rely on co-pilot, which actually has the context of my codebase. Edit: I will say this because someone else made me think of it: I find this tech all super useful on my like 20k loc and smaller side projects which are all self contained where as when I tried co pilot at work on a fragmented big corp code base I found it comparatively lackluster.
- bugglebeetle 3y agoIt’s both under and over-hyped. The whole “AI is going to write your entire app for you with a couple sentences prompt” crowd is full of shit, but for small sections of code, variations on existing scripts and functions, writing tests, etc, GPT-4 is pretty incredible. I work in data science and have to write a lot of repetitive stuff for parsing and cleaning data. It’s reduced my toil so dramatically in this respect, I can’t imagine going back to writing all that stuff again. I’m now much more ambitious in what I’ll experiment with as well because I know setting up the first stages of the data pipeline are going to be 10X less work than before.
- throwaway09223 3y agoYou have to put it in context: How good is the human alternative? I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer. It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly? It often misapplies interfaces on the first go-around. Again, this is just like humans. We make mistakes, notice them, fix them. If you simulate this by telling GPT that its code produced an error it will often correct itself. The absolutely killer feature of GPT4 is that it has these skills in every subject. It's fluent in kernel operations. Databases. Networking. Various UX frameworks. Any language. It's definitely not perfect. But, if the alternative is hiring a mid-level human engineer, GPT4 is a really compelling alternative.
- xtagon 3y agoA mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?
- throwaway09223 3y agoYes, like I said if you give it the results, eg "That code resulted in this error. What's wrong?" it will more often than not correct its mistake. Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.
- jhugo 3y agoMy experience has usually been that it will apologize and then produce another snippet with a different flaw. When that is pointed out, it will usually go back to the original snippet with the original flaw. It sometimes also insists that it has “run the test suite and they all pass”, just like a human programmer that is trying to fake it until they make it I guess?
- tomjakubowski 3y ago
- corethree 3y agoThe over-hype is causing it to look under-hyped. 48% of the code being freaking correct from 0% about 5 years ago is huge. With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes. This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side. It's typical. It's like an indie band is only popular when not many people know about it. Once everybody starts talking about it all the time it loses it's popularity. People start thinking it's not cool anymore. That's the bias you are suffering from.
- skepticATX 3y agoThere's a big gap between people using it to write greenfield applications and/or smaller tools, and people working on large codebases. I am in the latter group, and I don't find it all that helpful. There simply aren't any tools that can plug into a massive codebase with millions of lines. I never just work on one specific repository either - a change generally involves multiple repositories. On the other hand, if you're writing a smaller standalone utility, or working on a greenfield application, it can be helpful (with all the useful caveats about hallucations). I think that a lot of ML folks, especially those in research, fall into the former category, which is why there has been so much hype.
- ianbutler 3y agoYeah I think this is really one of the key issues. I found all of this tech to be really great with my side projects and really lacking with my big corpo codebase fragmented across multiple teams, projects and libraries.
- CharlesW 3y ago> There simply aren't any tools that can plug into a massive codebase with millions of lines. If Sourcegraph's offering (https://about.sourcegraph.com/cody https://about.sourcegraph.com/cody) has limits I'm not aware of, this is just a matter of time. But in my experience, this is a "nice to have" rather than a "must have" when it comes to the benefits of GenAI as it applies to coding.
- version_five 3y agoI'd go further and say the sweet spot is probably writing a simple function that fits in the model's context window, in a language you don't know. Even here it screws up, but you can go back and forth and get somewhere. I can only imagine the mess one would make using it to build an actual application of any size. The analogy that comes to mind is using google translate to try and write an essay.
- plorkyeran 3y agoI do think that's the big disconnect. There's jobs where you basically only ever create brand new code from scratch - writing lots of little scripts, creating new screens in a webapp, or whatever - and current AI tools can save a ton of time for that. There's also jobs where it's normal to spend a week figuring out what to change and then five minutes changing it, and current AI tools are worthless for that. Most jobs are somewhere in between those extremes, but the hype is coming from people pretty far on the "mostly writing new code" end.
- drewfis 3y agoChat GPT is a game changer for a lot of my work. I make it write stored procedures in MS SQL along with the calling code in .NET, C#, and Dapper. The generated code will likely have a few small issues, but it still makes me way more productive. It allows me to work ~10 hours a week less and enjoy my life.
- toyg 3y agoUntil your boss/customer gets wind of it, and claims back that time.
- m3kw9 3y agoMy experience is that you need to do a few back and forth get get it right especially for a more detailed functions. What works really well is if you need it to explain to you how something works, but to verify, you need to have a good sniff test which novices won’t have so they take it dangerously straight up
- justinlloyd 3y agoI have been happy about 90% to 95% of the time with code that gets spat out by ChatGPT, though I don't fear for my job being taken by an LLM any time soon. It usually gives good insight in to unfamiliar code, or code in a language that I am familiar with but might have forgotten how to use. That said, I don't use it for every piece of code I write. Usually sections where I am exploring an idea. At times, I need to correct the LLM on some usually obscure detail. It got the memory layout of the video screen on the BBC Micro completely mixed up with the ZX Spectrum, and I had to correct it about five times before it got it, and then it stuck for the rest of the conversation. I've been on some wild goose chases, where I have fooled myself into thinking a particular approach would work and ChatGPT has been enthusiastically right there cheering me on. It's like my dog going on an adventure with me, the dog doesn't care about where we're going or what we're doing, it's just excited to be part of the journey. And in other cases, ChatGPT has been exceptionally useful in pointing out some blindingly obvious mistakes. It is like having a very knowledgeable, but exceptionally junior developer at your elbow. If you ask the right questions, and don't except one shot questions to provide perfect answers, it works well for the most part.
- ChatGTP 3y agoyou probably don’t fear your job loss because you’re still generating a bunch of shit your PMO is still confused by. That’s the game developers are going to be playing for a while I think.
- beatthatflight 3y agoI've used it for performance boosts. It's not perfect, but if I paste a method (my own code) and say 'rewrite in fewer lines' to simplify, or 'rewrite to run faster', I've been in awe at what it comes up with that hadn't occured to me. Tricks in python I wasn't aware of, or hadn't considered using a different datastructure that can be searched faster (for some reason my brain always defaults to lists, when a set may be better, for example). It absolutely makes errors though, so unit tests are critical, but I've got code running so much faster as a result.
- FeepingCreature 3y agoMy experience is that ChatGPT often makes errors, but the errors are different than the errors I'd make. So it's easy to work with it collaboratively: I take a turn, GPT takes a turn, I take a turn... and it has an extremely broad spectrum of surface-level knowledge about tech. The most annoying part is it's impossible for it to say that it doesn't know something. Hard to train for, I know.